I guess my experiences must be unique 
I remember reading similar negative stories and warnings prior to buying our farm and bringing them home and was really concerned about how it would go- but my experience has been the polar opposite. My paddock does not share a fence line with any other properties (it’s in the middle of ours, next house is separated by about 1 acre of woods) but there is a sidewalk on the street that goes by the paddock that is very frequently traveled. But other than 2 or 3 odd experiences (in nearly 6 years) we just have not experienced the negativity that is being warned by others.
We are very open and friendly with all of our neighbors who have expressed interest. Most people came over one time to introduce themselves and ask to pet the horses (which we obliged under supervision, with the conversation about not petting them or feeding them alone) and then not come back, content to just watch from the sidewalk outside of our property. I’ll even pause my rides sometimes to go say hello to a curious child walking by with his/her parents and let them pet my horse (my riding area is next to the street, with a stone wall along the front as a boundary). We smile and wave to people walking by if we’re in the yard, lots of families with kids will stop outside of our fence to watch them for a little while, but it’s been such a positive experience. People love them and have for the most part been very respectful of boundaries. Beyond that, we have multiple neighbors to whom we have become very friendly and close who help keep an eye on the property and who have our contact info and will tell us if anything is wrong or questionable (or just the latest antics of the crew).
I guess to each their own. I recognize that there are good and bad neighbors, and varying experiences, but taking the friendly, open approach has absolutely been the right one in our experience. So OP, YMMV, but taking an initial friendly approach and then reassessing if it becomes a problem might be the way to go. Personally, I can’t imagine putting up “no trespassing” signs all over our property and other warnings (we do have the electric fence warning on our fence)- it would just totally change the vibe for our place and the relationships we’ve developed. I was a horse crazy, horseless kid- I remember what that feels like and rather than have a “this is my horse, none of your business” attitude, I try to give them a fun experience and memory that I would have loved to receive were I in their position. JMO :yes: