I hope it works out. We have electric fences up but most of our neighbors have horse neighbors and are respectful. One lady even bakes our horses carrot treats and gives them to US to give to the horses.
So when I saw they were having a bbq with young kids, I whipped out the pony and offered pony rides. Boy was that a hit.
Perhaps if you do the right out reach, you can get beyond their ignorance. In the end though, do what makes YOU comfortable.
When I was ten, my best friend and I walked past a horse property every day. We knew NOTHING about horses and thought they were poor neglected beasts. So we snuck them grass and carrots. Looking back I realize of course they weren’t, but when that lady ran out ripping our heads off it didn’t help. To this day I wish she had calmly explained to us the situation.
I don’t blame her, but I tried to learn from it and it has done me well.