I have a mare who does this. It’s hard for her in a boarding stable.
Moving your gelding somewhere quieter, like an end stall, and feeding him with his head away from other horses and traffic can help.
But it might also help the “politics” of the situation to help your barn mates (and even your BO?) understand. I try to explain to folks that the problem is that we have put a flight animal in a cage. We took away their ability to leave when they felt threatened, and we made getting fed a big, infrequent event, where the whole herd is riled up to compete for the scarce resource. So what do they have left to do with that worry about their self-preservation? Their only option left is “a good defense is a good offense,” so they do that because they think they have to.
People get pissy when a horse won’t just roll with the program, no matter how unnatural or hard for them it is. I feel sorry for them, so I try to stick up for these misunderstood horses.
That said, I would not tolerate the horse pinning his ears at me in the stall, or anything but polite behavior. My own mare turns to face me with her ears up when I open the door. I don’t think this was always her demeanor, 100% of the time, but she has been taught. All of that “defense/offense” behavior must evaporate when a human being is in the mix, whether that’s in-hand or when someone goes in to feed the horse (I don’t let my mare pin her ears at me if I’m carrying her grain to her, either).
Teaching them to be pleasant (and the “relaxed” will follow) actually helps them feel more secure when they are with you. That’s valuable for their training and their ability to just get along in this world.
Good luck!