That right there is a herd-sour horse. Aka herd-bound. If it is horses or his favorite stall or other spot.
He’s big enough to have gotten away from people, so he knows that he can. This quickly becomes a deeply ingrained behavior. It can become intractable.
Of course I haven’t seen/experienced it first hand, as you have. But herd-bound behavior can become very, very difficult, even dangerous, if the horse is getting away with it.
Quite honestly, if he gets away to the herd at all, this will become worse and worse. Whatever it takes, it has to be stopped. No exceptions. Not by fighting with him physically (he has all the advantage) but by manipulating the environment and situation, with brains, not strength.
It is up to the humans to manipulate the environment and everything being done to make 100% sure he cannot get away from anyone, ever. Creating physical barriers he cannot overcome; restricting who is allowed to lead him. Anywhere.
A chain alone isn’t going to help, at this point. You have to find paths or even erect fence panels to control him while he is being led. Don’t count on humans to be strong enough. And don’t let anyone lead him that he gets away from.
Every time he gets away from someone, it drills the behavior in more deeply. For the rest of his life.
As his owner, this has to be the most important priority with him, for the rest of his life. Even after he gets better about it – it will always be there in the back of his brain, ready to come forward. This can become a very dangerous behavior to some people who may handle him.
It sounds as if you were making good progress. If spring grass has made things worse, you must take measures to bridge the spring season without him breaking away.