Your horse is not seeing you as his mentor. He is not believing what you are telling him, that it is safe to come with you, that you are a superior being who can protect him, and from whom he should take instruction. So you need to get this relationship fixed, gain respect from your horse, so that he trusts you, listens to you, and believes you, and thinks that you are powerful and benevolent, and “kinda neat”, and wants to hang out with you rather than his horsey friends.
Easiest to do this in a non-herd situation. Because currently he is comfortable in his herd situation, and would rather stay with them than hang out with you, or go with you away from the herd. So remove him from his herd situation, keep him in a paddock situation with neighbours over the fence, but ALONE. Then YOU become more important to him when you come to take him into a training situation. You and he are “one on one”, with fewer distractions, easier for you to instigate the relationship you seek.
If he is reluctant to lead for you, use a rump rope, as you would with a youngster who is not educated about leading. A long rope or lariat loop around his butt, you hold the tail end. You can knot the loop, or leave it sliding, your choice. You apply pressure to the halter, and ask for him to yield to that pressure, and walk forward with you. If he is reluctant, you increase pressure on your second rope, the rump rope. This encourages him to step forward from behind. You release the pressure once he is moving as you wish, and praise him with confidence. Should he run backwards, with a spook or resistance to your plans, you go with him until he stops, then ask him to come forward with you again in the same manner. If he is frightened, you reassure him with a pat, and a word. If he is afraid of something, you ask him to approach closer to it, and touch it yourself first with your free hand, then he will approach his nose and touch it also. Then it is OK, because you told him it’s OK, and he has checked it out for himself. Your input rises in importance to him.
Do your ground work with this horse. Lunge him, ground drive him. He needs to learn about “forward” from your cues. When you have forward on the ground, and have the upper hand in the relationship on the ground, you can start to ride. Not before this.
When you have established the relationship with this horse that works, where he looks forward to his training time with you, and wants to be with you because he holds you in high esteem, he may go back out into a herd situation, and you will have no further problems with him not wanting to leave the herd, or “spooking” at things (for any reason- whether he is actually scared, OR if he is using the spook and running backwards as an evasion).
If you do not have the horsemanship skills to do this with this horse, he is not the horse for you. Because right now, he is in control, not you, and this is a dangerous and unproductive situation.
