Don’t be too concerned. Your horse is worried about because he knows you can’t protect him. Now.
If the BO plans to return the other two horses to the group it would be worth asking if you can do some ground work with the bully first. Just you moving the horse around a paddock or ring with body language, voice, and a flag or longe whip. If you’re uncertain about working him loose start with in hand work. This will allow you to get some respect from the bully horse and learn the body language you need to keep him away from yours while you are bringing yours in.
When you walk past bully’s paddock does he come over to the fence? If he does you could send him away and show your horse you’ll protect him. If he doesn’t a brief halt, back, turn on forehand, walk, halt, turn on forehand other way with your horse could help remind him that he is to be paying attention to you. An alert ear and eye on the other horse is fine, but most of his focus should be what you are doing with him.
There is a paddock sharing a fence line with the sand ring where I board and last summer one of the horses in the paddock started charging at us as I rode along the fence. After some yelling and whacking the fence with my whip I now only need to growl “Giiit!” or his name at him when he looks like he’s thinking about trying it. My horse was a bit worried about the horse, the yelling and the whip at first but he quickly learned that he wasn’t in trouble. I had to make the other horse my priority for a few rides so that he didn’t surprise me.