You are PAYING him to ride your horse. “Bother” him all you want!! (Not to mention, you are not bothering him.)
It’s not hard to make arrangements; this is your horse. Be clear with him. Ultimately, it is your horse, your money, and you call the shots.
If you’ve been riding for 25 years, and the trainer has already had him for 6 months, I’ll bet you are plenty qualified to ride him. Sure, he’s young at 3 years old and still has things to learn, but he should very much have a decent handle on him by now. No, he’s not going to be perfect especially if he is the spooky type, but I’m sure you can handle it.
Back in my college days, I started colts for people. Out of all the ones I worked with, I only had one horse who I was worried about the owner riding; simply because this horse was one of the rare ones that was mean. I normally am out of the round pen in a week or two and riding in the open on the colts, but I wasn’t daring to go out of the round pen even after 30 days with this one. Not spooky … just waited for the opportune moment to try to “get” the rider (although she never was successful with me). I had her for 60 days and warned the owner when she took her home (she did ride her at my place before going home). Sure enough, she bucked her off at home and did what I suggested and sold her. But that’s a little different.
My mom currently has a coming 7 year old and he is STILL so darn spooky. Not at anything right in front of you, but things he “sees” a ways off. Of course, he’s also cutting bred so that may have something to do with his reactive personality. My mom has been riding her entire life (she’s now 60) and she’s owned him since he was 6 months old and has done everything with him (except for 30 days to be “started” at the trainer), but she just still doesn’t trust him because of the spooking. She’s “accepted” he’s more of an arena horse and has been riding him as such.
So it’s possible your horse may just be the spooky type, but at 3, he is still very young and he may grow out of it with time. Is trainer riding him only in the arena, or out on the trails? Sometimes covering a bunch of miles out on the trail can do the trick.