I’ve had a few friends go through this, with horses that were just not a fit for their riding skills or goals. They too grappled with the thought that they were failing the horse so there’s a few ways to look at this:
Failing the horse is continuing to force him to do a job he’s not suited to that also makes you miserable (and I can guarantee, he is also miserable), which is be a riding horse for you. What’s best for the horse is finding him a home where he is a fit for the riders skills and desires. The current situation you’re in will never be what’s best for him or you. You will both be happier with someone else.
These situations are a lot like a relationship gone bad. Think of someone you don’t particularly like, someone who has some characteristics that you really dislike and not enough redeeming qualities to make you interested in a real relationship with them (maybe a good looking person who seems pretty nice to you but is a total jerk to all your friends and family). Now what would you say if I said “you need to date this person and just figure it out”. No one is happy. You’re not happy, they’re not happy. No one does that, because you can’t fundamentally change someone so that you can suddenly be well matched. You’ve done everything right by this horse in terms of comfort and veterinarian concerns and yet he still goes like this. You cannot change who he is.
Like with people in relationships, there’s the right horse for you (in fact, there are several right horses for you out there) and there is the wrong horse for you (again, several wrong horses for you). It takes time, experience and maturity to recognize when it’s the wrong fit and to get out of it. Getting out of it quickly takes a lot of that AND a professional who is being honest with you. Your trainer is openly telling you this is a bad fit and yet you persevere. Rely on your trainer’s years and years of experience, he is not lying, he is not being mean. He has you and your horses best interests at heart.
I’ve been a horse owner for 16+ years and it took me the better part of a year to realize a horse I had needed to get out of my life immediately and it was an enormous relief once I finally made that call (he too was super sweet on the ground but became dangerous under saddle). He’s happier in his new job, I’m happier without him and with a different horse. Every horse owner ever has gone through this at least once, often multiple times over decades of ownership. It’s the nature of a sport that requires another living creature, sometimes personalities clash or people (or animals) don’t grow up how you thought they would.