I want to say too - I’ve known horses like yours, OP. One in particular stands out.
Warmblood mare. By warmblood, I mean 1/4 draft against very nicely bred and temperamented Oldenburg and Hanoverian. She was bred, born, raised on the farm. It’s the local place’s “home” breeding program that is dedicated to ammy-friendly horses. The horses live outside 100% of their lives, on a total forage based diet. This breeding program and farm has produced hundreds of successful horses. Some fancier and more talented than others, but no issues with attitude. It’s their #1 priority.
This mare was different. Everyone had quit her, because she was absolutely terrified of anything and everything, despite a full physical workup. Even the very experienced and esteemed farrier had said “nope” after she blew her lid one too many times.
Some middle aged woman owned her, and it pained me to watch her handle her. That mare was on the verge of an explosion every second of her life. Remember, she was born and raised on this farm - she had seen this stuff for 9 years at this point. The lunge lines hanging by the arena - explosion. The saddle pads stacked on the shelf - explosion. The mats on the floor of the aisle - explosion. Having her feet rasped (by the owner, slowly, carefully) - explosion. A barn cat - explosion. Riding was out of the question. As soon as anyone saw her coming, the aisle got cleared and everyone tucked into a nook to be out of the line of fire from what we knew was coming.
Something was clearly wrong with her physically and/or mentally. You could spend a bunch of money trying to figure out what it is - and this woman did! - but nothing changed.
Think of the money she spent, and the kind of horse she could have bought for that money. How many years she spent not having fun, not improving her riding skills, etc.
Now for you and your horse - Could you maybe make something of him in 5 years? Sure! Lots of these glitter-filled princess stories on youtube and facebook. But those 5 years will be a flush for you in terms of money, fun, and skill improvement. They also could result in giant medical bills for you. They will almost certainly destroy any confidence you might have.
Get out from under this horse now. It’s good money after bad at this point.
Ok, stepping off my story-soap-box now.