My best advice is that it is an individual decision for everyone and that a good trainer should be able to help make the decision. I know you’re already working with a pro, so I would follow their lead.
That said… I commented on the previous iteration of this thread about a horse I had that made me quit riding for 17 years. She was amazingly talented, incredibly gorgeous and scopey (GP level scope), but she was a pro ride and I was a junior. Because she was amazingly talented, incredibly gorgeous, and scopey, the ambitious idiot junior I was didn’t understand that she was a pro ride until it was too late. She could have been the horse for me in a few years, but was not the horse for me at that time.
My current horse is quirky and not a beginner ride, and I got her less than two months after getting back in the tack. It hasn’t always been easy, and we have gone through phases where I haven’t been having fun and I worried that I was overhorsed. That said, we worked through the downs and she is 100% the horse for me. It just took us a while to get there - which was primarily me getting over my fears, figuring her out, and getting fit. Upping my game as you put it. And, my trainer had a come to Jesus talk with me about me believing that I was good enough. Once I believed in myself - that I could ride her well, and get through any rough spots - things turned around quite quickly. We also figured out management changes that have mellowed her out considerably.
So really, I had to commit to me getting better mentally and physically and to figuring out what it took to make her happy. If I wasn’t committed, and if I hadn’t been able to find the mental toughness and confidence to commit, then I would have probably been overhorsed. Now, we are great partners and I know that her spooks aren’t going to get me off, that I can influence the ride and make her easier by staying balanced in the saddle and educated in my aids, and that we will always get to the other side of the fence. I trust her to take care of me, and I trust me to take care of her. And now, she is SO MUCH FUN.
Others don’t want to have a horse that takes that kind of work, or cannot make the commitment for myriad reasons, which is fine too. I have a friend who cuts bait when they are intimidated quite quickly and is very specific about what they need to feel safe. That’s fine too. It’s about knowing your limits and enlisting the advice of others more knowledgeable than you when you’re unsure.
All that said, the getting kicked out of a program and unpredictability give me pause. I know he’s a 4 year old and an OTTB, but have you explored ulcers or other causes of unpredictability? For me, it turned out my horse had asthma and between treating that and putting her on lady hormones, she has become a very different horse. Ditto for an asthmatic, ulcer-prone gelding I know who is a different horse after those things were addressed.
Horses are animals, and to some degree unpredictable. But for my horse and the others I ride, I know what their spooks, scoots, and fresh attitudes feel like, and I am confident I can ride them out. If I was at all unsure about how a horse might behave day-to-day or whether it was beyond my skill set to ride it out (or deal with it in-hand), then I would 100% move on personally. If my trusted pro wasn’t confident with the horse, even more so.
Long-winded. Hope that helps a little.