I had a horse very like this. A big GORGEOUS bay ISH gelding. He was the most beautiful jumper and absolutely lovely to live with and handle. But he could drop his rider like you wouldn’t believe. I lost confidence and the will to ride. He just wasn’t started correctly and all of the trainers I sent him to and all of the “back to basics” just didn’t really help in the end. I sold him to a barn where the trainer would be the one primarily riding him, but from what I can tell, hes never been successful. Fortunately, I think they will keep him forever, because he really is that pretty and that sweet. Just zero confidence under saddle. The horse I got after him was un started when I got her, I did all the work on her, and she is as trustworthy as they come. In all the time I’ve owned her (and all the time shes been under saddle), she has NEVER dropped a rider, and the few times someone has gotten unsteady, she makes sure they stay on. With the gelding, I was about ready to quit jumping, I just didn’t trust him, We tried so hard to make it work, and it just never did.
OP, good luck with a tough decision!