I used to feel this way, too.
Until I had a horse who really, truly benefited from big fences. Ask @Xctrygirl. My ingrained instinct is to keep fences low until the rideability is sufficient to move up. But I had to learn SOME horses are different and you adapt to what works best for them. My big, strong, athletic gelding railroaded through life over 1m and below, I did poles/cavaletti/bounces all the usual low stuff. I was still working REALLY hard to hold him together and keep his attention on the little fences and do the job correctly. Because there were no consequences to doing it wrong…he could run through the exercise, legs a-flailing, and athletically come out the other side saying “See, I didn’t need to do it your way I got it done.”
Xctrygirl suggested I put the SJ fences up-- way up, outside my normal comfort zone for a N/T horse. At 3’6", 3’9" and 4’, my big horse changed; he was rideable, my half-halts worked, he picked up his shoulders, he sat down on his hocks, and he LEARNED from his mistakes. And I learned, too-- to be willing to change, adapt, and try new methods. (Note: I did not bump him up on XC until our jumping rideability was more confirmed). I am VERY careful with this horse; he has talent, he is super brave, but he’s the type of horse who will get someone hurt, because he Never Says No and will absolutely try to jump anything, anywhere despite poor balance. So I spend a LONG time working on balance and strength, and I’m keenly aware of my decision-making when riding him. He’s turned into a really fun preliminary horse, and when he’s mentally ready for more complex questions, he could move up to Intermediate. This horse has bullet-proof confidence; no danger of losing it when he makes a mistake, but that comes with its own important rider responsibility.