Apparently, there are more than a couple of people that do play the piano and think it is that simple; well at least for them.
In the spirit of full disclosure, I always struggled with bass clef so was never great at piano; much better at organ when I could just use chords, coronet, French Horn and classical guitar. Treble clef on a piano is light-years easier than on a guitar, IME, where it’s also not unusual to see notes ledger lines above or below the staff. Obviously, the piano has a larger range.
I think what you’re poking at is probably the correlation between being good at math (which includes pattern recognition) and good at music. That, and people just learn differently.