More importantly to me than height is barrel size and neck length.
I agree with everyone that how your leg lays on the horse’s side is affected much more by the horse’s barrel than simply height, and constantly forcing your leg to stay in the right spot creates considerably more work for the rider. Your leg can lay the same on a couch-sized pony and a twiggy 17h beast. The saddle obviously helps with positioning, but any changes the saddle makes between you and the horse’s natural contact points diminish your ultimate “feel” and communication routes. I always liked to mount a prospective horse bareback and see if my leg feel in the right spot. Then, compare in the saddle to see how much your position has changed and whether one felt significantly more comfortable than the other.
Neck length is another thing I always check for; I’m 5’7", but I feel “precariously perched” on any horse whose neck is too short. (Obviously, shorter horses typically have shorter necks.)
Personally, I prefer to see a too-small rider on a too-big horse than the other way around, so that when the rider inevitably finds herself in the wrong position, her relative weight will disturb the horse’s balance less. A horse, in my opinion, should be able to shift his weight and his rider’s when he finds himself in a bad spot–if the rider is too big for the horse to maneuver under, there’s an issue. If the rider is going to fall, don’t wreck the horse with you. My big TB saved my hiney plenty of times when I found myself bounced out of position. My leg fit him great and I never lacked control, but weight-wise I was a fly on his back. It gave me a little breathing room: if I got left behind over a jump, my weight on the saddle was less likely to be detrimental than if I had done the same on a pony. (Of course, that was back when I was a wee-thing in high school!)