Like everyone else has said, the relationship of your upper body to the length of the horse’s topline, and your leg length, hip width, and hip flexibility to the horse’s rib cage; and the general scale of the human respective to the horse.
I’m an average sized person at 5’7” but I have long legs and a somewhat short torso for my height. I’m also built like a brick house with broad shoulders and a lot of muscle. My horse is 15.2 1/2. If you stand there and look at him for awhile, you realize that his skeleton above the legs is built to be 16.2, and the legs are built to be 14.2. Fortunately his overall length averages out. Because he’s fairly compact nose to tail, he suits my upper body; and because he has a massive ribcage for his overall stature, he suits my leg. My sister is 5’9” and willowy. Because of his ribcage he takes up her leg, and because he is built on a bigger scale than she is, she doesn’t look inappropriate on him, either. Conversely, my mom is 5’7” but long in the torso with average length legs and less hip adduction than I have. She can’t get her leg around him comfortably and it doesn’t make as nice a picture on the flat. But all three of us looked suitable on my sister’s late TB, who was 16.3 and built fairly narrow. He was tall but slab sided, so he took up everyone’s leg while not taxing my mom’s hip flexors so her leg could fall in a correct position. But, since he was a more slender horse than I am a human, if he had not been tall, I would have looked too big on him. I rode a friend’s 16.1 TB a few weeks ago, who is also built finely. He’s not a small horse, but my proportions made him look a little small.