There’s nothing inherently wrong with such a breeding, people do it all the time.
However, there’s no way to accurately predict the size of the resulting foal. It might grow to be as big as the mare, or bigger, or wind up as small as the sire. (If you know what the mare produces it’s a safer bet.)
If size is your main objective, you might think about looking for a mature (as in, finished growing) horse that’s the size you want.
Hunter pony breeders might have just what you’re looking for - a Welsh/TB cross that grew too big for the pony hunter ring. They’re usually very pretty, too.