I think this is the way to go. People at my barn clip their horse’s ears, but they keep their horses inside in an unheated, but still relatively warm barn. They put slinkies on under the rugs (with fill), so the horse’s ears aren’t covered, but their faces are fully covered.
When it’s really cold out (I’m talking -40/-50 with the windchill), I check my horse’s ears to make sure she is insulating heat properly. She is basically a yak because she gets so woolly, though.