My best advice, to layer properly and remember to take off layers during the ride! When I get on, I have my jacket on, but I also have an extra cooler over my horse to walk with. When I am ready to start working, I take off his cooler and my jacket (he usually has a quarter sheet under the cooler). Then I warm up some, and if it is warm enough, I take his quarter sheet and also shed another layer.
The other key is my layers. I always always have a wicking base layer on the bottom (underarmour but also have some others that work, even use cotton turtlenecks sometimes), then a vest (synthetic but wears a lot like a down vest), and then my jacket(s). If it is extra cold, I wear a sweatshirt between the base layer and the vest. Usually the top jacket comes off before I get on, then the second one (or the vest) comes off mid ride. Then while I’m cooling out, first layers go back on (same with the pony’s layers) and when I get back to his stall the top layers get back on.
Another thing, the headband might not be helping while you ride, I find that until it is solidly in the low 20’s and below (in the ring), I can’t put my earmuffs on, it is amazing how much warmer it makes you (I also can’t wear a scarf riding until it is super cold). Take them off, put your helmet on, take the helmet off, put it (or a hat) back on