I love cotton! It breathes! I rode years in Pittsburg winters with cotton turtlenecks and sweaters! I was never cold or clammy, if anything, the low temps were hard to breath when really working. I decided not to ride under 25 degrees.
Layers. In very cold weather, I will start with a cheap, tight-fitting camisole. It’s cheaper than smart wool. Then, silk undershirt, with or without turtleneck. Or smartwool. And/or polyester or cotton turtleneck. Cotton breathes, polyester doesn’t. Then, wool sweater over this. Thick wool, people, not fleece! Wool breathes! A thin jacket or barrier (I used to like flannel lined polyester) to keep in the heat because the wool is permeable and not windproof. GOOD gloves - I have literally 6 pairs of different weights. I usually wear a silk scarf I can take off. Fleece ear warmers under your helmet.
Definitely, fleece breeches for the legs.
Wool socks for wither riding. I ride in Paddocks and have the Ariat Insulated for cold weather and the Ariat Frostbiters for really cold weather.