In my experience - most of the time “spooking” isn’t because the horse is actually scared, it is because they know it is a fantastic way to do what THEY want, rather than what YOU want. My horse tends to want to ZOOOOM home - and she will use spooking as an excuse to scoot or bolt home faster, its not because she is scared, it is because she is SMART.
He has your number, and has figured out this little game he can play on you to get his way, he tested the waters and it worked, so he escalated it. I have seen many horses develop “spooking habits” as it is a great evasion – not because they are naturally flighty horses (I am betting your drafty doesn’t spook and whirl in his paddock).
Like others have said, forward forward forward! Become VERY attune to his body language, and catch those spooks BEFORE they happen. When my THINKS about adding a spook, she gets put on the bit, and asked to side pass - if she does manage to sneak in a spook, she gets a strong aids, corrected (including voice) and sent immediately to WORK. They need to learn that spooking results in MORE WORK and not getting THEIR WAY. Make it NOT fun for them.
And on that note - you do not want to keep the ride all negative, when he IS complying, and marching along - frequent praise! Give him a stroke on the neck, and tell him how happy you are with him. Figure out what scritches he likes - my goal is to find a “button” on my horse that makes them perk their ears or sigh with satisfaction. They needs to know not only what is undesirable behavior, but what IS desirable. Make it clear, let him know when he is doing right – and make it something he can enjoy as well. I am astonished that horses even let us ride them, and I try to be considerate of them and make my rides rewarding and enjoyable as possible. I want a horse who LIKES to head out, because they know that there will be fun and rewards to be had – not just a workout full of punishments.