Don’t ask for the downward transition right away after you get the correct response. Just leave him alone for a minute. If he slows down again, that’s fine. You want all pressure to be off. You then just repeat your ask for more forward. Asking for the downward is still an application of pressure. Just in the opposite direction from the question you are working on. Once you get to where you ask for forward and he stays forward after you release pressure, then you can ask for the downward to reward him with rest.
I’m working on some of the same issues with my horse. I have to be very aware not to ask for a slow down in any fashion if the upward happens with too much energy or a bit of a temper tantrum. I have to let go of everything except maybe one rein to steer. You want to encourage relaxation even though you are having to work on responsiveness. You can keep him mentally engaged by turning, circling, changing directions, with a light hand. Or put some poles out or something. I think you are skipping ahead a little bit asking him to come back so much.