The most important question is do YOU care?
If so, then sure, shave some time here and there with suggestions above. If you don’t truly care and are just looking for a way to feel guilty about spending barn time as opposed to any other time (cleaning/whatever), drop the guilt. You live once. You’ve got this horse and this trailer now. Enjoy it all as guilt free as possible.
To answer your question - sometimes. Travelling 5 minutes for a hack could be pretty quick. Going to a lesson, well, I’m old school, so everything needs to be cleaned and polished and perfect so half day is not out of the realm of possibility.
My favourite thing when I owned a dedicated hauling vehicle was to leave it at the barn with my trailer during trailering season so I could choose to hitch any time the day before and maybe even leave it parked after getting home if the weather was icky or I was too tired. It was much easier to face the task of unhitching in better conditions the next day.