Vitex agnus castus will be left alone by the horses - mine looks beautiful this year with a nice bunch of flowers. It only took 4 years to grow- some horses of mine kept stepping on it. This year it actually looks like a bush.
I love Vitex, may not be hardy as far North as OP
@4horses vitex may or may not survive my Midwest Winters.
Also… Tree sized is not what I had in mind for the planters 
