Leave the black walnuts. The risk of taking down a black walnut is far, far greater than leaving it, as you’d have to block off the entire area to keep horses away from all potential sawdust. 1000s and 1000s of horses graze in pasture with BW without issues. Just take care of broken branches, and if a tree does come down badly enough, you’ll want to block it off until you can assess the issue. It’s the wood itself which can cause laminitis if a horse stands on it long enough (which isn’t terribly long, but not instantaneous)
What kind of chestnut? There’s American chestnut (Castanea sp), there’s horse chestnut (Aesculus sp).
Castanea isn’t toxic, though I suppose it might be if a horse really chowed down, something they typically won’t do if there’s enough hay/grass.
Horse chestnut is much more of an issue, but it’s like oaks in that most horses have to eat more than a nibble or 3. These nuts aren’t safe for people to eat either, but American chestnuts are.
I’d be inclined to remove the Horse Chestnuts if it’s a feasible amount of work and cost.
How many acres is that?