Sorghum is a cane plant and those can be toxic at certain times, like after a dry spell or freeze or a burst of growth.
Other such plants are Johnson grass, that may grow wild right in your pastures or barditches and haygrazer.
When we baled any haygrazer as hay, we tested it to be sure it was not high in nitrates or cyanide.
Cyanide would go away in a week or so and the hay was fine then, but high nitrates will stay in the hay and can poison cattle or horses.
Horses are a little more sensitive than cattle, but not by much.
Any of those can be fed safely if it is tested clean, but we never did feed any such to horses because we didn’t want to take any chances.
We have raised and fed tons of haygrazer to cattle without any problems, but again, tested carefully first.
If any cane hay tests high in nitrates, generally is ground in feed rations, where the percentages become so low as not be a concern.
You just don’t want to feed that high testing cane hay directly to cattle.
A neighbor decades ago, before you could test for it, feed some haygrazer hay to his broodmares and had several affected and I think one or two may have died with kidney failure, the vet called it, terrible lesson for all.
Haygrazer is raised, tested and fed to cattle here in the winter without any problems.
Plenty of people here fed cane hay to horses now, but they test it very carefully.
To have any fields of any kind of cane next to your fence should not matter, as long as your horses don’t get in there the few times that cane is poison, like for a few days after a freeze.
Most sorghum is harvested as grain here, rarely someone makes hay out of the field after harvest, or makes hay out of a field that didn’t make it as grain.
The cane raised as hay are all those haygrazer specialty seeds.
That is because here we dryland farm those, may be different where you are and that may change how you manage those crops, test and such.
You should call your vet and they should tell you what to do, all this may change and demand different management, depending on where you live.