I wouldn’t be too worried about the rings getting banged up, but actually losing your finger! If the ring got caught in anything, and the horse takes off…
A friend of mine who is a nurse and runs her own barn has seen several broken fingers, warning, gory!, skinned fingers, sliced off fingers from accidents that had to do with people wearing rings while working at the barn.
If you don’t want to take your rings off, wear a glove.