Then you’d have to make it a rule that your groom has to be an amateur though. Or ideally an amateur who rides at your ability level or worse than you.
The spirit of the rule is that only the athlete rides the horse but at some venues that’s super inconvenient so the groom is allowed to hack it the half mile from stabling/parking to xc rather than trying to lead it. (Looking at you, Fair Hill)