I do both. It’s a matter of weighing pros and cons. Both places where I have lived with horses caring for them myself have been blessed with protective “layers” of fencing or space or both. So I have let horses graze without being held, usually not dragging their lead – if I felt I had to keep the lead on, I would probably insist on holding onto it. But I have put a halter on a horse, opened the gate, gone to get another horse, and picked up horse #1 on the way back. My senior horse used to have the run of the yard. And yes, sometimes I have just let one follow. There are some management benefits to their being herd animals.
If I ever were on a busy road, or close to any road, busy or not, or cursed with horses that would actually move away from grass to do something naughty – well that would be another story!
Re-reading – if you were asking specifically about hand grazing, so the lead is a necessity and you’re probably at a boarding stable, then I would probably hold onto the lead, more because I don’t like anything dragging around their legs.