Depending who asked and how they asked, I might let someone sit on my horse. My boyfriend has taken “pony rides” on him. So has an ex and my friend’s boyfriend. For some reason I get a huge kick out of watching non-horsey 20-something year old guys ride a horse. :lol: My horse is pretty safe, he’s extremely quiet and sensible. But I only let those people walk and I held the reins the whole time. He’s quiet and sensible, but if a leg swings back and kicks him, he will go faster. Probably just a trot, but still. Didn’t need any 20-something year old guys boinging off my horse at the trot.
If a random stranger asked, no way. For one, I just plain do not want to share my stuff - including my horse - with random strangers. But I’ve never had any actually ask. I must have that neon sign over my head, too. 
Co-workers…eh, it depends. Depends who it is and what mood I am in when they ask. I can think of a few who I would probably say yes to, and several I would say a firm no to.
But they too would be riding at the walk with me holding the reins.
Family members? Maybe. Again it depends. My mom, who does not ride, has sat on him. She actually took lessons for awhile, so I was going to tell her to trot, but he scared her. 16.2 hh is awfully big I guess when you’ve been riding the 15 hh school horses.
Her husband, possibly. We all went riding on vacation once and he had never been on a horse before. I was, frankly, impressed with his natural feel. For a first-timer he did a pretty damn good job.
I’m waiting (impatiently :)) for my nephews to get older so they can ride him. The one is 2, and while I see no problem tossing the kidlet on a pony and leading him around with my hand on his back, his mother would have my head. Plus, it would terrify him. We took him to a petting zoo several months ago and he was horrified of the farm animals. :lol: But someday I’d love to stick him up on my horse.
Come to think of it, I’m more inclined to let a non-rider on my horse than a rider! A non-rider slouching on his back won’t hurt him, but I’ve known several riders over the years who you couldn’t pay me to put on my horse. Not a lot, but definitely some.
Really, there’s nothing wrong with saying no to whoever asks. Your horse is YOURS and you do not have to let other people use your possesions, no matter what it is. If you want to, great. But if not, just say no.