I backed my 3 year old recently. I ride him a day or two a week with groundwork in between. We were basically at the same place that you are, OP. He seemed mentally ready, I did the prep work, so I backed him and he’s taken very well to being a ridden horse. I think he likes variety, so riding is just something else for him to do once a week or so.
I do a lot of posture work on the ground, and also make sure this continues under saddle. Fortunately, he naturally travels and carries himself quite well. Building up his strength before I sat on him definitely helped.
If you think your horse will take to it well mentally, then I would go ahead and back him. I think that a healthy 3 year old that’s been prepped and strengthened from the ground should be capable of carrying a rider of appropriate weight and do some light riding.
I just go as fast or as slow the horse tells me to, really. If I felt that my horse wasn’t ready, I wouldn’t have sat on him at all. I don’t back before 3, but don’t necessarily assign a number/age to all things. If one horse is ready at 3 and another at closer to 4, you just work with the individual horse infront of you, IMO.