Hi! I am also a long-time lurker and have never mustered the courage to post, but your story encourages me because I am in a very similar boat.
I rode consistently age 10 - 24, owned my own pony and horse through those years, and was very confident in my riding skills back then. I took a 6 year hiatus initially because I was getting married, moving states, buying a house, etc. Life got in the way. During that time period I sustained chronic injuries to both of my knees, had multiple surgeries, and still have trouble today, in addition to some foot and ankle pain.
I started back recently and have felt all of the things you are feeling. I can’t coordinate my body as well as I used to, and I don’t have the confidence in myself that I once did. I, too, find myself comparing to other riders, younger and older, who seem so much better than me. It actually makes me clam up and ride even more poorly when I am noticing other people riding around so much better. 
In addition, dealing with the knee and ankle issues has made things exponentially harder. But I love it, I’m determined, and reading in the “riders with health issues” section on this forum has given me a lot of courage and hope.
I don’t have much more to add to what everyone has said except you are not alone and I felt this post deeply! It is helpful to get more time in the saddle, but I guess just taking things slow and as your body allows is also good. I am still anxiously waiting for the day that I can ride and not have my inner thighs feel super sore the next day!