Purely anecdotal.
My sister and I, separately, bought an unstarted 6 year old mare, and a badly started 10 year old gelding from the same TB farm. The owner was primarily breeding for the track, but several of his breeding were successful sport horses (the most well known was JJBabu). But he was only in the breeding and selling business. If they didn’t sell as youngsters, he just hung onto them in the hope that they would sell the next year.
If they hadn’t sold by the time they were 3, the mares (if they were not part of his breeding program, were just turned out in the pasture. He put out feed and hay for them, and put wormer in their food twice a year, but they were otherwise unhandled.
The colts, on the other hand (he didn’t geld them) were kept in a barn, each with its own paddock. So they were handled (led between stall and paddock), but had no other training.
Some time when he (the breeder) was in his 90s, he was persuaded to stop breeding. A couple of years later, he was persuaded to geld all the males that were not breeding stallions, and put everything up for sale. Whatever had not sold by a certain date would be sold in an on-farm auction.
My sister bought a 6 year old mare, who was the alpha mare in her herd. She was very difficult to catch. We later found out that she had preciously been “sold” twice, but when they couldn’t catch her, the buyers switched to a different mare. The first couple of months were full of drama, as the mare was used to being “in charge”, and didn’t want to take direction from anyone. But once she decided to work WITH my sister instead of AGAINST her, she progressed very quickly. In fact she went from unbacked to competing (in eventing) at Prelim in 2 or 3 years.
Two years (on a Wednesday) later I got a call from one of the people involved with the breeding farm. One of the horses which had been sold at the auction, was in a bad situation. (he was also full brother to the horse my sister bought.) The guy who bought him had taught him to lunge, but when he tried to back him the horse just planted his feet and refused to move. He (and his wife) got fed up with paying board on a horse he couldn’t ride. Unless he was sold by that Friday, he was going to the Thurmont auction on Saturday. On top of being an unridable 10 year old, he had lost a shoe and was footsore. His prospects at Thurmont were not good. I went to look at him, agreed that the obvious lameness was consistent with being footsore, and determined that he was anxious to please.
I bought him (bringing him home on Friday), and went back to the very basic ground work. When I was ready to back him he was perfectly willing to move forward. He was not as brave as his sister, and WOULD plant his feet if asked to do something he thought was scary, and not as good a jumper, but turned out to be very competent eventer up through Novice. I leased him out to two junior riders, and he took good care of them. I also used him for the 6 year old niece of a co worker, who would come for a visit a couple of times a month.
He was the third horse I had trained “from scratch”, and he was actually much easier than the previous two (who were more appropriate ages).
Bottom line - from my perspective - being unstarted at 5 would not, in itself, concern me at all. What is much more important is whether the horse’s basic attitude is a desire to please, or “you and what army?”
Both of these horses ended up having arthritis in their mid teens, but I suspect that was genetic, rather than a result of starting late.