It is not about whether I have volunteered to help or if I can build a predictive model. There are many other epidemiologists and statisticians who have made their careers doing this work and who understand this far better than me.
It’s about the resources that are being used that are being touted as methods to enhance the safety of the sport. If those resources can’t even predict the likelihood of “failure” at the highest level then it shows distinct lack of validity. It is the same as saying “We can tell you how how many patients will survive a medical procedure, but we can’t tell you how many won’t nor why.”
Again, separate from the fan-facing data that predicts win likelihood, the data they serve the federation and that the federation now provides on an advisory basis to members is a go/no go/caution indicator based on the rider and combination’s history at the level at issue. It is not more specific than that, but it’s also not less specific than that — they’re not just picking winners, and it is predicting the likelihood of failure, just at a much more general level than you’re hoping for here.
But the fault, then, seems to be in the questions being asked by the client (USEA and USEF), not the analytics provider. Equiratings provides a consulting service — they’re not making choices for the federations themselves. If the issue is we’re not yet asking the right questions and not doing the right actions based on that information, that doesn’t mean the stats guys doing the math are useless, it means the USEA, USEF, and the FEI aren’t necessarily thinking through the safety question the way you think makes sense…
You are again, proving my point that equiratings has almost no validity to safety and risk analysis, “ …advisory basis to members is a go/no go/caution indicator based on the rider and combination’s history at the level at issue. It is not more specific than that, but it’s also not less specific than that…”
And that is my point using the large number of falls yesterday at Kentucky as the clear example. By Equratings, every rider was good to go. BUT there is NO mechanism in Equiratings to include the ACTUAL risk variable such as weather, footing, fences types,… as I keep saying.
You are actually proving why using Equratings as anything other than a training tool is pointless.
Ride starts at 53:30. There’s random head shaking on gallops between jumps. Head of the lake 57:30: Jumping out over 2nd box brush, shakes head mid- way over jump. My response, “Dude, you’ve got to take this more seriously!” 59:31: Uneven front legs. 59.35, 59:37 That under-neck, stabby with front legs. I just had the impression that the horse wasn’t mentally committed to his task.
Equiratings never said every rider was good to go. This was an FEI competition, and to my knowledge, the FEI does not ask them to assess risk in any way - they were never asked and they never said (at least not publicly).
I assume you are making this claim based on their publicly available prediction centre - which they have said many times is a fan engagement tool essentially meant to promote betting in the sport (be that in their Eventing Manager or elsewhere). It has absolutely nothing to do with risk. To be clear, that doesn’t mean their work has nothing to do with risk, it means the specific product you appear to be referencing has nothing to do with risk.
ETA: if I seem frustrated, it’s because I am. Equiratings doesn’t have the perfect answer to every single thing in our sport, but they provide many useful insights with respect to our sport, and have moved the needle a longggg way in the direction I want to see it go. Instead of tearing them down for the work they do, why not work to compliment it instead (as in perform complimentary work, not as in praise it). We aren’t going to get anywhere tearing each other down. We will get much further if we act as a team.
I’m going to make a snarky comment… since it’s apparent that horse and rider falls are so prevalent among ULR I propose that if you’re a pro rider who has a horse fall, you must go back a level and do 10 competitions at Advanced/4*, including one 4*-L, only 2 of which can have 20 penalties, before contesting a 5* again. Maybe that will make the sport safer.
@Marigold@RAyers, I don’t mean to be rude, so please don’t take it this way. My opinion is that EquiRatings, even as a training tool, is too simplistic. It’s metrics are easily fooled, biased, and can produce misleading and nonsensical results. You are indeed correct that we need to work as a team and move the sport forward. Take the best ideas from all corners. Have a wide open mind. Be willing to question preconceived notions. But, that sentiment is not a justification to use EquiRatings for something it is not designed to do, and will not do well.
It seems clear, in my little brain, that the organizing body is looking for a simplistic, algorithmic schema, to control sign-offs to move up, say to Prelim, or whatever. They don’t seem to wish to have any human involvement in that process. It promotes a “check the boxes” approach to training for a move-up. This methodology means there is no person signing off on a move-up. So, they can’t be blamed later.
I suggest you educate yourself to the relationship between Equiratings and the FEI. Equiratings is a European entity and is not unique to the US. It has always been part of the FEI efforts and it uses FEI data. As was stated, it uses simplistic analysis based on incomplete data. It only considers a “win” or clear round as proof of a safe ride. Yet, we all know riders who were “green” and killed because they were truly unsafe riders. And we all know super safe riders and horses who have stops and withdrawals because they were the safe thing to do.
As for doing complimentary work, THE FOUNDATIONAL WORK WAS DONE 5 years ago using EPIDEMIOLOGISTS from Bristol and Liverpool. The USEA AND USEF need to duplicate that here in order to establish the American data.
An eventing epidemiologist and hospital safety expert put together an ENTIRE program and even conducted initial analysis and the USEA and USEF decided that was not what they wanted because that was not easy nor cheap.
Don’t ever assume lots of us haven’t already shown that the models used are insufficient and incorrect.
I agree with a lot of your post, and I don’t take it as rude. My one point of disagreement is that the metrics they encourage riders to calculate for themselves to track their own performances are called “simple metrics” for a reason. They are meant to give riders some basic insights into their performance for no money and little effort. My understanding is that this is very different than what they offer the high performance groups they work with.
That said, I absolutely agree that organizing bodies are looking for algorithmic ways to sign off on upgrades as a way of protecting themselves from responsibility, and I think that used as a standalone is a bad fit. I also agree that it’s unlikely that one company or entity will be able to fully meet the needs of the sport for safety, fan engagement, and high performance targets, and we should be supporting companies in focusing on what they are good at and supporting others in entering the market and continuing to move our sport forward - as you say, “take the best ideas from all corners”.
I think I’d have to live under a rock not to realize that Equiratings is European and not unique to the US. Those Irish accents don’t exactly slide under the radar.
What I said was “the FEI does not ask them to assess risk in any way” (meaning rider/combination-specific risk, because I was responding to your comment that “by Equiratings, every rider was good to go”). I am fully aware that they use FEI data to support their work. To the very best of my knowledge - and please correct me if I am wrong - the FEI does not ask them to apply their safety rating system ERQI to FEI competitors. They do not make a go/no go call on each rider prior to the competition.
On this, we will never disagree. As I said above in response to MTT, I don’t believe any metric should be applied in a vacuum. At some point, human judgement and assessment needs to come into it and dangerous riders need to be pulled off course or prevented from running at the level at a much higher frequency than they currently are.
@Marigold Yes, I do agree with you on the notion that EquiRatings really are a simple / simplistic training tool for people. Even having said that, it is clear those simple metrics are easily fooled, as shown in the example I posted on the discussion about the metrics and Prelim move up MERs. As a training tool, the simple metrics should help you get better. But, the horse/rider pair who were bad before and good now, look the same as the pair who were good before and bad now. That’s the way the math of the simple metrics works. Simple averages are well known to have problems as forward looking statistical predictors.
Ironically, EquiRatings reared it’s head on the discussion on this board about who should be our Olympic eventing team. Haha! Let’s use EquiRatings for that and have no selectors. /end joke/. We would never pick a team with some algorithmic math schema with no human input. Yet, we seem to want to sign-off people to move up by doing precisely that. This thread is excellent. We get some statistics discussion as well as cognitive dissonance.
Bolding my own - 100% agreed, and in particular with the bold parts. They should be helpful as a training tool, but they need to be applied by someone who is using their brain to apply them in context, and they are only one piece of the puzzle.
Fortunately, Equiratings seems to agree, as they have been calling them “simple metrics” all along, and I certainly haven’t seen them saying anything that seems to imply they should be relied on upon all others. The vibe I get from their articles seems to be twofold: 1) encouraging people to use them to direct their focus of training when they want to improve their scores and 2) to help people see statistical red flags that may be appearing. They aren’t going to comprehensively tell you you’re good to go, but they might be telling you to slow down, if that makes sense.
I absolutely do not think we should get rid of selectors - some teams have more or less done that in the past and it just does not work (nor do I think we should use a algorithms alone to permit people to upgrade). That said, my understanding is that the high performance part of Equiratings goes far beyond the “simple metrics” the general population are encouraged to try out (otherwise how would they be charging for it?) and I do think that should be incorporated into selection as one part of the process. As much as I suspect the numbers bear out supporting “the old guard” this particular Olympic cycle, in general I think making sure selectors can look at objective criteria can help to prevent the “same old, same old” bias that can pop up in team selection, and that’s important too. Everything in moderation.
The head shaking isn’t random. Look at that horse’s bridle and bit set up. That crash happened because of what was in the horse’s mouth and on his head. Slow down the video and watch it a few strides before the crash.
No shade to WFP as he is one of my idols… but it does go to show you that your hardware and set up does impact how the horse jumps.
I never noticed the bit. No headshake before the fall, but the horse was hollow. Thanks for helping me pinpoint the intuitive, uneasiness I was experiencing about that ride. And I, also, have idolized WFP.
For the record, it was me who mentioned Equiratings in the Olympic selection thread, and it was not in this sense at all. All I did was make reference to a finishing score of 28 being an aspirational goal and cited that their “zone analysis,” a metric that is supposed to help people assess their own performance, suggests that a finishing score of 28 is a boundary line between the top “zone” and the next one down.
@skrosenthal. Hi, I hope my comment didn’t appear disparaging to your post. Sorry if it did.
My basic issue is the mismatch between the EquiRatings simple metrics and the notion that somehow they help in assessing safety for a move-up or were predictive of results in KY this last week (@RAyers has made that point succinctly). In addition, they create nonsensical results in certain cases. So user beware. This has been discussed for a while now.
