God forbid that the hoi polloi who pay their dues and plod along at the lower levels can access such arcane knowledge.
42 Likes
The internet never forgets. Here is Joel Smithās articles that he did for Minitab many years ago
Olympic Judging: Fair or Biased?
Most statisticians, Six Sigma practitioners, and quality professionals know to evaluate their measurement system before making any decisions based on their data. This is, of course, because we want to trust the data and the information it gives us. Long bothered by the idea of subjective judgesā opinions determining the best performances in Olympic events, I decided to analyzeāat length (now would be a good time to grab a cup of coffee before proceeding)āthe results of two events to evaluate whether or not the judges were consistent and fair.
He found that Menās 10m diving, the judge played no role (good) in the scores, and scored the dives according to performance.
The pairs figure skating was a different story where a judge admitted she had been pressured to vote for one team (Russians).
How Olympic Judging Stacks Up, Part II
There is a large amount of information to provide in a single graph, so Iāll explain how to read it using the last event listed, Equestrian Individual Dressage GP Special, as an example. The gold bar extends from the 1st place finisher through the 3rd place finisher, meaning that while in real life the 1st place finisher was awarded a gold medal, statistically you cannot differentiate that personās score from the individuals who finished 2nd and 3rd. The same would go for silver: you cannot differentiate the 1st and 3rd place riders from the 2nd place rider. Finally, the 3rd place rider cannot be differentiated from the 4th through 9th place riders.
How Olympic Judging Stacks Up, Part I
For the most part, Olympic judging is pretty good considering it involves human beings subjectively evaluating often-unique performances without the benefit of multiple viewing angles or replayā¦just think of how quickly an Olympic dive happens or, conversely, how long a gymnastics performance might be and how difficult it is to accurately judge that performance. And yet most events have no evidence of judging bias and a very high R-Squared value, indicating there arenāt other factors missing.
Itās troubling, however, that outside of Diving the interaction between Judge and Athlete could not be evaluated, and this term represents the most scandalous of judging biases ā one in which particular judges favor (or disfavor) specific athletes.
Although only one event showed a significant judging bias, it should be noted that in dressage events the judges are placed in different positions around the ring and the judge effect is completely confounded with and mathematically indistinguishable from a āpositionā effect.
The statistical methodology is out there. You donāt need some new-fangled dressage analyticsā¦you need a good statistician and the will to utilize the tools that are already in use by many industries.
4 Likes
While my statistician heart deeply wishes that to be trueā¦I actually really wouldnāt think that. Iād think, as we see playing out, that the governing orgs want to put on the best show to hold on to their meager audience, while ensuring the highest level of competition keeps the money happy and ostensibly feeding the upper levels.
There are other fields where subjective judgements translate directly to individualās future employment. For years I ran āinter-rater reliability panelsā (similar to the AAA you described) that were used to determine whether or not scores from external observers were normed well enough to be used to make decisions about certification, licensure, and bonuses. While I would stand behind the system, when weāre talking about subjective judgements translated through what feels to many like an opaque statistical black box, people push back hard. Even with trainings, rulebooks, rubrics, etc. scoring systems are viewed as inappropriately designed to make a numeric judgement on what many people believe to be a subjective craft. You could fault system design, implementation, the quality of the raters, etc. and all of that is valid. But at the end of the day, when you donāt have clear cut agreement on the way to measure the outcome, you canāt have clear cut agreement on how to rate the inputs, particularly when the system is high stakes and designed to pick winners and losers.
1 Like
Agreedā¦that is why the first place to start is a dialog to discuss the āgrading systemā for the quality of the judging. This is where TPTB did not want to go there.
I am familiar with inter-rater reliability, Kendallās coefficient of concordance and Fleiss Kappa.
Edited to add link on Minitabās on these tools for doing Attribute Agreement Analysis which includes topics touched on my @MorganMaresVT on inter-rater reliability.
4 Likes
The Dressage News article says sheās become known for ācalling out Parraā (what does that mean ?) and did so at last yearās US Championships as the Young Horse coach. If she filed a protest against him, it would have gone to a USEF Hearing Committee.
3 Likes
Wow. Shame on the USEF.
From COTHās article today about USEF rule changes:
"USEFās existing rules only allow it to discipline riders for abuse occurring in conjunction with a competitionāeither on show grounds or when a horse is presented for competition with evidence of recent abuseāor in response to action taken by ācertain other entities such as the authorities or a court of law,ā USEF spokeswoman Vicki Lowell said in an email.
Today, however, the USEF officers released a proposed extraordinary rule change to GR838.1 that would strengthen the definition of abuse, expand the scope of the USEFās jurisdiction over abuse to include abuse that happens away from competitions, require that members report abuse to a licensed official at a competition or the USEF if not, and clarify that a licensed official or competition management can take swift action when abuse occurs at a competition. "
13 Likes
USEF could have used the Code of Conduct rule in this case. What else is it there for? Apparently, meaningless window dressing.
16 Likes
- Although only one event showed a significant judging bias, it should be noted that in dressage events the judges are placed in different positions around the ring and the judge effect is completely confounded with and mathematically indistinguishable from a āpositionā effect.
The statistical methodology is out there. You donāt need some new-fangled dressage analyticsā¦you need a good statistician and the will to utilize the tools that are already in use by many industries.
[/quote]
How does this prove that the statistical methodology is out there? Serious question, because "
the judges are placed in different positions around the ring and the judge effect is completely confounded with and mathematically indistinguishable from a āpositionā effect.
I didnāt necessarily support CP when this thread was originally started, but did comment about the OPās quiteā¦intense issues with him. For the first time ever on this, or frankly any internet site, I have gone back and deleted my comment from back in the day. I would not want my comment about the way OP was presenting herself to ever be interpreted as support for CP. I was only interested in making sure the process of determining his culpability was followed rather than railroading without due process.
And as horrific as the horse abuse seems to be, the human trafficking investigation may have more teeth legally speaking. As much as he should be held accountable for the animal abuse, so long as the end result is never working with horses again by whatever means is at least better than nothing.
19 Likes
Iām not sure I understand this question.
The statistical methodology to determine āinter-rater reliabilityā is definitely out there and used in the social sciences, in medicine, and in industry.
3 Likes
I hadnāt noticed how old the thread was and your post is thoughtful. What are the details on the trafficking? I know nothing. Itās interesting because Cols OH has a quite organized system working to help women out of trafficking and I attended mentor training. One of the speakers was a trauma specialist and she said we might leave training feeling trauma. And I felt it.
Couldnāt stop thinking about the whole situation and it lasted for days. The same here for the horses. Iāve myself witnessed horse abuse and walked away and then later haunted by it. Even since a recent post about seeing a horse at QH Congress tied down with a bit between its legs in the stall. I went to the stall office and reported it and they looked at me like I was crazy.
I kept thinking these last couple days why I didnāt call the Humane Society locally and how I failed that horse. Ugh. We all must NOT turn away. These animals need us and we must make change.
10 Likes
How the hell do they plan to enforce that? While itās a great goal, I can certainly see opportunities for abuse of the system from disgruntled clients, etc. I just donāt see how USEF can claim any sort of jurisdiction over what someone does at home - like Parra.
12 Likes
1 Like
no backbone
1 Like
I donāt understand why Christine Traurigās complaint did not have legs. She is the national young horse coach. They send her around the country for private sessions with those who are on the developing horse invitation list. Parra has had several horses (with his assistant trainers) in that program year after year. This doesnāt make sense to me.
Without a clear definition of what āabuseā means, their new rule has no merit. I believe that all that the private barns will do is prohibit filming of any activities on their property. They can certainly make someone sign an agreement to that effect or kick them off the property if they catch them filming. They will just simply shut arena doors when they do this stuff.
If āabuseā is clearly defined as drawing blood (i.e. producing evidence) then what happens to all the horses with their heads tied down between their knees? Is it ok to tie down a head as long as you donāt beat a horse at the same time? Is beating ok as long as it doesnāt leave welts? Is one welt ok but not three? Is a spur rub punishable or must blood be drawn? Must the spur wound be deeper than 1/4 inch to get you suspended? If blood is found on the horseās mouth, do we blame it on the horse biting itself or on the rider who caused that much tension? BLAH BLAH BLAH This whole response is absurd. Are they saying that they can only punish people at shows, but not at member paid and sanctioned events?
They are not going to police this. They canāt. This is lip service. What they need to do is take apart their flawed system that is encouraging poor training and rewarding it.
Parra has been seen hitting horses at shows for decades. Upon information and belief, the whip rule was instituted because of what he was doing at shows with whips. It apparently did not stop him because Robert Dover said he saw him whipping a horse unfairly.
Woodshedding a horse after a bad test happens all the time. I have seen trainers take the horse to the paddock area and run it around a small paddock. I have seen people ride a horse for long periods of time AFTER their test. I have never seen any TD do anything about it.
30 Likes
Im pretty sure that thereās no way to do this, but work through this hypothetical with me. What if competitors of a certain FEI level and trainers of competitors of a certain FEI level had to consent to have surveillance cameras in all training areas and that date stamped videos have to be made available to the organization on demand. If there is a significant gap in the videos, further investigation may be undertaken. Just a thought.
2 Likes
There is a video I watched recently of when they put up cameras at european indoor shows. It looked like they deliberately put the cameras up to cut off all four corners and both short ends of the arena. You could see the riders go into the corners and not come out for a long period of timeā¦ There would have to a requirement that the cameras cover the whole of the warm up arena.
2 Likes
My GUESS is that she spoke directly to him re whatever she saw, thus the phrase ācalled him outā. Maybe even mentioned it to others in the USEF org structure. But did not actually file a complaint.
Edited to add: apparently this was at the Young Horse Championships show, so there was a clear ability to file an actual complaint, or even to have issued a Yellow Card.
5 Likes
What complaint ? We donāt know if she made an official complaint and/or protest to USEF.
