The AI Raters That Scored Like Humans and Judged Nothing Like Them

4 min read ▶ Listen to this post

LLMs are increasingly used to rate subjective content at scale — how hateful a comment is, how helpful a response is, how a survey answer should be coded. The standard way to validate this is to check whether the AI's scores correlate with a trusted human-labeled set. If the correlation is strong, the rater is considered good enough to deploy.

In a 2026 study, Pratik Sachdeva (UC Berkeley) and Nathan Boudol (Grenoble INP) ran exactly that check, then kept going. They used the Measuring Hate Speech corpus — 50,070 comments rated by 11,143 human annotators on a ten-item instrument, a dataset built using Rasch measurement theory specifically so it could serve as a trustworthy yardstick — and had nine LLMs (including Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, and Grok 4.1) rate the same comments. The overall correlation between the LLMs' hate-speech scores and the human scale: a Spearman correlation of 0.911. On a standard validation check, that's a pass. Then the researchers used Rasch measurement models to break each rating down into its separate components — rater severity, item-by-item calibration, sensitivity to question order, sensitivity to who a comment was about — instead of collapsing everything into one number. Underneath that 0.911, the two scales disagreed almost everywhere.

The mechanism: a strong correlation can be built from several separate wrongs

All nine LLMs were more trigger-happy than the median human rater overall — quicker to flag content as showing some feature of hate speech. But on the specific items describing the worst content — dehumanization, incitement to violence, calls for genocide — the same models flipped and became more cautious than expected, requiring stronger evidence before escalating a rating. The two scales lined up well at the low-severity end of the construct and pulled apart sharply at the high end — exactly the region where getting it right matters most.

The ratings weren't stable, either. Simply reversing the order of the survey questions in the prompt — asking about genocide first instead of last — shifted every model's threshold, significantly so for the four lowest-capability models in the study; for one of them, the reversal cut roughly in half the odds that it would escalate its rating on the exact same comment. And the models weren't consistent across who a comment targeted: comments about Muslims and Jewish people were rated with a lower hate threshold across nearly every model (quicker to flag), while comments about gay people were rated with a higher one (slower to flag) — 82 of 126 model-by-identity-group combinations were statistically significant. None of this shows up in an overall correlation number, because an overall correlation is an average, and an average is exactly what a stable rater and an inconsistent one landing in the same place by coincidence look identical on.

Where else this shows up

• Academic and grant-funded research. A paper reporting a strong correlation between an AI-assisted coder and a human coder is reporting one number for what is actually several separate questions — severity, item-by-item agreement, stability — any of which can diverge while the correlation still looks fine.
• Non-profit program evaluation. An LLM used to code open-ended survey responses can validate well against a human-coded sample on average while still treating some subgroups, or some response categories, in a systematically different way than a human coder would.
• AI and LLM evaluation. An LLM-as-judge pipeline can report high agreement with human preference labels while being unstable to how a rubric is ordered, or inconsistent depending on who or what is named in the content it's judging — the exact failure modes this study went looking for and found.

What to ask for instead

Don't stop at "what's the correlation with human labels." Ask: has the rater's severity, item-level calibration, and stability to prompt order and subject matter been checked separately, or only averaged into one number? A 0.911 correlation and a rater that's diverging sharply on the cases that matter most can be the same number.

If you're relying on an LLM to rate, judge, or code anything consequential, talk to us before you take a single agreement statistic as proof it's working.

Related reading

• How to Evaluate an LLM Beyond Average Accuracy — the broader case for checking subgroup and stability diagnostics instead of trusting one aggregate metric, of which this study's rater decomposition is one rigorous example.
• The Essay That Said Nothing and Scored a Perfect 6 — a different way an automated scoring system can look reliable on the metric you're checking while missing what you actually care about.

Sources and further reading

• Sachdeva, P., & Boudol, N. (2026). Rating the Raters: Rasch Measurement Theory for LLM Evaluation. arXiv preprint arXiv:2608.27463.
• Kennedy, C. J., Bacon, G., Sahn, A., & von Vacano, C. (2020). Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application. arXiv preprint arXiv:2009.10277.

One email whenever we publish something new. No spam, unsubscribe anytime.