This is an opinion piece. The facts come from the linked paper. The argument is mine.
When one AI model grades the answers of another, the score looks objective. It is a sensible shortcut, because human evaluation, in the paper discussed below, is described as slow and costly. My view is that the shortcut is fine, as long as readers and builders stop treating the AI grader as a neutral referee. It is a rater with habits, and the habits are known.
The clearest account of those habits is a 2023 paper, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, from researchers at UC Berkeley, UC San Diego, Carnegie Mellon, Stanford and MBZUAI. The authors call the approach "LLM-as-a-judge," and they are unusually frank about where it bends. The models it tested are from 2023, so I would not carry any single number over to a current model. I am using the paper for its pattern of failures, which I think still tells you what to check.
What the paper found in favor of AI judges
The case for AI judges is real, and the paper makes it. Its authors write that human evaluation is "exceptionally slow and costly," and in their abstract they call LLM-as-a-judge "a scalable and explainable way to approximate human preferences."
The headline result is strong. In the abstract, the authors report that a strong judge reached "over 80% agreement" with human preferences, "the same level of agreement between humans." The detail matters here. On the paper's MT-bench data, using only the votes that were not ties, GPT-4 agreed with human experts 85% of the time, while the experts agreed with each other 81% of the time. If you want to know whether an AI judge can be useful, that is the evidence.
What the paper found against treating a judge as neutral
The same paper lists the ways a judge can be wrong. Its abstract names "position, verbosity, and self-enhancement biases, as well as limited reasoning ability." Each one is worth knowing.
Position. The authors gave each judge two nearly identical answers and then swapped their order. Their finding: "we found all of them exhibit strong position bias." In their Table 2, with the default prompt, Claude-v1 gave consistent results in 23.8% of cases and favored the first answer 75.0% of the time. GPT-3.5 was consistent in 46.2% of cases. GPT-4 did best at 65.0%, and the paper says "Only GPT-4 outputs consistent results in more than 60% of cases." This was a challenging test, because the answers were "very similar and occasionally indistinguishable even to humans." The authors also say position bias is "less prominent in some cases." Still, a judge that flips its verdict when you swap the order has told you something about what it is measuring.
Length. The authors built a "repetitive list" attack on 23 answers that contained numbered lists. They had GPT-4 rephrase each list and put the rephrased items in front of the original items, so the answer was longer but held no new information. The attack succeeded against Claude-v1 and GPT-3.5 in 91.3% of cases and against GPT-4 in 8.7%. That is more than 10 times the failure rate for the weaker two. The paper defines verbosity bias as favoring "longer, verbose responses, even if they are not as clear, high-quality, or accurate as shorter alternatives."
Self-preference. The paper compared each judge's win rates with human win rates. In its words, "GPT-4 favors itself with a 10% higher win rate; Claude-v1 favors itself with a 25% higher win rate." It then adds that GPT-3.5 does not favor itself, and that "our study cannot determine whether the models exhibit a self-enhancement bias." I want to be fair to that hedge. The paper does not prove a judge prefers its own writing. It also does not rule it out.
Grading what it cannot do. The authors note that a judge can be "misled by the provided answers," even on a basic math problem it can solve when asked separately. On 10 math questions, GPT-4 as a judge with the default prompt said an incorrect answer was correct in 14 of 20 trials. With a reference answer supplied, that fell to 3 of 20.
What this means for how you read a score
My argument is simple. If a judge has these habits, then a judge's score is a measurement that depends on how the judging was set up. Two teams can run the same models past two AI judges and report different numbers without either of them being dishonest. Whoever publishes the judge prompt, the answer order and the check against human raters is giving you more than whoever publishes only the score.
This is the same concern we raised about public rankings in Opinion: A Leaderboard Rank Tells You Less About an AI Model Than You Think. There the problem was how scores get lifted by testing choices. Here the problem sits one layer down, in the grader itself. In both cases the number is a hint, not a verdict.
There is also a quieter reason to be careful. The paper says that the models it uses as judges "are often trained with RLHF," short for reinforcement learning with human feedback. We argued in Opinion: An AI Chatbot That Always Agrees With You Is a Design Choice that models learn from what people approve of, and people tend to approve of agreement. I do not know whether that carries over to judging, and the paper does not claim it. I raise it as a question worth testing, not as a finding.
Five habits I would want from anyone using an AI judge
This list is mine. It is not from the paper, and I have not tested it on a real project, though several items follow fixes the authors describe.
- Swap the order. The authors describe a "conservative approach": call the judge twice with the answers in opposite order, "only declare a win when an answer is preferred in both orders," and call it a tie if the results are inconsistent.
- Watch for length. If the winning answers are mostly the longer ones, treat the result with suspicion.
- Give the judge a reference answer for math and logic. In the paper's math test, a reference answer cut failures from 14 of 20 to 3 of 20.
- Keep the judge separate from the models it scores. The paper could not settle whether self-preference exists, so this is a precaution on my part, not a requirement the paper sets.
- Spot-check against people. The paper's own headline number comes from comparing judges with human votes, and I would not skip that step. Humans are not a perfect check either, which is the point we made in Opinion: Expert Sign-Off on AI Work Is No Longer Proof, but a sample of human reads shows you where the judge and people split.
A fix is not a guarantee. The authors note that few-shot examples raised GPT-4's consistency from 65.0% to 77.5%, then caution that "high consistency may not imply high accuracy." That is the right attitude for all of it.
Where I could be wrong
The paper's agreement numbers are good, and its biases are measured on 2023 models. Newer judges may be much steadier than the ones in Table 2, and I have not tested any. The paper also says it "emphasizes helpfulness but largely neglects safety," so its results should not be read as covering every kind of judging. It is possible that careful setup removes most of the problems I listed, and that the real risk is people skipping the setup.
If so, my conclusion still holds, because it is about the setup. An AI judge can be a useful instrument. It is a worse referee than its confident tone suggests, and a score is only as good as the checks behind it.
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al., 2023
