Subscribe
6 min read

The Grading Industry: How Benchmarking AI Became a Business

Arena now carries a $3.1 billion valuation, Artificial Analysis moved the majority of its index to private datasets, and Epoch is grading the graders. Measurement of AI models is a real industry now, with real conflicts.

The Grading Industry: How Benchmarking AI Became a Business

In late 2023, Chatbot Arena was a research project at UC Berkeley. Users typed a question, two anonymous models answered, the user picked a winner, and an Elo rating moved. It was a way to sidestep contaminated test sets and marketing claims, and it cost almost nothing to run.

In October 2026, the company built on that project, Arena (it was LMArena until a January 2026 rebrand), closed a $200 million Series B at a $3.1 billion valuation. The round was co-led by Lightspeed and Khosla Ventures, with Salesforce Ventures and Dell Technologies Capital joining. The company says it exceeded $100 million in annualized revenue, up from a $30 million run rate when it raised its Series A in January. Grading AI models is no longer a public service. It is an industry, and the industry now has funding rounds, subscription tiers, and the same conflicts of interest as every other business that audits things.

Anonymous voter figures weigh two blank ballots on a balance scale

Why the demand is real

Three forces turned independent evaluation from a nice-to-have into a market.

Saturation. Stanford's 2026 AI Index Report notes that frontier models gained 30 percentage points in a single year on Humanity's Last Exam, a benchmark built to stay hard for AI. When six or eight frontier models score within one percentage point of each other on a test designed for a world where 75 to 85 percent was a capability shift, the test stops being a decision tool. The same Stanford report found the top four labs clustered within 25 Elo points of each other on the Arena leaderboard as of March 2026. Buyers cannot choose between products using numbers that no longer separate them, so the evaluation firms keep being paid to build harder tests.

Distrust of lab-reported numbers. In April 2025, The Verge reported that Meta had "got caught gaming AI benchmarks": Llama 4 Maverick posted a strong ELO of 1417 and a number-two Arena ranking from a crafted submission, while the build that shipped to the public performed noticeably worse. Meta denied training on test sets, and the incident became the standard reference point for why vendor-run numbers belong in the screening pile, not the business case.

Measurement itself is under audit. Stanford's report puts error rates on some widely used evaluations as high as 42 percent. Epoch AI, a nonprofit, now runs a Benchmark Reviews program that inspects a benchmark's tasks, scorers, harness, and model settings before certifying it. Across 21 published reviews, 10 benchmarks are rated Flawed, including SWE-bench Verified, Humanity's Last Exam, and Terminal-Bench 4.0. The message to buyers is that a leaderboard score, even an independent one, is a claim about a particular version run at particular settings, not a fact.

Three ways to monetize a scoreboard

Arena sells data generated by the public and consumed by the labs. The free platform is the data engine: the company says the platform has logged 350 million sessions and 62 million votes from tens of millions of monthly visitors in more than 150 countries. The paid product, an AI Evaluations service launched in September 2025, sells analyses of that preference data to model providers and enterprises. Arena works with OpenAI, Google, and xAI as paying customers, and has open-sourced 375,000 data points including its leaderboard methodology. The pitch and the problem are the same: labs want the preferences of millions of real users, and the labs are also the customers. Contrary Research's company analysis puts it bluntly: the easiest path to profitability is selling evaluation tools and premium services to the same labs whose models Arena ranks.

Artificial Analysis sells independence, and increasingly sells privacy. The project started in a Sydney basement in 2024 by George Cameron and Micah Hill-Smith, took seed funding from Nat Friedman and Daniel Gross through the AI Grant program, and incorporated into a company in early 2026. Its Intelligence Index became one of the most-cited composite scores in the industry. In November 2025 it overhauled the index, dropping the staples labs loved to quote (MMLU-Pro, AIME 2025, LiveCodeBench) in favor of economically framed tests: GDPval-AA, built on OpenAI's dataset of real work products across 44 occupations, CritPt research physics (the leading model scored 11.5 percent), and AA-Omniscience, which penalizes guessing instead of abstaining. The November 2026 Index v5 goes further: for the first time, Artificial Analysis says, the majority of the index is built on private datasets no model has ever seen. Revenue, per third-party buyer's guides, mixes free leaderboards with a Pro subscription reported at around $417 per seat per month and private benchmarking sold to labs.

An auditor with a magnifying glass examines a vault of padlocked boxes

Epoch AI stays nonprofit and spends its credibility on audits and open data. Its Benchmarking Hub publishes results under a permissive license, its Epoch Capabilities Index fits a Bayesian model over many benchmark scores (top value 173, set by Claude Fable 5.2 in November 2026, on pre-launch testing), and it builds genuinely hard tests: FrontierMath Erdős hands AI systems 68 unsolved problems posed by Paul Erdős, and the best model has solved three. Epoch funds research rather than selling rankings, which is precisely why its Flawed verdicts carry weight, including on benchmarks other outlets quote uncritically.

The structural problem nobody has priced in

Every business model here has the same fault line. The referee needs customers, and only two things in AI have money: the labs and the enterprises buying from the labs. Sell to labs and you rank your paymasters in public. Sell private test sets to enterprises and the public cannot verify your numbers. Retire famous benchmarks to stay ahead of contamination and you burn the comparability that made you cited. Artificial Analysis's v5 pivot to private data is the cleanest example: it trades the one thing a scoreboard sells, auditability, for the one thing a scoreboard cannot keep from models, secrecy.

The 2026 equilibrium is a stack of partial trust: Arena for what humans prefer, Artificial Analysis for what models can do on private work-like tests, Epoch for what benchmarks are actually worth reading, and labs' own system cards for what labs claim. None of these numbers transfer cleanly to your workload. Which is why the enterprise buyer's guides that emerged this year all converge on the same advice: treat third-party scores as screening, then run 50 or more tasks from your own failure cases at the configuration you will actually pay for.

The bottom line

Independent evaluation grew up in three years from a Berkeley spreadsheet to a $3.1 billion valuation because model buyers needed a neutral party and labs could not provide one. It stayed neutral only as long as it stayed small and public. The money arrived, the tests went private, and the graders started charging for the answer key. The machines have not changed. The grading industry now sells what a benchmark never could: defensible cover for a procurement decision.


Sources