Subscribe
AI
4 min read

OpenAI MedChemBench tests whether models can reason like medicinal chemists

OpenAI's new benchmark reveals frontier models excel at reading molecules but struggle with compound selection decisions in drug discovery.

Diagram illustrating OpenAI MedChemBench evaluating GPT models on medicinal chemistry tasks including molecular structure and assay data analysis
Diagram illustrating OpenAI MedChemBench evaluating GPT models on medicinal chemistry tasks including molecular structure and assay data analysis

OpenAI published MedChemBench, a benchmark consisting of 99 problems from 117 sources that tests if large language models can perform medicinal chemist reasoning. While top models like GPT-6 Astra achieve high scores in molecular structure reconstruction (95.4%), they remain mediocre at critical decision-making tasks like compound selection.

OpenAI MedChemBench illustration

OpenAI published MedChemBench on Friday, a benchmark that asks whether large language models can do the day-to-day reasoning of a medicinal chemist: read a molecular structure, interpret assay data, and decide which compound to synthesize next.

The short version of the results: the frontier models are now good at knowing what a molecule is, and still mediocre at knowing what to do with it.

99 problems, four skills

MedChemBench contains 99 problems drawn from 117 published papers and patents, covering 761 distinct molecular structures. Most problems place the model at a chosen decision point inside a real research program. It receives the biological target, the stated goals, the compounds studied so far and selected experimental findings. It must then work forward.

Scoring covers four components:

  • Molecular structures - reassembling the correct molecule, with correct atom connections and stereochemistry, from drawings and chemical groups.
  • Scientific reasoning - explanations grounded in the supplied evidence, such as why one compound was chosen over another.
  • Ranking and selection - ordering and picking compounds that meet stated requirements.
  • Property predictions - matching the numerical values and categories reported in the source work.

OpenAI ran eleven of its own GPT models across reasoning-effort settings. At the xhigh setting, GPT-6.2 Astra earned 84.5% of available credit, GPT-6 Astra 83.3%, GPT-6.1 Sol 82.0%, and GPT-6.1 Terra 75.1%. GPT-5.6 Sol, an older generation, managed 58.2%. Higher reasoning effort helped, but the gaps between models stayed large.

Where the models pass and where they fail

The component scores split cleanly. On structure reconstruction, GPT-6 Astra took 95.4% of available credit at xhigh. In one worked example, a task requiring the model to assemble a molecule while preserving an 18-atom ring, GPT-6 Astra produced the correct compound in all ten responses. GPT-5.6 Sol built a 15-atom ring, and GPT-5.6 Luna split the molecule into two disconnected fragments, failing all ten times.

Those are not cosmetic errors. Get the ring wrong and the model is analyzing a different chemical entity than the one in the paper.

Compound selection and numerical property prediction, the parts closest to actual bench decisions, landed at roughly half the available credit. OpenAI illustrates this with a PRMT5 inhibitor problem, a protein target in cancer research: given earlier structures and assay results, the model must predict cell potency, membrane permeability and liver-enzyme breakdown for three related compounds. Later experiments, withheld from the model, showed the classic tradeoff - compound 23 was more potent in cells, while compound 24 permeated better and metabolized slower. Predicting all three properties at once is where current models give up the most credit.

The caveats OpenAI flags itself

OpenAI states plainly that the benchmark does not show models can discover drugs or improve laboratory decisions. Three limits matter for anyone citing the headline scores. Prior exposure to the published sources cannot be ruled out - the problems come from real papers and patents, and those papers are on the internet. Each response was a single attempt, with no tools, no feedback and no new experiments. And the benchmark scores reasoning over historical evidence, not prospective chemistry.

There is also a first-party effect worth noting: the models that top the chart are the publisher's own, and the strongest configurations are internal ones (Astra variants) not generally described here before. An independent lab measuring the same tasks with open scaffolds would firm up the comparison.

Why it matters anyway

Medicinal chemistry is the expensive middle of drug discovery: iterate, synthesize, assay, repeat. If models can compress even part of that loop, the savings are measured in wet-lab months. MedChemBench's usefulness is less the leaderboard than the shape of the results - near-perfect molecule reading, coin-flip decision making. That is a concrete target list for the next round of science models, and a warning that 84.5% aggregate credit should not be read as 84.5% of a medicinal chemist.

Sources

Frequently asked questions

What is the highest score achieved on MedChemBench?

At the xhigh reasoning setting, GPT-6 Astra earned 84.5% of available credit, while GPT-6 Astra scored 83.3%, GPT-6.1 Sol 82.0%, and GPT-6.1 Terra 75.1%.

What are the four components of scoring in MedChemBench?

Scoring covers Molecular structures, Scientific reasoning, Ranking and selection, and Property predictions based on evidence from published papers and patents.

How did older models compare to GPT-6 in the benchmark?

The older generation GPT-5.6 Sol managed only 58.2% credit and struggled significantly more with structural integrity, such as building incorrect ring sizes or splitting fragments.

Can MedChemBench results prove AI can replace medicinal chemists?

No, OpenAI states the benchmark does not show models can discover drugs or improve laboratory decisions due to limitations like single attempts and lack of prospective chemistry evaluation.

AILLMEvalsDrug Discovery