Plurai Launches Agent Simulation, Raises $13M Seed
Plurai opened its agent simulation tool to all developers with a free tier, run from Claude Code, Codex or Copilot CLI, and raised a $13M seed led by Team8.

How to evaluate LLMs and AI agents, and how to trust the results. Posts compare evaluation harnesses like LM Evaluation Harness and Lighteval, frameworks like DeepEval, Ragas and Promptfoo, and agent benchmarks such as SWE-bench and GAIA.
Plurai opened its agent simulation tool to all developers with a free tier, run from Claude Code, Codex or Copilot CLI, and raised a $13M seed led by Team8.

Most AI benchmarks give no credit for "I don't know" and full credit for a lucky guess, an OpenAI paper argues. My view: fix the scoring and check error rates.

A 2023 NeurIPS award paper found many sudden AI skill jumps fade under partial-credit scoring. My view: ask how a jump was scored before calling it new.

Garry Tan tested 7 AI coding harness setups: 71 to 79% pass rates, $0.17 to $0.44 per solved task. X replies say the tasks were too easy and name missing tools.

A viral X post said Claude Opus 5.5's reasoning changed on November 3. Theo's data points to a new thinking summarizer, and HN and X are still split.

OpenAI's BioBasicsBench tests AI agents on 100 routine wet-lab tasks. GPT-6 Astra earned 66.7% of credit at xhigh, versus 43.8% for GPT-5.6 Sol.

A 2023 paper found AI judges can match human raters, and also that they flip verdicts when answer order changes and can be fooled by length. My view: treat an AI judge as a rater with habits, and check them.

A context window sets how much text a model can take in. Three studies suggest that is not the same as using it well. My view: test your own task and send less.

Google published a framework that tests health AI for accuracy, safety and usefulness with benchmarks, autoraters and expert review. Here is how it works.

Cognition says SWE-2 Flash runs at 3x the speed and one-fifth the price of SWE-2 in Devin Desktop and CLI, with weights not public yet.

MMLU, GPQA, SWE-bench and Chatbot Arena: a plain-English guide to what AI benchmarks measure and why saturation, errors and contamination can mislead.

Factory's Droid 1.0 lets models call tools from a short program; Factory reports higher scores than Codex and Claude Code in its Terminal-Bench 4 tests.

A study of 11 agent memory systems found many score below plain RAG over the raw chat history. Why it happens, and the cheap fix the authors tested.

API-TAIL checks from returned text alone whether an LLM API swaps in a cheaper model. It caught most simulated 10% swaps at 200 requests in the paper's tests.

A CLEAR study of 19 model families finds a model's costliest language can need up to 17 times the tokens of its cheapest for the same text. Google's Gemini family earns a B.

A Télécom Paris and Nokia Bell Labs study of 11 open models finds constrained decoding matches or beats free decoding when prompts and formats match.

IBM and Kaggle's ScarfBench leaderboard tests coding agents on Java migrations. Gemini 4 Argon with Antigravity leads; about 72% of failures show up at runtime.

Why do chatbots invent facts? OpenAI says tests reward guessing, Anthropic found a default "can't answer" circuit that misfires, and a Stanford and Yale study found RAG legal tools still hallucinate 17% to 33% of the time. What causes hallucinations and how to catch them.

Into the Unknown, a new benchmark from CMU and DOE lab scientists, asked nine frontier agents to write 135 follow-up papers. None passed all three gates, and 114 failed integrity, mostly for claiming work never done or numbers their logs contradict.

whatmodel.sh matched 611 lab and independent scores across 180 models and 103 tests. Labs average 6.4% higher, but the median gap is 1.2%. SWE-bench and MBPP show the biggest gaps; MMLU and HumanEval barely move.
