Subscribe
Topic / page 3

Benchmarks

Articles, page 3

41-51
  1. Autonomous Retail AI Agent Luna Fires Employee Following Context Retrieval Breakdown and Human Intervention

    In an empirical field deployment examining autonomous AI workforce management, research firm Andon Labs reported that its storefront manager agent, Luna, decided to fire a human retail employee after months of operational infractions. The incident, which unfolded at the Andon Market retail location in San Francisco, represents one of the first documented instances of an autonomous large language model agent managing physical store operations and executing a personnel termination decision. Luna,

    Agents
    3 min read
  2. Stealth Model 'Ox Alpha' Surfaces with 1M Multimodal Context and 100 Trillion Daily Token Allocation

    An anonymous frontier model named "Ox Alpha" has launched across OpenRouter and the OpenCode developer environment, offering free access backed by a reported 100 trillion tokens per day in serving capacity. The model features a 1,048,576-token context window with native multimodal support spanning text, image, and video inputs. The unannounced release follows a pattern of stealth evaluations across AI model aggregators, where frontier labs deploy pre-release checkpoints to collect blind compara

    LLM
    2 min read
  3. NVIDIA AVO Achieves 100% on ARC-AGI-3 with Claude Opus 5

    NVIDIA Research has published details on Agentic Variation Operators (AVO), a general-purpose autonomous agent architecture designed to sustain long-horizon engineering and reasoning workflows. In benchmark evaluations on ARC-AGI-3, an interactive reasoning suite that tests fluid intelligence without instructions or stated goals, AVO scored 100.00 Relative Human Action Efficiency (RHAE) across all 25 public-set environments and 183 levels using Anthropic's Claude Opus 5. The result highlights a

    AI
    3 min read
  4. Study Exposes Benchmark Overfitting and Acoustic Leakage in Speech Recognition Models

    A joint empirical study by researchers from Hugging Face and Hume AI has uncovered evidence that top-ranking automatic speech recognition (ASR) models frequently exhibit benchmark optimization, commonly referred to as "benchmaxxing." The findings demonstrate that models often achieve low word error rates on public leaderboards by memorizing test-set reference anomalies and responding to subtle acoustic channel signatures rather than generalizing audio transcription. Evaluating 11 open-source sp

    AI
    2 min read
  5. Vals AI Raises $40M Series A at $400M Valuation Led by a16z to Build Real-World AI Benchmarks

    San Francisco evaluation startup Vals AI announced a $40 million Series A funding round at a $400 million post-money valuation, led by Andreessen Horowitz. The round included participation from existing seed backers 8VC, Pear VC, and Bloomberg Beta, alongside new institutional investors HRT Ventures and Next Ladder Ventures. The financing brings total capital raised by the company to $45 million, following a $5 million seed round. Founded by Stanford computer science graduates Rayan Krishnan a

    AI
    2 min read
  6. Benchmarking LLM Inference in Production: Architecture, Metrics, and Tooling Across AIPerf, GuideLLM, and LLMPerf

    Traditional HTTP load-testing tools such as Apache Bench, wrk, and Locust evaluate systems using uniform request-response cycles. These tools send a payload, wait for the full response, and compute metrics such as requests per second (RPS) and round-trip latency percentiles. For stateless REST APIs, this model aligns directly with user experience. Large language model (LLM) serving fundamentally breaks this abstraction. An inference request is not an atomic computation. It consists of two disti

    AI
    7 min read
  7. GLM-5.3 Scores 60 on Artificial Analysis Intelligence Index, Matching Kimi K3

    Independent AI evaluation platform Artificial Analysis has published its benchmark results for Z.ai's GLM-5.3, awarding the reasoning model a score of 60 on its Intelligence Index v4.1.1. The result places GLM-5.3 level with Moonshot AI's Kimi K3 and three points behind frontier leader Claude Opus 5 (63). The evaluation tested GLM-5.3 at its maximum reasoning effort configuration across a nine-part battery that measures agentic tool execution, terminal coding, graduate-level scientific problem-

    AI
    2 min read
  8. AI Agent Evaluation in Production: Trajectory Benchmarks, Sandbox Harnesses, and Flakiness Mitigation

    Evaluating standard large language models relies on static input-output pairs: a fixed prompt produces a completion that an automated script compares against reference strings or grades with a calibrated judge. Autonomous AI agents break this paradigm completely. An agent executes a multi-step trajectory consisting of planning, tool invocation, environment state observation, error recovery, and variable-length decision loops. Evaluating an agent requires testing not just the final string output,

    Agents
    5 min read
  9. Artificial Analysis Launches Search Index Benchmark for AI Agent Search APIs

    Artificial Analysis has released the Search Index, a benchmark suite designed to evaluate web search APIs for autonomous AI agents across retrieval quality, query latency, and end-to-end task economics. The initial evaluation tests seven dedicated search providers: Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave. Benchmark Setup and Evaluation Methodology To isolate search API performance from model variance, the evaluation executes all tests with GPT-5.6 Luna inside Stirrup,

    Agents
    2 min read
  10. DeepSeek V4 Flash tops charts but fails half its real agent tasks

    DeepSeek's V4 Flash has become the most-used AI model on OpenRouter and one of the highest-rated open-weight models available. In real agent tests, though, it finished barely more than half of the jobs it was given. The gap between leaderboard and workplace is the story. What Composio found The integration company Composio ran V4 Flash through eight agent harnesses, including Claude Code, Codex, and OpenCode, on 30 deliberately hard multi-step tasks. The tasks used live tools: Gmail, GitHub,

    AI
    2 min read
  11. Zhipu's GLM-5.2 Narrows the Gap With Anthropic's Fable 5 on Coding Benchmarks

    Zhipu AI's open-weight GLM-5.2 model has placed second globally on the Code Arena coding benchmark, trailing only Anthropic's Claude Fable 5 and sitting within one point of Claude Opus 4.8 on the hardest agentic tasks. The result, recorded after the model's June 13 release under an MIT license, signals a fast-closing capability gap between Chinese open-weight systems and the leading U.S. frontier models. Benchmark Performance On Code Arena's front-end coding leaderboard, GLM-5.2 places second

    AI
    3 min read