Subscribe
Topic

Evaluation

How to evaluate LLMs and AI agents, and how to trust the results. Posts compare evaluation harnesses like LM Evaluation Harness and Lighteval, frameworks like DeepEval, Ragas and Promptfoo, and agent benchmarks such as SWE-bench and GAIA.

Articles

1-20
  1. Opinion: An AI Judge Is Not a Neutral Referee

    A 2023 paper found AI judges can match human raters, and also that they flip verdicts when answer order changes and can be fooled by length. My view: treat an AI judge as a rater with habits, and check them.

    Opinion
    6 min read
  2. Google Shares Framework to Test Health AI

    Google published a framework that tests health AI for accuracy, safety and usefulness with benchmarks, autoraters and expert review. Here is how it works.

    News
    6 min read