Subscribe
Topic

MLOps

MLOps for LLM applications: testing, evaluation and training pipelines. Posts compare evaluation frameworks like Ragas, DeepEval, Promptfoo and Inspect, and post-training RL frameworks like verl, OpenRLHF, TRL and DeepSpeed-Chat.

Articles

1-2
  1. LLM Evaluation Frameworks in Production: Comparing Promptfoo, DeepEval, Ragas, and Inspect Architecture, Metric Calibration, and Quality Gate Economics

    Testing large language model applications in production requires shifting from deterministic software unit tests to probabilistic evaluation harnesses. Traditional software engineering relies on binary assertions (assert output == expected), but generative models exhibit non-deterministic outputs, variable token distributions, and nuanced semantic drift across prompt revisions, model updates, and temperature configurations. To prevent regressions and quantify system capabilities before deployme

    Evaluation
    7 min read