A team from Beijing University of Posts and Telecommunications, Peking University, Jilin University, Zhongguancun Academy and the Kling Team has posted VibeVideoBench, a benchmark that asks AI coding agents to make a finished video with sound using only code. The paper, submitted to arXiv on October 26, 2026, tests nine agent systems from six labs. GPT-6 Astra and Claude Opus 5.5 come out first and second, but the more useful finding is about time: most missing videos were lost because the agent started its final render too late.
What the benchmark asks
Video models such as diffusion generators paint pixels. The authors argue that this approach cannot reliably show an exact formula, a correct chart, a game that follows its rules, or narration that lines up with what is on screen. A program can. People already make such videos by hand with tools like Manim or Remotion, and coding agents can now drive a renderer, a headless browser, a text-to-speech engine and ffmpeg on their own.
VibeVideoBench turns that into a test. It has 239 tasks in 12 categories, including data stories, music pieces, classroom lessons, product demos, games and research paper explainers. 209 tasks are in English and 30 in Chinese, and target lengths run from 8 seconds to 15 minutes. Each agent gets a written specification, often with input files such as data, fonts, sprites or 3D scans, and must deliver an MP4 within one hour inside a sandbox. No image or video generation model is available. The only generative tool is a text-to-speech command backed by Qwen3-TTS-Flash, so every frame has to be drawn by code.
Each model runs in its own vendor's harness (Codex CLI, Claude Code, Antigravity CLI and others) at the highest effort its provider accepts, with no follow-up turns. The authors say they evaluate the whole agent system, not the model alone. The nine systems produced 2,151 scored runs.
How the videos are graded
Grading has three layers:
- Program checks: code verifies resolution, frame rate, duration, an audible audio track and a delivered source project. There are 1,449 of these checks per system.
- Rubric checks: 12,985 pass or fail checks, about 54 per task, each tied to a line in the task. A "strict success" requires that no core check fails and at least one passes, and a missing video counts as a failure.
- Quality scores: five 1 to 5 ratings for visual quality, aesthetics, content logic, timing and audio.
Three separate panels of vision-language model judges score the rubric and quality layers. One panel sends audio checks only to a model that can hear. The team also ran a blind human preference study on all 239 tasks to check the judges.
The results
On strict success (Table 2, missing video counts as a failure), the nine systems scored:
- Claude Opus 5.5 (Claude Code): 76.3%
- GPT-6 Astra (Codex CLI): 74.5%
- Claude Sonnet 5.5 (Claude Code): 67.0%
- DeepSeek-V4.1-Flash (DeepSeek's own harness): 57.6%
- Gemini 4 Argon (Antigravity CLI): 55.2%
- Kimi K3 (Kimi Code): 46.3%
- GPT-5.6 Sol (Codex CLI): 39.0%
- GLM-5.3-Flash (ZCode): 33.5%
- Gemini 3.8 Flash (Antigravity CLI): 26.6%
On delivered videos only (also Table 2), GPT-6 Astra has the top overall quality score, 4.126 out of 5, and Claude Opus 5.5 is second at 4.074. Human annotators put them in the same order, with Elo ratings of 1242 and 1203. Opus is the only system that delivered all 239 videos. Astra delivered 238. GLM-5.3-Flash delivered 179 and Gemini 4 Argon 184.
The paper treats delivery as its own axis. Gemini 4 Argon's delivered videos score well, but it delivered only 77% of them, and when missing videos count as zero it falls from fourth to seventh place on overall score.
Why videos went missing
The authors read all 224 runs that ended with no playable video and assigned each one main cause (Figure 4a):
- 169 final renders started too late, ran too slowly or never started
- 45 renders stalled
- 7 runs destroyed their own output
- 3 had other causes
The paper calls this a failure of how agents plan their hour. Agents polished pieces and previews and left the full render for last. In GPT-5.6 Sol's failed runs, the final render started a median of 9 minutes before the deadline, often using frame-by-frame browser capture that needs 5 to 35 minutes. 65 runs never started a final render, 31 of them from GLM-5.3-Flash. All the stalled renders came from the two Gemini systems: in 33 of them, the agent's script piped frames into ffmpeg and never read its error stream, so the process deadlocked. In one self-destruct case, DeepSeek verified its video twice and then re-encoded it in place a minute before the deadline, leaving a 0.26 MB file.
Opus, the only system with no failed delivery, finished a median of 24 minutes before the deadline. The paper says it first made a complete video, then improved only copies.
How agents make video
Agents mostly built their own pipelines. ffmpeg appeared in 96 to 100% of runs, while Manim appeared in 8 of 2,151 runs and MoviePy in 13. Each system had a preferred stack: Claude agents and Gemini 4 Argon most often drew vector frames with cairo (42 to 54% of runs), GPT-5.6 Sol, Kimi and GLM drew frames with PIL and numpy (43 to 55%), and Astra, DeepSeek and Gemini 3.8 Flash captured browser canvases and three.js scenes (52 to 60%).
95% of runs looked at their own rendered frames, but the amount of looking did not predict quality. Astra looked at a median of 3 images per run and scored best on delivered quality; DeepSeek looked at a median of 35. Final reports were not always accurate: among Astra's playable videos that failed the spec, 10 of 14 reports still claimed the video was verified.
Mean list-price API cost per task ran from $0.32 for GLM-5.3-Flash to $15.38 for Claude Opus 5.5. The paper puts three systems on the cost and score frontier: GLM-5.3-Flash, DeepSeek-V4.1-Flash at $0.47 and GPT-6 Astra at $8.19.
Why it matters
For anyone who asks an agent for a lesson video or a data story, the paper gives two practical lessons. Pick a system that delivers reliably, not only one whose best output looks good. And tell the agent to render a complete draft early, because the hour usually runs out on the render, not on the ideas.
There is also a warning about AI judges. The three-panel mean ranked systems the way humans did, but it agreed with humans on only 73.3% of individual video pairs (Table 3). The authors conclude that judge scores can rank systems over hundreds of tasks but should not be trusted alone to grade a single video or to serve as a per-video training reward.
