The October 7 revision of the SWE-Game preprint tests coding agents on more than producing code that launches. Its 247 tasks come from 41 executable Godot reference games across 13 gameplay categories, with 24 projects in 2D and 17 in 3D.
The five workflows cover building from a short brief, implementing a supplied game design document, completing a project skeleton, repairing 83 injected faults and porting a Godot game to Unity. Evaluators replay inputs and inspect engine state, while separate visual rubrics judge how well the result communicates the intended game.
Construction still needs close supervision
Across the six tested model-and-agent configurations, the highest totals for the three construction modes were 50.38, 59.68 and 54.06 out of 100. The leading results were higher for bug repair at 83.46 and Godot-to-Unity porting at 72.40. Opus5 led each workflow, but the authors paired every model with its own agent framework and tools, so this is not a controlled model-only ranking or a guarantee for production projects.
Reviewed failures were dominated by missed requirements and gameplay-logic errors. For an indie team, that supports a practical workflow: give the agent explicit mechanics and acceptance criteria, then compile, run scripted interactions and play the result. A polished screenshot or successful launch is not proof that hazards, progression, saving or win conditions behave correctly.
Runtime evidence beats watching alone
On human-labelled behaviours from 100 agent-built games, the paper's executable checks reached 92.59% balanced accuracy, compared with 78.41% for a video-based vision-language judge. Visual scoring still mattered: its correlation with human ratings reached 0.829 across 200 clips.
The useful lesson is not to remove human review. Combine build tests, repeatable input sequences and state assertions with footage and hands-on playtesting, especially before accepting a large agent-authored change.
Original sources
Release plans and availability can change. Check the original source for the latest information.