Benchmarking coding agents that turn open-ended scene descriptions into engine-native 3D scenes in Unreal Engine. Measuring spatial reasoning, task fulfillment and physical validity.
Task fulfillment, artifact integrity and static physical validity.
0.2 × Detailed + 0.6 × Overview + 0.2 × Physical
+ Full task instructionTEXT-TO-SCENE
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
20 scenes in the Research Track: affordable, reproducible evaluation for academic and model-development use160 scenes in the Full Benchmark: comprehensive evaluation for leaderboard and final reporting
Old Industrial Pallet Bay · Image-to-Scene
Repair the corrupted scene using its reference image; preserve all unrelated actors.
Restore each target within 5 cm, 5° and 5%, without unintended edits.
0.8 × Repair F1 + 0.2 × Physical
+ Full task instructionIMAGE-TO-SCENE
Repair the current outdoor scene so it matches the provided reference image. Make only the minimum changes needed, and preserve all unrelated actors and properties.
160 editing cases in the full benchmark target, including private cases72 public editing cases (22 indoor, 50 outdoor)
Leaderboard
Leaderboard
-
Research Track · 20 scenes · 15 coding-agent configurations · re-run in October 2026.
Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.
-
Code4Scene Research TrackImage-to-Scene
20 scenes · score out of 10075 scenes · score out of 100
Swipe to view every configuration →
Sorted by score. Bar labels are scores ×100, rounded down; the table truncates to three decimal places. One color per model provider.
Sorted by score. Bar labels round scores ×100; the table preserves three decimal places. One color per model provider.
-
Code4Scene Pareto Frontier
Quality against the median cost of one case · Research Track, 20 scenes
Quality against the cost of one case · Image-to-Scene, 75 public cases
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the median USD per case. Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case.
-
15 configurations
Ranking
Research Track, 20 scenes, re-run in October 2026 with every agent in the same sandbox. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical; a case without a valid scene scores 0. Detailed, Overview and Physical are means of the per-case scores. Cost is the median per case. *Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.
#
Agent configuration
Score
Detailed
Overview
Physical
Median cost
1
Claude Sonnet 5.5 (max)ParetoAnthropic
0.837
0.803
0.812
0.945
$26.93
2
Claude Opus 5.5 (max)ParetoAnthropic
0.832
0.763
0.818
0.943
$26.90
3
Claude Fable 5.1 (max)ParetoAnthropic
0.800
0.746
0.756
0.983
$23.42
4
GPT-6 Astra (max)ParetoOpenAI
0.790
0.760
0.811
0.759
$14.25
5
GPT-6.1 Sol (max)ParetoOpenAI
0.784
0.742
0.778
0.845
$2.59
6
Gemini 4 Argon (high)Google
0.777
0.688
0.756
0.930
$4.83
7
GPT-5.6 Sol (max)OpenAI
0.723
0.640
0.713
0.836
$6.87
8
Claude Haiku 5.5 (max)ParetoAnthropic
0.718
0.652
0.667
0.937
$2.43
9
Muse Wonder Full 1.0 (max)Meta
0.715
0.666
0.696
0.819
$3.66*
10
Gemini 3.8 Flash (high)Google
0.694
0.578
0.662
0.905
$3.85
11
Grok 4.6 (xhigh)xAI
0.625
0.573
0.645
0.619
$7.41
12
Muse Spark 1.3 (max)ParetoMeta
0.548
0.517
0.547
0.582
$0.23
13
GLM-5.3 Flash (max)open weightsZ.ai
0.542
0.506
0.545
0.569
$0.94
14
Grok 4.7 (xhigh)xAI
0.541
0.521
0.546
0.549
$7.29
15
DeepSeek V4.1 Flash (max)open weightsDeepSeek
0.366
0.343
0.321
0.526
$0.35
75 public cases (25 indoor, 50 outdoor). Case score = 0.8 × Repair F1 + 0.2 × Physical. Repair F1 counts target actors restored to the withheld ground truth.
Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.
+
Code4Scene Research TrackImage-to-Scene
20 scenes · score out of 10075 scenes · score out of 100
Swipe to view every configuration →
Sorted by score. Bar labels are scores ×100, rounded down; the table truncates to three decimal places. One color per model provider.
Sorted by score. Bar labels round scores ×100; the table preserves three decimal places. One color per model provider.
+
Code4Scene Pareto Frontier
Quality against the median cost of one case · Research Track, 20 scenes
Quality against the cost of one case · Image-to-Scene, 75 public cases
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the median USD per case. Gemma 4 31B (thinking on) has no point: it ran on self-hosted weights, so nothing was billed. Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case.
+
19 configurations
Ranking
Research Track, 20 scenes, re-run in October 2026 with every agent in the same sandbox. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical; a case without a valid scene scores 0. Detailed, Overview and Physical are means of the per-case scores. Cost is the median per case. *Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.
#
Agent configuration
Score
Detailed
Overview
Physical
Median cost
1
Claude Sonnet 5.5 (max)ParetoAnthropic
0.837
0.803
0.812
0.945
$26.93
2
Claude Opus 5.5 (max)ParetoAnthropic
0.832
0.763
0.818
0.943
$26.90
3
Claude Fable 5.1 (max)ParetoAnthropic
0.800
0.746
0.756
0.983
$23.42
4
GPT-6 Astra (max)ParetoOpenAI
0.790
0.760
0.811
0.759
$14.25
5
GPT-6.1 Sol (max)ParetoOpenAI
0.784
0.742
0.778
0.845
$2.59
6
Gemini 4 Argon (high)Google
0.777
0.688
0.756
0.930
$4.83
7
GPT-5.6 Sol (max)OpenAI
0.723
0.640
0.713
0.836
$6.87
8
Claude Haiku 5.5 (max)ParetoAnthropic
0.718
0.652
0.667
0.937
$2.43
9
Muse Wonder Full 1.0 (max)Meta
0.715
0.666
0.696
0.819
$3.66*
10
Gemini 3.8 Flash (high)Google
0.694
0.578
0.662
0.905
$3.85
11
GPT-6 Luna (max)ParetoOpenAI
0.688
0.606
0.666
0.834
$0.16
12
Grok 4.6 (xhigh)xAI
0.625
0.573
0.645
0.619
$7.41
13
Muse Spark Open (max)open weightsMeta
0.611
0.583
0.591
0.700
$2.01
14
Muse Spark 1.3 (max)Meta
0.548
0.517
0.547
0.582
$0.23
15
GLM-5.3 Flash (max)open weightsZ.ai
0.542
0.506
0.545
0.569
$0.94
16
Grok 4.7 (xhigh)xAI
0.541
0.521
0.546
0.549
$7.29
17
MiMo V2.6 Pro (max)open weightsXiaomi
0.527
0.484
0.507
0.630
$0.92
18
Gemma 4 31B (thinking on)open weightsGoogle
0.517
0.448
0.469
0.731
–
19
DeepSeek V4.1 Flash (max)open weightsDeepSeek
0.366
0.343
0.321
0.526
$0.35
75 public cases (25 indoor, 50 outdoor). Case score = 0.8 × Repair F1 + 0.2 × Physical. Repair F1 counts target actors restored to the withheld ground truth.
Click a column to sort. Pareto: on the score-against-cost frontier.
Open the gallery: what each of the 15 configurations built on every scene, with its score and cost.
+
Click a column to sort. Pareto: on the score-against-cost frontier.
Open the gallery: what each of the 19 configurations built on every scene, with its score and cost.
Cases
Visualizing agent outputs
-
Four agents build the same scene: a time-lapse of each build, then an orbit of the result. Pick a scene, or open the gallery for all 15 configurations on all 20 scenes.
Drag any scene to inspect the saved 3D output. Switch each repair between the saved scene, its corrupted input and the ground truth, or open a case to explore every agent.
-
Swipe to see all four agents →
Text-to-Scene · re-run, October 2026
Blue City
Build me a compact Moroccan medina street inspired by Chefchaouen, with a tranquil, sunlit atmosphere and roughly fifteen to twenty tightly adjoining two- to four-story buildings arranged along both sides of a broad, straight road. Layer the irregular façades with recessed courtyards, narrow alleys, horseshoe-arched passages, exterior stair landings, projecting timber galleries, small balconies, and overlapping rooflines. Construct the road from worn, slightly reflective paving with broad shallow puddles, pale raised curbs, and narrow dark cobbled sidewalks. Finish the buildings in weathered cobalt and powder blue, white, pale turquoise, dusty coral, and peach stucco, frequently exposing dark stone masonry beneath chipped plaster. Use predominantly orange terracotta roof tiles, punctuated by pale-blue tiled pyramidal roofs and flat blue terraces. Make a tall blue tower house with a dark wooden room projecting below its roof the central landmark. Frame deep-set wooden doors and arched windows with alternating blue-and-white bands, then add flowering vines, tall palms, twin-lantern streetlights, and occasional wooden benches.
Each tile: the build as a time-lapse, then an orbit of the scene the agent saved. Chips: the case score ×100, rounded down.
+
Four agents build the same scene: a time-lapse of each build, then an orbit of the result. Pick a scene, or open the gallery for all 19 configurations on all 20 scenes.
Drag any scene to inspect the saved 3D output. Switch each repair between the saved scene, its corrupted input and the ground truth, or open a case to explore every agent.
+
Swipe to see all four agents →
Text-to-Scene · re-run, October 2026
Blue City
Build me a compact Moroccan medina street inspired by Chefchaouen, with a tranquil, sunlit atmosphere and roughly fifteen to twenty tightly adjoining two- to four-story buildings arranged along both sides of a broad, straight road. Layer the irregular façades with recessed courtyards, narrow alleys, horseshoe-arched passages, exterior stair landings, projecting timber galleries, small balconies, and overlapping rooflines. Construct the road from worn, slightly reflective paving with broad shallow puddles, pale raised curbs, and narrow dark cobbled sidewalks. Finish the buildings in weathered cobalt and powder blue, white, pale turquoise, dusty coral, and peach stucco, frequently exposing dark stone masonry beneath chipped plaster. Use predominantly orange terracotta roof tiles, punctuated by pale-blue tiled pyramidal roofs and flat blue terraces. Make a tall blue tower house with a dark wooden room projecting below its roof the central landmark. Frame deep-set wooden doors and arched windows with alternating blue-and-white bands, then add flowering vines, tall palms, twin-lantern streetlights, and occasional wooden benches.