diff --git "a/index.html" "b/index.html" --- "a/index.html" +++ "b/index.html" @@ -22,16 +22,16 @@ svg.ch .dim{opacity:.42}svg.ch g.row:hover{opacity:1}svg.ch .hot{fill:var(--c4s-

Constructing
3D Scenes with Code

Benchmarking coding agents that turn open-ended scene descriptions into engine-native 3D scenes in Unreal Engine. Measuring spatial reasoning, task fulfillment and physical validity.

Task Format

Nordic Harbour · Text-to-Scene

An open-ended scene description, an asset catalog and the Unreal Engine editor.

Agent inputText + assets
Task instruction

Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.

Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.

Given: an empty level and the content pack’s asset catalog.

EvaluationEngine-native scene
CASE SCORE0.877
Detailed Alignment
0.757
Overview Alignment
0.909
Physical Safety
0.900

Task fulfillment, artifact integrity and static physical validity.

0.2 × Detailed + 0.6 × Overview + 0.2 × Physical

+ Full task instructionTEXT-TO-SCENE

Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.

20 scenes in the Research Track: affordable, reproducible evaluation for academic and model-development use160 scenes in the Full Benchmark: comprehensive evaluation for leaderboard and final reporting

Leaderboard

-

Research Track · 20 scenes · 15 coding-agent configurations · re-run in October 2026.

Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.

-

Code4Scene Research Track

20 scenes · score out of 10075 scenes · score out of 100

Swipe to view every configuration →

Sorted by score. Bar labels are scores ×100, rounded down; the table truncates to three decimal places. One color per model provider.

-

Code4Scene Pareto Frontier

Quality against the median cost of one case · Research Track, 20 scenes

Swipe to explore all configurations →

Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the median USD per case. Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.

-
15 configurations
Ranking

Research Track, 20 scenes, re-run in October 2026 with every agent in the same sandbox. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical; a case without a valid scene scores 0. Detailed, Overview and Physical are means of the per-case scores. Cost is the median per case. *Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.

#Agent configurationScoreDetailedOverviewPhysicalMedian cost
1Claude Sonnet 5.5 (max)ParetoAnthropic
0.837
0.8030.8120.945$26.93
2Claude Opus 5.5 (max)ParetoAnthropic
0.832
0.7630.8180.943$26.90
3Claude Fable 5.1 (max)ParetoAnthropic
0.800
0.7460.7560.983$23.42
4GPT-6 Astra (max)ParetoOpenAI
0.790
0.7600.8110.759$14.25
5GPT-6.1 Sol (max)ParetoOpenAI
0.784
0.7420.7780.845$2.59
6Gemini 4 Argon (high)Google
0.777
0.6880.7560.930$4.83
7GPT-5.6 Sol (max)OpenAI
0.723
0.6400.7130.836$6.87
8Claude Haiku 5.5 (max)ParetoAnthropic
0.718
0.6520.6670.937$2.43
9Muse Wonder Full 1.0 (max)Meta
0.715
0.6660.6960.819$3.66*
10Gemini 3.8 Flash (high)Google
0.694
0.5780.6620.905$3.85
11Grok 4.6 (xhigh)xAI
0.625
0.5730.6450.619$7.41
12Muse Spark 1.3 (max)ParetoMeta
0.548
0.5170.5470.582$0.23
13GLM-5.3 Flash (max)open weightsZ.ai
0.542
0.5060.5450.569$0.94
14Grok 4.7 (xhigh)xAI
0.541
0.5210.5460.549$7.29
15DeepSeek V4.1 Flash (max)open weightsDeepSeek
0.366
0.3430.3210.526$0.35

75 public cases (25 indoor, 50 outdoor). Case score = 0.8 × Repair F1 + 0.2 × Physical. Repair F1 counts target actors restored to the withheld ground truth.

#Agent configurationScoreRepair F1PhysicalIndoorOutdoorCost / case
1Gemini 3.8 Flash (high)ParetoGoogle
0.581
0.5270.7960.7170.513$1.35
2GPT-6 Astra (max)OpenAI
0.515
0.4450.7960.7850.380$3.32
3Claude Opus 5 (max)Anthropic
0.468
0.3890.7860.6180.393$4.46
4Claude Fable 5.1 (max)Anthropic
0.424
0.3320.7950.5350.369$3.09
5GPT-5.6 Sol (high)OpenAI
0.393
0.3190.6890.4970.341$1.88
6Muse Spark 1.3 (medium)ParetoMeta
0.358
0.2460.8070.6330.221$0.07
7GLM-5.3 Flash (max)open weightsZ.ai
0.320
0.1880.8480.4650.247$0.09
8Qwen 3.8 27B (thinking on)open weightsAlibaba
0.226
0.1180.6600.2970.191$0.26
9DeepSeek V4.1 Flash (high)open weightsParetoDeepSeek
0.210
0.0660.7870.1890.221$0.05
10Qwen 3.8 27B (thinking off)open weightsAlibaba
0.200
0.0830.6670.2660.166$0.28
11