diff --git "a/index.html" "b/index.html" --- "a/index.html" +++ "b/index.html" @@ -24,8 +24,8 @@ svg.ch .dim{opacity:.42}svg.ch g.row:hover{opacity:1}svg.ch .hot{fill:var(--c4s-
Leaderboard

Leaderboard

Research Track · 20 scenes · 19 coding-agent configurations · re-run in October 2026.

Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.

Code4Scene Research Track

20 scenes · score out of 10075 scenes · score out of 100

Swipe to view every configuration →

Sorted by score. Bar labels are scores ×100, rounded down; the table truncates to three decimal places. One color per model provider.

-

Code4Scene Pareto Frontier

Quality against the median cost of one case · Research Track, 20 scenes

Swipe to explore all configurations →

Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the median USD per case. Gemma 4 31B (thinking on) has no point: it ran on self-hosted weights, so nothing was billed. Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.

-
19 configurations
Ranking

Research Track, 20 scenes, re-run in October 2026 with every agent in the same sandbox. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical; a case without a valid scene scores 0. Detailed, Overview and Physical are means of the per-case scores. Cost is the median per case. *Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.

#Agent configurationScoreDetailedOverviewPhysicalMedian cost
1Claude Sonnet 5.5 (max)ParetoAnthropic
0.837
0.8030.8120.945$26.93
2Claude Opus 5.5 (max)ParetoAnthropic
0.832
0.7630.8180.943$26.90
3Claude Fable 5.1 (max)ParetoAnthropic
0.800
0.7460.7560.983$23.42
4GPT-6 Astra (max)ParetoOpenAI
0.790
0.7600.8110.759$14.25
5GPT-6.1 Sol (max)ParetoOpenAI
0.784
0.7420.7780.845$2.59
6Gemini 4 Argon (high)Google
0.777
0.6880.7560.930$4.83
7GPT-5.6 Sol (max)OpenAI
0.723
0.6400.7130.836$6.87
8Claude Haiku 5.5 (max)ParetoAnthropic
0.718
0.6520.6670.937$2.43
9Muse Wonder Full 1.0 (max)Meta
0.715
0.6660.6960.819$3.66*
10Gemini 3.8 Flash (high)Google
0.694
0.5780.6620.905$3.85
11GPT-6 Luna (max)ParetoOpenAI
0.688
0.6060.6660.834$0.16
12Grok 4.6 (xhigh)xAI
0.625
0.5730.6450.619$7.41
13Muse Spark Open (max)open weightsMeta
0.611
0.5830.5910.700$2.01
14Muse Spark 1.3 (max)Meta
0.548
0.5170.5470.582$0.23
15GLM-5.3 Flash (max)open weightsZ.ai
0.542
0.5060.5450.569$0.94
16Grok 4.7 (xhigh)xAI
0.541
0.5210.5460.549$7.29
17MiMo V2.6 Pro (max)open weightsXiaomi
0.527
0.4840.5070.630$0.92
18Gemma 4 31B (thinking on)open weightsGoogle
0.517
0.4480.4690.731–
19DeepSeek V4.1 Flash (max)open weightsDeepSeek
0.366
0.3430.3210.526$0.35

75 public cases (25 indoor, 50 outdoor). Case score = 0.8 × Repair F1 + 0.2 × Physical. Repair F1 counts target actors restored to the withheld ground truth.

#Agent configurationScoreRepair F1PhysicalIndoorOutdoorCost / case
1Gemini 3.8 Flash (high)ParetoGoogle
0.581
0.5270.7960.7170.513$1.35
2GPT-6 Astra (max)OpenAI
0.515
0.4450.7960.7850.380$3.32
3Claude Opus 5 (max)Anthropic
0.468
0.3890.7860.6180.393$4.46
4Claude Fable 5.1 (max)Anthropic
0.424
0.3320.7950.5350.369$3.09
5GPT-5.6 Sol (high)OpenAI
0.393
0.3190.6890.4970.341$1.88
6Muse Spark 1.3 (medium)ParetoMeta
0.358
0.2460.8070.6330.221$0.07
7GLM-5.3 Flash (max)open weightsZ.ai
0.320
0.1880.8480.4650.247$0.09
8Qwen 3.8 27B (thinking on)open weightsAlibaba
0.226
0.1180.6600.2970.191$0.26
9DeepSeek V4.1 Flash (high)open weightsParetoDeepSeek
0.210
0.0660.7870.1890.221$0.05
10Qwen 3.8 27B (thinking off)open weightsAlibaba
0.200
0.0830.6670.2660.166$0.28
11