diff --git "a/index.html" "b/index.html" --- "a/index.html" +++ "b/index.html" @@ -24,8 +24,8 @@ svg.ch .dim{opacity:.42}svg.ch g.row:hover{opacity:1}svg.ch .hot{fill:var(--c4s-
Research Track · 20 scenes · 19 coding-agent configurations · re-run in October 2026.
Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.
Quality against the median cost of one case · Research Track, 20 scenes
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the median USD per case. Gemma 4 31B (thinking on) has no point: it ran on self-hosted weights, so nothing was billed. Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.
Research Track, 20 scenes, re-run in October 2026 with every agent in the same sandbox. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical; a case without a valid scene scores 0. Detailed, Overview and Physical are means of the per-case scores. Cost is the median per case. *Muse Wonder Full 1.0 (max) is sold by subscription; cost = the token counts its CLI logs, at Muse Wonder's per-token list price.
| # | Agent configuration | Score | Detailed | Overview | Physical | Median cost |
|---|---|---|---|---|---|---|
| 1 | Claude Sonnet 5.5 (max)ParetoAnthropic | 0.803 | 0.812 | 0.945 | $26.93 | |
| 2 | Claude Opus 5.5 (max)ParetoAnthropic | 0.763 | 0.818 | 0.943 | $26.90 | |
| 3 | Claude Fable 5.1 (max)ParetoAnthropic | 0.746 | 0.756 | 0.983 | $23.42 | |
| 4 | GPT-6 Astra (max)ParetoOpenAI | 0.760 | 0.811 | 0.759 | $14.25 | |
| 5 | GPT-6.1 Sol (max)ParetoOpenAI | 0.742 | 0.778 | 0.845 | $2.59 | |
| 6 | Gemini 4 Argon (high)Google | 0.688 | 0.756 | 0.930 | $4.83 | |
| 7 | GPT-5.6 Sol (max)OpenAI | 0.640 | 0.713 | 0.836 | $6.87 | |
| 8 | Claude Haiku 5.5 (max)ParetoAnthropic | 0.652 | 0.667 | 0.937 | $2.43 | |
| 9 | Muse Wonder Full 1.0 (max)Meta | 0.666 | 0.696 | 0.819 | $3.66* | |
| 10 | Gemini 3.8 Flash (high)Google | 0.578 | 0.662 | 0.905 | $3.85 | |
| 11 | GPT-6 Luna (max)ParetoOpenAI | 0.606 | 0.666 | 0.834 | $0.16 | |
| 12 | Grok 4.6 (xhigh)xAI | 0.573 | 0.645 | 0.619 | $7.41 | |
| 13 | Muse Spark Open (max)open weightsMeta | 0.583 | 0.591 | 0.700 | $2.01 | |
| 14 | Muse Spark 1.3 (max)Meta | 0.517 | 0.547 | 0.582 | $0.23 | |
| 15 | GLM-5.3 Flash (max)open weightsZ.ai | 0.506 | 0.545 | 0.569 | $0.94 | |
| 16 | Grok 4.7 (xhigh)xAI | 0.521 | 0.546 | 0.549 | $7.29 | |
| 17 | MiMo V2.6 Pro (max)open weightsXiaomi | 0.484 | 0.507 | 0.630 | $0.92 | |
| 18 | Gemma 4 31B (thinking on)open weightsGoogle | 0.448 | 0.469 | 0.731 | – | |
| 19 | DeepSeek V4.1 Flash (max)open weightsDeepSeek | 0.343 | 0.321 | 0.526 | $0.35 |