KoeYe commited on
Commit
38e63d8
·
verified ·
1 Parent(s): e998ba1

Add Scholar-friendly paper page and compressed full-text PDF

Browse files

Expose the full abstract, nine authors and citation metadata on a dedicated paper page linked from the benchmark homepage. Include a complete 44-page PDF under 5 MB with original text and references preserved.

Files changed (4) hide show
  1. .gitattributes +1 -0
  2. code4scene.pdf +3 -0
  3. index.html +1 -1
  4. paper.html +40 -0
.gitattributes CHANGED
@@ -108,3 +108,4 @@ media/i2s/comp-51-suburban-driveway-mailbox-recovery-v2/qwen-on_full.jpg filter=
108
  media/i2s/comp-72-suburban-umbrella-pair-repair-v2/gemini_full.jpg filter=lfs diff=lfs merge=lfs -text
109
  media/i2s/comp-72-suburban-umbrella-pair-repair-v2/gemma-off_full.jpg filter=lfs diff=lfs merge=lfs -text
110
  media/i2s/comp-72-suburban-umbrella-pair-repair-v2/ref-1.jpg filter=lfs diff=lfs merge=lfs -text
 
 
108
  media/i2s/comp-72-suburban-umbrella-pair-repair-v2/gemini_full.jpg filter=lfs diff=lfs merge=lfs -text
109
  media/i2s/comp-72-suburban-umbrella-pair-repair-v2/gemma-off_full.jpg filter=lfs diff=lfs merge=lfs -text
110
  media/i2s/comp-72-suburban-umbrella-pair-repair-v2/ref-1.jpg filter=lfs diff=lfs merge=lfs -text
111
+ code4scene.pdf filter=lfs diff=lfs merge=lfs -text
code4scene.pdf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8c8265af82b41bb5e4ff32fe190fbf6f5dc46293fd1b3f935a481490263c05c7
3
+ size 4454655
index.html CHANGED
@@ -20,7 +20,7 @@ svg.ch .dim{opacity:.42}svg.ch g.row:hover{opacity:1}svg.ch .hot{fill:var(--c4s-
20
  .bigstats{display:flex;flex-wrap:wrap;gap:12px 34px;margin:0 0 12px}.bigstats b{display:block;font-size:30px;letter-spacing:-.02em;font-variant-numeric:tabular-nums}
21
  .bigstats span{font-size:13px;color:var(--text-secondary)}
22
  </style><link href="assets/buildingbench.css?v=20261004harborlive" rel="stylesheet"/><script type="importmap">{"imports":{"three":"./assets/vendor/three-r160/build/three.module.js","three/addons/":"./assets/vendor/three-r160/addons/"}}</script></head><body><a class="skip-link" href="#leaderboard">Skip to benchmark explorer</a>
23
- <header class="site-header"><div class="brand"><a aria-label="Code4Scene home" class="brand-home" href="index.html#top"><svg aria-hidden="true" fill="none" viewbox="0 0 30 32"><path d="M15 2 27 9v14l-12 7L3 23V9L15 2Z M3 9l12 7 12-7 M15 16v14 M9 5.5l12 7v14 M21 5.5l-12 7v14" stroke="currentColor" stroke-linejoin="round" stroke-width="1.6"></path></svg><span>Code4Scene</span></a><a class="brand-by" href="https://simworld.org/">by SimWorld</a></div><nav aria-label="Main navigation"><a class="nav-active" href="index.html#top">Benchmark</a><a href="index.html#leaderboard">Leaderboard</a><a href="index.html#cases">Visualization</a><a href="https://arxiv.org/abs/2609.36777" rel="noopener" target="_blank">Paper ↗</a><a href="https://github.com/SimWorld-AI/Code4Scene" rel="noopener" target="_blank">Github ↗</a></nav></header><main class="page-main"><section class="intro" id="top"><div class="intro-copy"><h1>Constructing<br/><em>3D Scenes with Code</em></h1><p class="intro-text">Benchmarking coding agents that turn open-ended scene descriptions into engine-native 3D scenes in Unreal Engine. Measuring spatial reasoning, task fulfillment and physical validity.</p><div class="intro-links"><a href="https://arxiv.org/abs/2609.36777">Read the paper ↗</a><a href="https://github.com/SimWorld-AI/Code4Scene">Code &amp; Data ↗</a><a href="#hero-video">Watch the teaser ↘</a></div></div><div class="featured featured-video" id="hero-video"><video aria-label="Code4Scene overview video" controls="" playsinline="" poster="media/teaser-poster.jpg" preload="metadata"><source src="media/teaser-v31.mp4" type="video/mp4"/>Your browser does not support embedded video. <a href="media/teaser-v31.mp4">Watch the Code4Scene video</a>.</video><div class="featured-video-caption"><strong>Code4Scene in 90 seconds</strong><span>Construction and evaluation</span></div></div></section><section class="task-section" id="task-format"><div class="container"><h2 class="section-title">Task Format</h2><div class="task-shell"><div aria-label="Task format" class="task-tabs" role="tablist"><button aria-controls="task-t2s" aria-selected="true" data-task="t2s" id="task-tab-t2s" role="tab">Text-to-Scene</button><button aria-controls="task-i2s" aria-selected="false" data-task="i2s" id="task-tab-i2s" role="tab" tabindex="-1">Image-to-Scene</button></div><div aria-labelledby="task-tab-t2s" class="task-content" data-task="t2s" id="task-t2s" role="tabpanel"><h3>Nordic Harbour <span>· Text-to-Scene</span></h3><p class="task-subtitle">An open-ended scene description, an asset catalog and the Unreal Engine editor.</p><div class="task-columns"><article class="task-panel"><div class="panel-heading"><strong>Agent input</strong><span>Text + assets</span></div><div class="prompt-preview"><span class="eyebrow">Task instruction</span><p>Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.</p><p class="prompt-detail">Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.</p></div><p class="task-foot"><strong>Given:</strong> an empty level and the content pack’s asset catalog.</p></article><article class="task-panel output-panel"><div class="panel-heading"><strong>Agent output</strong><span>GPT-6 Astra</span></div><div class="live-scene" data-agent="astra" data-case="nordic-harbour"><canvas aria-label="Interactive 3D scene: Nordic Harbour. Drag to orbit, scroll or use plus and minus to zoom." tabindex="0"></canvas><span class="scene-loading" role="status">Loading 3D scene…</span><span class="scene-hint">Drag to orbit · Scroll to zoom</span><button aria-label="Pause scene rotation" class="scene-motion">Ⅱ</button></div><a class="task-foot text-button" href="cases.html#nordic-harbour">Compare agents and evaluator views ↗</a></article><article class="task-panel evaluation-panel"><div class="panel-heading"><strong>Evaluation</strong><span>Engine-native scene</span></div><div class="score-block"><span>CASE SCORE</span><strong>0.877</strong><dl><div><dt>Detailed Alignment</dt><dd>0.757</dd></div><div><dt>Overview Alignment</dt><dd>0.909</dd></div><div><dt>Physical Safety</dt><dd>0.900</dd></div></dl></div><p class="task-foot">Task fulfillment, artifact integrity and static physical validity.</p><p class="task-formula">0.2 × Detailed + 0.6 × Overview + 0.2 × Physical</p></article></div><details class="task-full"><summary><span>+ Full task instruction</span><span>TEXT-TO-SCENE</span></summary><p>Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.</p></details><div class="availability"><span><b>20</b> scenes in the <strong>Research Track</strong>: affordable, reproducible evaluation for academic and model-development use</span><span><b>160</b> scenes in the <strong>Full Benchmark</strong>: comprehensive evaluation for leaderboard and final reporting</span></div></div><div aria-labelledby="task-tab-i2s" class="task-content" data-task="i2s" hidden="" id="task-i2s" role="tabpanel"><h3>Old Industrial Pallet Bay <span>· Image-to-Scene</span></h3><p class="task-subtitle">Repair the corrupted scene using its reference image; preserve all unrelated actors.</p><div class="task-columns"><article class="task-panel input-panel"><div class="panel-heading"><strong>Agent input</strong><span>Scene + reference image</span></div><div class="live-scene" data-agent="astra" data-case="comp-31-old-industrial-pallet-bay-repair-v2" data-state="input"><canvas aria-label="Interactive 3D: Old Industrial Pallet Bay, input. Drag to orbit; scroll to zoom." tabindex="0"></canvas><span class="scene-loading" role="status">Loading 3D scene…</span><span class="scene-hint">Drag to orbit · Scroll to zoom</span><button aria-label="Resume scene rotation" class="scene-motion">▶</button></div><a class="task-reference" href="media/i2s/comp-31-old-industrial-pallet-bay-repair-v2/ref-1.jpg?v=1790549706" rel="noopener" target="_blank"><img alt="Provided reference image for Old Industrial Pallet Bay" loading="lazy" src="media/i2s/comp-31-old-industrial-pallet-bay-repair-v2/ref-1.jpg?v=1790549706"/><span>Provided reference image ↗</span></a><p class="task-foot">A translated and rotated pallet stack, plus an unintended duplicate.</p></article><article class="task-panel output-panel"><div class="panel-heading"><strong>Agent output</strong><span>GPT-6 Astra</span></div><div class="live-scene" data-agent="astra" data-case="comp-31-old-industrial-pallet-bay-repair-v2" data-state="repair"><canvas aria-label="Interactive 3D: Old Industrial Pallet Bay, repair. Drag to orbit; scroll to zoom." tabindex="0"></canvas><span class="scene-loading" role="status">Loading 3D scene…</span><span class="scene-hint">Drag to orbit · Scroll to zoom</span><button aria-label="Resume scene rotation" class="scene-motion">▶</button></div><div aria-label="Output scene state" class="scene-versions"><button class="on" data-state="repair">Repair</button><button data-state="input">Corrupted input</button><button data-state="gt">Ground truth</button></div><p class="task-foot">Inspect the saved repair from the same reference camera.</p><a class="text-button task-foot" href="cases.html#comp-31-old-industrial-pallet-bay-repair-v2">Compare agents and evaluator views ↗</a></article><article class="task-panel evaluation-panel"><div class="panel-heading"><strong>Evaluation</strong><span>Actor-level repair</span></div><div class="score-block"><span>CASE SCORE</span><strong>1.000</strong><dl><div><dt>Repair F1</dt><dd>1.000</dd></div><div><dt>Physical Safety</dt><dd>1.000</dd></div><div><dt>TP / FP / FN</dt><dd>2 / 0 / 0</dd></div></dl></div><p class="task-foot">Restore each target within 5 cm, 5° and 5%, without unintended edits.</p><p class="task-formula">0.8 × Repair F1 + 0.2 × Physical</p></article></div><details class="task-full"><summary><span>+ Full task instruction</span><span>IMAGE-TO-SCENE</span></summary><p>Repair the current outdoor scene so it matches the provided reference image. Make only the minimum changes needed, and preserve all unrelated actors and properties.</p></details><div class="availability"><span><b>160</b> editing cases in the full benchmark target, including private cases</span><span><b>72</b> public editing cases (22 indoor, 50 outdoor)</span></div></div></div></div></section><section id="leaderboard"><div class="container">
24
  <div class="section-header centered"><span class="section-label">Leaderboard</span><h2 class="section-title">Leaderboard</h2>
25
  <p class="section-desc" data-view="t2s">Research Track · 20 representative scenes · 14 coding-agent configurations.</p><p class="section-desc" data-view="i2s" style="display:none">Image-to-Scene · 75 public cases (25 indoor, 50 outdoor) · 14 coding-agent configurations.</p><p class="tracks-why" data-view="t2s">Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.</p></div>
26
  <div id="lbwrap"><div class="lb-tabs"><div class="seg seg-lg" data-for="leaderboard"><button class="on" data-view="t2s">Text-to-Scene</button><button data-view="i2s">Image-to-Scene</button></div></div><div class="card bar-card"><div class="bar-heading"><h3>Code4Scene <span class="q" data-view="t2s">Research Track</span><span class="q" data-view="i2s" style="display:none">Image-to-Scene</span></h3><span><span data-view="t2s">20 scenes · score out of 100</span><span data-view="i2s" style="display:none">75 scenes · score out of 100</span></span></div><p class="bar-mobile-hint">Swipe to view all 14 configurations →</p><div aria-label="All 14 model configurations" class="bar-scroll" role="region" tabindex="0"><div data-view="t2s"><svg aria-label="Research Track scores for the 14 Code4Scene configurations" class="kit-bars" id="bars-t2s" role="group" viewbox="0 0 1200 420"></svg></div><div data-view="i2s" style="display:none"><svg aria-label="Image-to-Scene scores for the 14 Code4Scene paper configurations" class="kit-bars" id="bars-i2s" role="group" viewbox="0 0 1200 420"></svg></div></div><p class="note">Sorted by score. Bar labels round scores ×100; the table preserves three decimal places. One color per model provider.</p></div>
 
20
  .bigstats{display:flex;flex-wrap:wrap;gap:12px 34px;margin:0 0 12px}.bigstats b{display:block;font-size:30px;letter-spacing:-.02em;font-variant-numeric:tabular-nums}
21
  .bigstats span{font-size:13px;color:var(--text-secondary)}
22
  </style><link href="assets/buildingbench.css?v=20261004harborlive" rel="stylesheet"/><script type="importmap">{"imports":{"three":"./assets/vendor/three-r160/build/three.module.js","three/addons/":"./assets/vendor/three-r160/addons/"}}</script></head><body><a class="skip-link" href="#leaderboard">Skip to benchmark explorer</a>
23
+ <header class="site-header"><div class="brand"><a aria-label="Code4Scene home" class="brand-home" href="index.html#top"><svg aria-hidden="true" fill="none" viewbox="0 0 30 32"><path d="M15 2 27 9v14l-12 7L3 23V9L15 2Z M3 9l12 7 12-7 M15 16v14 M9 5.5l12 7v14 M21 5.5l-12 7v14" stroke="currentColor" stroke-linejoin="round" stroke-width="1.6"></path></svg><span>Code4Scene</span></a><a class="brand-by" href="https://simworld.org/">by SimWorld</a></div><nav aria-label="Main navigation"><a class="nav-active" href="index.html#top">Benchmark</a><a href="index.html#leaderboard">Leaderboard</a><a href="index.html#cases">Visualization</a><a href="https://arxiv.org/abs/2609.36777" rel="noopener" target="_blank">Paper ↗</a><a href="https://github.com/SimWorld-AI/Code4Scene" rel="noopener" target="_blank">Github ↗</a></nav></header><main class="page-main"><section class="intro" id="top"><div class="intro-copy"><h1>Constructing<br/><em>3D Scenes with Code</em></h1><p class="intro-text">Benchmarking coding agents that turn open-ended scene descriptions into engine-native 3D scenes in Unreal Engine. Measuring spatial reasoning, task fulfillment and physical validity.</p><div class="intro-links"><a href="paper.html">Read the paper ↗</a><a href="https://github.com/SimWorld-AI/Code4Scene">Code &amp; Data ↗</a><a href="#hero-video">Watch the teaser ↘</a></div></div><div class="featured featured-video" id="hero-video"><video aria-label="Code4Scene overview video" controls="" playsinline="" poster="media/teaser-poster.jpg" preload="metadata"><source src="media/teaser-v31.mp4" type="video/mp4"/>Your browser does not support embedded video. <a href="media/teaser-v31.mp4">Watch the Code4Scene video</a>.</video><div class="featured-video-caption"><strong>Code4Scene in 90 seconds</strong><span>Construction and evaluation</span></div></div></section><section class="task-section" id="task-format"><div class="container"><h2 class="section-title">Task Format</h2><div class="task-shell"><div aria-label="Task format" class="task-tabs" role="tablist"><button aria-controls="task-t2s" aria-selected="true" data-task="t2s" id="task-tab-t2s" role="tab">Text-to-Scene</button><button aria-controls="task-i2s" aria-selected="false" data-task="i2s" id="task-tab-i2s" role="tab" tabindex="-1">Image-to-Scene</button></div><div aria-labelledby="task-tab-t2s" class="task-content" data-task="t2s" id="task-t2s" role="tabpanel"><h3>Nordic Harbour <span>· Text-to-Scene</span></h3><p class="task-subtitle">An open-ended scene description, an asset catalog and the Unreal Engine editor.</p><div class="task-columns"><article class="task-panel"><div class="panel-heading"><strong>Agent input</strong><span>Text + assets</span></div><div class="prompt-preview"><span class="eyebrow">Task instruction</span><p>Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.</p><p class="prompt-detail">Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.</p></div><p class="task-foot"><strong>Given:</strong> an empty level and the content pack’s asset catalog.</p></article><article class="task-panel output-panel"><div class="panel-heading"><strong>Agent output</strong><span>GPT-6 Astra</span></div><div class="live-scene" data-agent="astra" data-case="nordic-harbour"><canvas aria-label="Interactive 3D scene: Nordic Harbour. Drag to orbit, scroll or use plus and minus to zoom." tabindex="0"></canvas><span class="scene-loading" role="status">Loading 3D scene…</span><span class="scene-hint">Drag to orbit · Scroll to zoom</span><button aria-label="Pause scene rotation" class="scene-motion">Ⅱ</button></div><a class="task-foot text-button" href="cases.html#nordic-harbour">Compare agents and evaluator views ↗</a></article><article class="task-panel evaluation-panel"><div class="panel-heading"><strong>Evaluation</strong><span>Engine-native scene</span></div><div class="score-block"><span>CASE SCORE</span><strong>0.877</strong><dl><div><dt>Detailed Alignment</dt><dd>0.757</dd></div><div><dt>Overview Alignment</dt><dd>0.909</dd></div><div><dt>Physical Safety</dt><dd>0.900</dd></div></dl></div><p class="task-foot">Task fulfillment, artifact integrity and static physical validity.</p><p class="task-formula">0.2 × Detailed + 0.6 × Overview + 0.2 × Physical</p></article></div><details class="task-full"><summary><span>+ Full task instruction</span><span>TEXT-TO-SCENE</span></summary><p>Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.</p></details><div class="availability"><span><b>20</b> scenes in the <strong>Research Track</strong>: affordable, reproducible evaluation for academic and model-development use</span><span><b>160</b> scenes in the <strong>Full Benchmark</strong>: comprehensive evaluation for leaderboard and final reporting</span></div></div><div aria-labelledby="task-tab-i2s" class="task-content" data-task="i2s" hidden="" id="task-i2s" role="tabpanel"><h3>Old Industrial Pallet Bay <span>· Image-to-Scene</span></h3><p class="task-subtitle">Repair the corrupted scene using its reference image; preserve all unrelated actors.</p><div class="task-columns"><article class="task-panel input-panel"><div class="panel-heading"><strong>Agent input</strong><span>Scene + reference image</span></div><div class="live-scene" data-agent="astra" data-case="comp-31-old-industrial-pallet-bay-repair-v2" data-state="input"><canvas aria-label="Interactive 3D: Old Industrial Pallet Bay, input. Drag to orbit; scroll to zoom." tabindex="0"></canvas><span class="scene-loading" role="status">Loading 3D scene…</span><span class="scene-hint">Drag to orbit · Scroll to zoom</span><button aria-label="Resume scene rotation" class="scene-motion">▶</button></div><a class="task-reference" href="media/i2s/comp-31-old-industrial-pallet-bay-repair-v2/ref-1.jpg?v=1790549706" rel="noopener" target="_blank"><img alt="Provided reference image for Old Industrial Pallet Bay" loading="lazy" src="media/i2s/comp-31-old-industrial-pallet-bay-repair-v2/ref-1.jpg?v=1790549706"/><span>Provided reference image ↗</span></a><p class="task-foot">A translated and rotated pallet stack, plus an unintended duplicate.</p></article><article class="task-panel output-panel"><div class="panel-heading"><strong>Agent output</strong><span>GPT-6 Astra</span></div><div class="live-scene" data-agent="astra" data-case="comp-31-old-industrial-pallet-bay-repair-v2" data-state="repair"><canvas aria-label="Interactive 3D: Old Industrial Pallet Bay, repair. Drag to orbit; scroll to zoom." tabindex="0"></canvas><span class="scene-loading" role="status">Loading 3D scene…</span><span class="scene-hint">Drag to orbit · Scroll to zoom</span><button aria-label="Resume scene rotation" class="scene-motion">▶</button></div><div aria-label="Output scene state" class="scene-versions"><button class="on" data-state="repair">Repair</button><button data-state="input">Corrupted input</button><button data-state="gt">Ground truth</button></div><p class="task-foot">Inspect the saved repair from the same reference camera.</p><a class="text-button task-foot" href="cases.html#comp-31-old-industrial-pallet-bay-repair-v2">Compare agents and evaluator views ↗</a></article><article class="task-panel evaluation-panel"><div class="panel-heading"><strong>Evaluation</strong><span>Actor-level repair</span></div><div class="score-block"><span>CASE SCORE</span><strong>1.000</strong><dl><div><dt>Repair F1</dt><dd>1.000</dd></div><div><dt>Physical Safety</dt><dd>1.000</dd></div><div><dt>TP / FP / FN</dt><dd>2 / 0 / 0</dd></div></dl></div><p class="task-foot">Restore each target within 5 cm, 5° and 5%, without unintended edits.</p><p class="task-formula">0.8 × Repair F1 + 0.2 × Physical</p></article></div><details class="task-full"><summary><span>+ Full task instruction</span><span>IMAGE-TO-SCENE</span></summary><p>Repair the current outdoor scene so it matches the provided reference image. Make only the minimum changes needed, and preserve all unrelated actors and properties.</p></details><div class="availability"><span><b>160</b> editing cases in the full benchmark target, including private cases</span><span><b>72</b> public editing cases (22 indoor, 50 outdoor)</span></div></div></div></div></section><section id="leaderboard"><div class="container">
24
  <div class="section-header centered"><span class="section-label">Leaderboard</span><h2 class="section-title">Leaderboard</h2>
25
  <p class="section-desc" data-view="t2s">Research Track · 20 representative scenes · 14 coding-agent configurations.</p><p class="section-desc" data-view="i2s" style="display:none">Image-to-Scene · 75 public cases (25 indoor, 50 outdoor) · 14 coding-agent configurations.</p><p class="tracks-why" data-view="t2s">Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.</p></div>
26
  <div id="lbwrap"><div class="lb-tabs"><div class="seg seg-lg" data-for="leaderboard"><button class="on" data-view="t2s">Text-to-Scene</button><button data-view="i2s">Image-to-Scene</button></div></div><div class="card bar-card"><div class="bar-heading"><h3>Code4Scene <span class="q" data-view="t2s">Research Track</span><span class="q" data-view="i2s" style="display:none">Image-to-Scene</span></h3><span><span data-view="t2s">20 scenes · score out of 100</span><span data-view="i2s" style="display:none">75 scenes · score out of 100</span></span></div><p class="bar-mobile-hint">Swipe to view all 14 configurations →</p><div aria-label="All 14 model configurations" class="bar-scroll" role="region" tabindex="0"><div data-view="t2s"><svg aria-label="Research Track scores for the 14 Code4Scene configurations" class="kit-bars" id="bars-t2s" role="group" viewbox="0 0 1200 420"></svg></div><div data-view="i2s" style="display:none"><svg aria-label="Image-to-Scene scores for the 14 Code4Scene paper configurations" class="kit-bars" id="bars-i2s" role="group" viewbox="0 0 1200 420"></svg></div></div><p class="note">Sorted by score. Bar labels round scores ×100; the table preserves three decimal places. One color per model provider.</p></div>
paper.html ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!doctype html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="utf-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1">
6
+ <title>Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes</title>
7
+ <meta name="description" content="Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely control scene state remains unclear. The generated 3D scene is a persistent, executable artifact: a convincing render can hide incorrect spatial relations, intersecting objects, or unintended modifications. We introduce Code4Scene, a benchmark of 190 Unreal Engine cases built from human-assembled scenes that evaluates coding agents on two complementary settings under a shared execution interface. Construction tests scene-level spatial reasoning from open-ended language specifications, where many realizations are valid; editing tests precise control of scene state, where the agent must recover the target scene from reference images while preserving everything else. Rather than scoring code or rendered views, Code4Scene evaluates the generated engine-native scene for task fulfillment, artifact integrity, and static physical validity, with edits additionally compared against withheld ground truth. Across 14 coding-agent configurations on the 95-case public set, construction and editing performance are strongly correlated but not interchangeable (Spearman ρ = 0.78): Claude Fable 5.1 leads construction, Gemini 3.8 Flash leads editing, and GPT-6 Astra narrowly leads overall. Spatial Composition is the weakest construction category for every agent, while editing remains imprecise: the best Repair F1 is only 0.527, and 35.8% of edits that fully recover the target still introduce unintended changes elsewhere in the scene. These results expose a gap between plausible 3D generation and reliable spatial reasoning and state control.">
8
+ <meta name="citation_title" content="Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes">
9
+ <meta name="citation_author" content="Xiaokang Ye">
10
+ <meta name="citation_author" content="Siddhant Hitesh Mantri">
11
+ <meta name="citation_author" content="Zimeng Chen">
12
+ <meta name="citation_author" content="Edward Zhang">
13
+ <meta name="citation_author" content="Zhaoxu Zheng">
14
+ <meta name="citation_author" content="Yuanheng Li">
15
+ <meta name="citation_author" content="Yizhao Chen">
16
+ <meta name="citation_author" content="Tianyang Huang">
17
+ <meta name="citation_author" content="Lianhui Qin">
18
+ <meta name="citation_publication_date" content="2026/09/29">
19
+ <meta name="citation_arxiv_id" content="2609.36777">
20
+ <meta name="citation_pdf_url" content="https://simworld-ai-code4scene.static.hf.space/code4scene.pdf">
21
+ <link rel="canonical" href="https://simworld-ai-code4scene.static.hf.space/paper.html">
22
+ <link rel="icon" href="assets/code4scene-mark.svg">
23
+ <style>
24
+ *{box-sizing:border-box}body{margin:0;background:#faf9f6;color:#242722;font:17px/1.65 system-ui,-apple-system,sans-serif}
25
+ main{max-width:980px;margin:auto;padding:36px 28px 64px}a{color:#365b43;text-underline-offset:4px}nav{margin-bottom:32px;font-size:14px}
26
+ .label{font-size:12px;letter-spacing:.13em;text-transform:uppercase;color:#697365}h1{font-size:clamp(27px,4vw,40px);line-height:1.18;letter-spacing:-.03em;margin:14px 0 22px;font-weight:650}
27
+ .authors{font-size:16px;line-height:1.7}.date{color:#687065;font-size:14px}.links{display:flex;flex-wrap:wrap;gap:12px;margin:22px 0 30px}.links a{border:1px solid #bcc8b9;padding:8px 16px;border-radius:5px;text-decoration:none;font-size:14px}.links a:first-child{background:#365b43;color:white;border-color:#365b43}
28
+ h2{font-size:22px;margin:24px 0 10px}.abstract{font-size:16px;line-height:1.7}footer{border-top:1px solid #dce0d8;padding-top:18px;margin-top:32px;font-size:13px;color:#687065}
29
+ </style>
30
+ </head>
31
+ <body><main>
32
+ <nav><a href="index.html">← Code4Scene benchmark</a></nav>
33
+ <div class="label">Research paper · arXiv:2609.36777</div>
34
+ <h1>Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes</h1>
35
+ <p class="authors">Xiaokang Ye · Siddhant Hitesh Mantri · Zimeng Chen · Edward Zhang · Zhaoxu Zheng · Yuanheng Li · Yizhao Chen · Tianyang Huang · Lianhui Qin</p>
36
+ <p class="date">Preprint · September 29, 2026</p>
37
+ <div class="links"><a href="code4scene.pdf">Read paper (PDF · 4.5 MB)</a><a href="https://arxiv.org/abs/2609.36777">arXiv</a><a href="https://github.com/SimWorld-AI/Code4Scene">Code &amp; data</a></div>
38
+ <section aria-labelledby="abstract"><h2 id="abstract">Abstract</h2><p class="abstract">Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely control scene state remains unclear. The generated 3D scene is a persistent, executable artifact: a convincing render can hide incorrect spatial relations, intersecting objects, or unintended modifications. We introduce Code4Scene, a benchmark of 190 Unreal Engine cases built from human-assembled scenes that evaluates coding agents on two complementary settings under a shared execution interface. Construction tests scene-level spatial reasoning from open-ended language specifications, where many realizations are valid; editing tests precise control of scene state, where the agent must recover the target scene from reference images while preserving everything else. Rather than scoring code or rendered views, Code4Scene evaluates the generated engine-native scene for task fulfillment, artifact integrity, and static physical validity, with edits additionally compared against withheld ground truth. Across 14 coding-agent configurations on the 95-case public set, construction and editing performance are strongly correlated but not interchangeable (Spearman ρ = 0.78): Claude Fable 5.1 leads construction, Gemini 3.8 Flash leads editing, and GPT-6 Astra narrowly leads overall. Spatial Composition is the weakest construction category for every agent, while editing remains imprecise: the best Repair F1 is only 0.527, and 35.8% of edits that fully recover the target still introduce unintended changes elsewhere in the scene. These results expose a gap between plausible 3D generation and reliable spatial reasoning and state control.</p></section>
39
+ <footer>The PDF is a compressed copy of the complete arXiv v1 paper, including all 44 pages and references. The original is available on arXiv.</footer>
40
+ </main></body></html>