YaoTang23 commited on
Commit
35c11bb
·
verified ·
1 Parent(s): 6b2cf36

Add RL-oriented intro with hidden test counts

Browse files
Files changed (1) hide show
  1. README.md +11 -2
README.md CHANGED
@@ -19,8 +19,17 @@ pinned: false
19
  <a href="https://huggingface.co/datasets/Program-Lab/ProgramLab"><img alt="Dataset" src="https://img.shields.io/badge/Dataset-ProgramLab-ffcc4d?style=for-the-badge&logo=huggingface"></a>
20
  </p>
21
 
22
- 1,261 environments where an agent runs a compiled program it cannot read, then rebuilds the whole program. Hidden behavioral tests score the result.
 
 
 
 
 
 
 
 
 
23
 
24
  - [ProgramLab](https://huggingface.co/datasets/Program-Lab/ProgramLab): tasks, metadata, and build files
25
  - [ProgramLab-Tests](https://huggingface.co/datasets/Program-Lab/ProgramLab-Tests): hidden tests
26
- - Prebuilt task images: `ghcr.io/programlab-agent/programlab`
 
19
  <a href="https://huggingface.co/datasets/Program-Lab/ProgramLab"><img alt="Dataset" src="https://img.shields.io/badge/Dataset-ProgramLab-ffcc4d?style=for-the-badge&logo=huggingface"></a>
20
  </p>
21
 
22
+ 1,261 RL environments where an agent runs a compiled program it cannot read, then rebuilds the whole program in any language.
23
+
24
+ **Built for RL**
25
+
26
+ - **Graded reward**: the reward is the pass rate over hidden behavioral tests. There are 187,070 in total, a median of 127 per task, and over 90% of tasks have at least 50, so partial progress earns partial reward.
27
+ - **Validated reward**: tests that the original program fails, that an empty program passes, or that are flaky are removed.
28
+ - **Hard to hack**: the agent never sees the source or the tests, and the reference program is execute-only and offline.
29
+ - **Ready to run**: one prebuilt image per task, scored by the official ProgramBench evaluator.
30
+
31
+ RL on ProgramLab raises Qwen3.6-35B-A3B on ProgramBench from 13.7% to 24.0% (Claude Code, two-hour budget).
32
 
33
  - [ProgramLab](https://huggingface.co/datasets/Program-Lab/ProgramLab): tasks, metadata, and build files
34
  - [ProgramLab-Tests](https://huggingface.co/datasets/Program-Lab/ProgramLab-Tests): hidden tests
35
+ - Prebuilt task images: `ghcr.io/programlab-agent/programlab`