Spaces:
Running
Running
Add RL-oriented intro with hidden test counts
Browse files
README.md
CHANGED
|
@@ -19,8 +19,17 @@ pinned: false
|
|
| 19 |
<a href="https://huggingface.co/datasets/Program-Lab/ProgramLab"><img alt="Dataset" src="https://img.shields.io/badge/Dataset-ProgramLab-ffcc4d?style=for-the-badge&logo=huggingface"></a>
|
| 20 |
</p>
|
| 21 |
|
| 22 |
-
1,261 environments where an agent runs a compiled program it cannot read, then rebuilds the whole program
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
- [ProgramLab](https://huggingface.co/datasets/Program-Lab/ProgramLab): tasks, metadata, and build files
|
| 25 |
- [ProgramLab-Tests](https://huggingface.co/datasets/Program-Lab/ProgramLab-Tests): hidden tests
|
| 26 |
-
- Prebuilt task images: `ghcr.io/programlab-agent/programlab`
|
|
|
|
| 19 |
<a href="https://huggingface.co/datasets/Program-Lab/ProgramLab"><img alt="Dataset" src="https://img.shields.io/badge/Dataset-ProgramLab-ffcc4d?style=for-the-badge&logo=huggingface"></a>
|
| 20 |
</p>
|
| 21 |
|
| 22 |
+
1,261 RL environments where an agent runs a compiled program it cannot read, then rebuilds the whole program in any language.
|
| 23 |
+
|
| 24 |
+
**Built for RL**
|
| 25 |
+
|
| 26 |
+
- **Graded reward**: the reward is the pass rate over hidden behavioral tests. There are 187,070 in total, a median of 127 per task, and over 90% of tasks have at least 50, so partial progress earns partial reward.
|
| 27 |
+
- **Validated reward**: tests that the original program fails, that an empty program passes, or that are flaky are removed.
|
| 28 |
+
- **Hard to hack**: the agent never sees the source or the tests, and the reference program is execute-only and offline.
|
| 29 |
+
- **Ready to run**: one prebuilt image per task, scored by the official ProgramBench evaluator.
|
| 30 |
+
|
| 31 |
+
RL on ProgramLab raises Qwen3.6-35B-A3B on ProgramBench from 13.7% to 24.0% (Claude Code, two-hour budget).
|
| 32 |
|
| 33 |
- [ProgramLab](https://huggingface.co/datasets/Program-Lab/ProgramLab): tasks, metadata, and build files
|
| 34 |
- [ProgramLab-Tests](https://huggingface.co/datasets/Program-Lab/ProgramLab-Tests): hidden tests
|
| 35 |
+
- Prebuilt task images: `ghcr.io/programlab-agent/programlab`
|