Spaces:
Paused
Paused
Commit History
-
feat(v2): execution-grounded rewards via DuckDB -- the key differentiator
Abhinav SinghTHE CORE INNOVATION: optimized SQL queries are ACTUALLY EXECUTED against a real in-memory DuckDB database. Reward is computed from measured performance, not keyword heuristics. New files: executor.py — DuckDB engine with 4 synthetic tables (users 10k, orders 500k, products 1k, events 1M). Runs both the original and optimized query 3x each, returns median timing, result correctness, and EXPLAIN plan. leaderboard.py — In-memory best-score tracker per task, surfaced via the new /leaderboard endpoint. test_env.py — Integration test confirming real speedup (3-4x on Task 1 measured on local machine). Updated reward function (graders.py): Real Execution Speedup 35% (was: not measured at all) Result Correctness 20% (NEW: both queries must return same data) Issue Detection 25% (was: 60% keyword-only) Approval Correctness 8% Summary Quality 7% …c15d346
-
feat(core): add grader and SQLOptimEnv environment class
Abhinav Singhgraders.py — 6-component composite reward function: 1. Issue Detection 60% keyword-match against ground truth issues 2. Optimized Query 15% length + anti-pattern removal heuristics 3. Approval Correct 10% bool match vs. approved_expected 4. Summary Quality 8% progressive scoring on summary length 5. Improvement Est. 4% keyword-match on estimated_improvement field 6. Severity Labels 3% checks severity values are present Minimum reward of 0.02 for any non-empty submission (partial signal) env.py — SQLOptimEnv class: - reset(task_id): validates task, initialises episode state, returns Observation - step(action): grades action, tracks issues_found_so_far, returns StepResult - state(): returns EnvironmentState snapshot without advancing episode - Episode terminates on max_steps OR reward >= 0.95 (early exit)7841be7