Commit History
-
Bill eval jobs to lvwerra instead of michaelr27
Michael Rabinovich190633b
-
leaderboard: serve renders from the public bucket, not the dataset proxy
Michael Rabinovich +1: CursorCandidate renders (turntables + edit-diff WebP) now live in the public HF Storage Bucket under renders/<id>/, uploaded once by the eval job. The gallery grid and the hosted report reference them by anonymous bucket URL, so the Space is out of the render read path and the binary renders stop bloating the submissions dataset / its commit queue. - leaderboard: HF_RENDER_BUCKET + render URL/path/prefix helpers. - app: render resolvers return public bucket URLs; the two render proxy routes + fetchers are retired. Only the private GT render keeps its token-held proxy. - submit: merge passes render_base_url to generate_html and commits report-only; dispatched job gets CADGENBENCH_RENDER_BUCKET/HF_ENDPOINT. - admin: delete also purges the bucket renders/<id>/ prefix (previously orphaned). - Dockerfile: bump cadgenbench pin to 3d49822 (render_base_url support). Co-authored-by: Cursor <cursoragent@cursor.com>d2161b1
-
leaderboard: open reports in new tab
Michael RabinovichNow that the Space is public, link completed submissions directly to their proxied HTML reports instead of inlining large report payloads through the row-click detail panel.5fb3ebc
-
add visual Gallery tab (top-10 verified, sticky GT, fixture picker)
Michael Rabinovich +1: CursorVisual-first leaderboard as a new default tab. Self-contained HTML (gallery.py) inlined into an iframe srcdoc with base64 thumbnails, so it works on the private Space where HF's edge 404s in-browser custom-route fetches (same constraint the report viewer's srcdoc path already handles). Top-10 verified only, teal sticky ground-truth row, 3-fixture picker (drop-oldest, global re-render), dashed cell for invalid/missing fixtures, per-row generation/editing/validity expander, and a GT-vs-output compare modal per thumbnail. Existing Leaderboard / Submit / About / Admin tabs unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>01d67e9 -
leaderboard: drop silent fallback; boot resilient on Hub read failure
Michael RabinovichThe leaderboard previously fell back to a bundled/local results.jsonl on any Hub error, which let an under-scoped Space HF_TOKEN silently serve stale baked-in rows that looked up-to-date. Remove the fallback: the live results.jsonl is the single source of truth and any read failure raises LeaderboardDataError. To stay robust rather than crash the Space at construction time, app.py wraps the readers (_safe_load_split / _safe_load_admin): a failed read yields empty, correctly-shaped tables plus a loud, persistent banner and a gr.Warning toast on refresh / Timer tick, and is logged via logger.exception. No stale or cached data is ever shown in place of a failed read. Also git rm the bundled results.jsonl and add regression tests for the no-fallback contract and the boot-resilience wrappers.a662bfa
-
leaderboard: log the local-fallback path instead of printing
Michael Rabinovich3dba2e2
-
license: add apache-2.0 header to space modules
Michael Rabinovichb048bc2 -
admin: bulk promote/demote/delete + admin table loader
Michael Rabinovichadds load_admin_table to the leaderboard reader: a single flat frame of every row (both tiers) with a leading editable select column, for the admin tab to act on. reworks admin.py around bulk helpers (promote_rows, demote_rows, delete_rows) that each do one results.jsonl write for the whole selection; delete also best-effort removes the companion zip and report artifacts. the singular promote_row/demote_row stay as thin wrappers. tests cover the bulk paths, a missing-id abort, empty-selection guards, and artifact deletion, all with the hub mocked.0957a56
-
leaderboard: format submitted_at as `YYYY-MM-DD HH:MM UTC`; lock tables read-only
Michael RabinovichTable cells were editable (gradio_leaderboard inherits gr.Dataframe's interactive default); add interactive=False to both Validated and Unvalidated widgets so cell contents aren't accidentally typed over. submitted_at column rendered as raw ISO-8601 ("2026-05-28T07:13:16Z"); apply the same _fmt_timestamp helper the detail panel already uses so the table reads as "2026-05-28 07:13 UTC". Helper moved into leaderboard.py (formatter module) and re-imported by app.py.c4e21b3
-
leaderboard+app: combined CSV download with validation_status discriminator
Michael RabinovichBundle 1+2 C8. One file, both tiers, downloaded via `gr.DownloadButton` next to the existing Refresh button. leaderboard.py: - New `CSV_COLUMNS` constant (wider than the on-screen table: raw aggregate_score / validity_rate, plus provenance fields like submission_sha256 + cadgenbench_data_revision + notes + failure_reason). `validation_status` is the discriminator that lets a reader grep the file for one tier or the other. - New `build_combined_csv()` reads the same rows the leaderboard reader does, applies the same status / validation_status defaults, projects to CSV_COLUMNS, sort: validated rows on top (highest score first) then unvalidated, then writes to a unique /tmp file and returns the path. Fresh on every click so the export reflects the latest data, not a snapshot captured at boot. app.py: - Imports `build_combined_csv`. The Refresh button and the new `gr.DownloadButton("Download CSV", size="sm")` live in a single `gr.Row` so they sit side-by-side unde…f585077 -
app: render reports inline via iframe srcdoc (private-Space safe)
Michael RabinovichPolish follow-up on C6. Reports now render inside the Space, inside the same iframe HF uses to host the leaderboard, with no second HTTP request leaving the browser. Why: the Space is private, and HF's edge gates same-origin pathname navigations on a short-lived __sign JWT that browsers don't carry forward when a user clicks a relative link. The previous design (submission_name -> /reports/<id>.html) worked server-side under Bearer auth (verified) but 404'd in the browser. Confirmed via external review and reproduced with a requests.Session cookie test. This commit: - Drops the submission_name -> /reports/<id>.html link wrapping from the rendered table; the cell stays plain text. The underlying `report_url` column (relative URL to the proxy) is still computed for the modern-pipeline gate but no longer surfaced as a clickable cell. - Adds a `gr.HTML` viewer below the detail panel. Row-click triggers `_format_detail_and_report`, which returns (detail_markdown, report_ifram…0e3b21f -
app+leaderboard: link submission_name to a Space-side report proxy
Michael RabinovichPolish follow-up #2 on C6. Three user-visible changes plus the plumbing required to make them work. 1. Drop the `report` and `zip` columns from the rendered table. `submission_name` becomes the report's deep link (typical HF leaderboard pattern: the system / model name doubles as the link target; cf. Open LLM Leaderboard, DABstep, BigCodeBench). Modern completed rows get `[<name>](/reports/<sid>.html)`; pending, failed, and legacy seed rows fall back to plain text so there's no broken click. The hidden `submission_blob_url` and the computed `report_url` ride along in the DataFrame and surface in the row-click detail panel only. 2. Rename the agent-URL column from `agent` to `model details (optional)`. Missing cells render as italic `_None_` so the optionality is explicit instead of a blank. "Agent" was confusing; the schema lets `agent_url` be code OR paper OR a blog post and "agent" reads narrow. 3. Add a Space-side proxy for the per-submission H…77edebf -
app+leaderboard: detail panel polish (rename link columns, fix broken report links)
Michael RabinovichPolish follow-up on C6, three fixes after the first detail-panel commit landed: 1. Rename display columns from snake_case schema names to clean short headers. `agent_url` -> `agent`, `submission_blob_url` -> `zip`, `report_url` -> `report`. The rename happens at projection time via `df.rename`, so the schema in results.jsonl is unchanged. Headers now read as plain English ("agent", "zip", "report") rather than a python dict. 2. The `agent` column shows the URL itself (scheme stripped, truncated past 40 chars) as link text instead of a fixed "code" label. The schema says agent_url can be code OR paper; the shortened URL is the only honest hint about what's behind the click. Missing values render as italic `_None_` rather than a blank cell, so the optionality is explicit. 3. The `report` link is now gated on the row's `submission_sha256` being non-null in addition to status == "completed". Legacy seed rows pre-date the modern submit pipeline (which …1a8f331 -
app: row-click detail panel below the leaderboard tables
Michael RabinovichBundle 1+2 C6. Single shared `gr.Markdown` panel below both Leaderboard widgets; either table's `.select(...)` event populates it with the clicked row's notes, agent_url, submitted_at, submission zip + report links (always), and failure_reason (failed rows only). Spec'd shape: one panel, two select sources. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS each grow by three hidden detail-only fields (submission_id, notes, failure_reason). They ride in the underlying DataFrame so the select handler can read them out directly without a parallel state cache. - New LEADERBOARD_HIDE_COLUMNS constant lists the columns hidden from view; both Leaderboard widgets in app.py consume it. - `_project_and_format` now fills missing columns with None (legacy rows can lack optional fields like notes; the panel needs the column-set to be stable regardless of source-row shape). app.py: - pandas imported (for the helper's pd.isna check). - New `_format_detail(df, evt)` helper …3112173 -
leaderboard: markdown link columns for agent_url, submission, report
Michael RabinovichBundle 1+2 C5. Three new columns on both tables (validated + unvalidated) carry clickable links rendered via gradio_leaderboard's `datatype="markdown"`: - agent_url: link to the agent's source / paper if the row's meta.json provided one (label "code"; empty cell when null). - submission_blob_url: link to the uploaded zip on the submissions dataset (label "zip"; populated on every non-failed row, the same URL the submit handler computed at upload time). - report_url: built at read time from `submission_id` plus HF_SUBMISSIONS_REPO; points at `reports/<id>.html` under /resolve/main/ so the browser renders the report directly. Only emitted for `status == "completed"` rows. leaderboard.py: - LEADERBOARD_COLS + VALIDATED_LEADERBOARD_COLS grow by three. - New LEADERBOARD_DATATYPES + VALIDATED_LEADERBOARD_DATATYPES constants mark the link columns as "markdown", everything else as "str". Tied to the column lists by a single helper so the two stay in lockstep. - New _agent_u…53de73a -
app: split Leaderboard into Validated + Unvalidated tables
Michael RabinovichBundle 1+2 C3. Swaps the single Leaderboard widget for two stacked ones (Validated on top, Unvalidated below) and wires the existing auto-refresh Timer + Refresh button to push fresh dataframes into both. Initial render reads via the C2 `load_leaderboard_split()`; existing rows all land in Unvalidated until a maintainer flips `validation_status` to `"validated"` on a row in the submissions dataset. - app.py: replaces `df_view` with `validated_view` + `unvalidated_view`. Single timer tick now fans out to both widgets (load split runs once per tick, both halves get pushed). Refresh button same fan-out shape. Inline comment points at the validation policy decision doc. - leaderboard.py: drops the now-unused single-DataFrame `load_leaderboard()` per the C3 cleanup half. Stale docstring references that pointed at the dead function got smoothed out (the split reader's doc is self-contained now). - tests/test_smoke.py: picks up the C3 acceptance with a new `test_two_leaderboa…046548a -
leaderboard: add load_leaderboard_split() for two-tier viewer
Michael RabinovichBundle 1+2 C2. Additive new function for the C3 two-stacked-table swap; existing single-DataFrame load_leaderboard() stays put until C3 flips the caller in app.py + drops it. Keeps the live Space green between commits. - VALIDATED_LEADERBOARD_COLS: same shape as LEADERBOARD_COLS plus `validation_method` (always null on unvalidated rows, so omitted from that tier rather than rendered). - load_leaderboard_split() -> (validated_df, unvalidated_df): reads results.jsonl the same way as load_leaderboard(), splits rows on `validation_status` (defensive: anything not literally "validated" lands in unvalidated, which covers legacy + null + any future-unknown value), runs the same status-aware cell formatting per tier via the new _project_and_format helper. - _project_and_format: shared sort + projection + pending/failed cell tagging so the two tiers produce identically-shaped cells. Legacy default: rows without `validation_status` get "unvalidated" via row-level fill. No resul…0dd7215
-
leaderboard: fetch results.jsonl via direct cache-busted HTTPS
Michael Rabinovichhf_hub_download (even with force_download=True) hands back stale bytes for several minutes after a commit: the local disk cache, revision pinning, and the Hub's CDN-fronted resolve endpoint all add their own staleness windows. End effect: pending and failed rows landed by the submit worker were on the Hub but invisible on the leaderboard tab until you waited it out, defeating the whole point of Refresh. Switch _load_rows_from_hub to a plain requests.get against the dataset's resolve/main URL with Cache-Control: no-cache and a millisecond cache-bust query param. Auth via huggingface_hub get_token() so the same code reads the still-private submissions dataset from the Space (HF_TOKEN secret) and from local dev (~/.cache/huggingface/token) without forking the code path. Adds requests to requirements.txt (was a transitive dep via huggingface_hub; making it explicit so a future upstream change doesn't silently drop it).9fcb684 -
leaderboard: render pending / failed score cells with a status tag
Michael RabinovichBlank score cells on a pending row left the only "this is in progress" cue in the status column, which read as broken to a user expecting at least *some* signal in the score columns. _fmt_pct and _fmt_score are now status-aware: pending rows render the tag "evaluating..." (hourglass prefix) in both score columns, failed rows render "failed" (cross prefix). Completed rows render the score number as before. Matches schema.md's "table renders the three states distinctly (spinner / score / error tag)" description and lines up with the visual pattern other HF leaderboards use while a row is queued.5aee3e5 -
leaderboard: handle null score fields on pending / failed rows
Michael RabinovichThe pending row landed by chunk 3 carries null validity_rate + aggregate_score (correct per the schema). pandas coerces JSON null to NaN on column construction, and _fmt_pct's `if x is None` gate didn't catch NaN; int(NaN) then raised ValueError inside .map(), bringing down load_leaderboard and rendering the leaderboard tab as a generic "Error". Fix: - _fmt_pct: gate on pd.isna(x) (catches both None and NaN), return "" for null. - _fmt_score: matching formatter for aggregate_score (was previously rendered as the raw float; with nulls in the mix it'd show as "NaN" cells). Returns "" for null, "%.4f" otherwise. Pending and failed rows now render with blank cells in both score columns; the row-level status column carries the meaning.00d091d -
submit: write pending row + upload zip on successful validation
Michael RabinovichStep 6 (E) chunk 3. Cheap-sync validation (chunk 2) now also commits to the Hub: on success the handler uploads submissions/<id>.zip and appends a status=pending row to results.jsonl under a process-wide threading.Lock. Eval still stubbed, the row stays pending forever until chunk 4 lands the background worker. Hub-write ordering (after validation passes): 1. Upload submissions/<id>.zip. Unique path per submission, no lock needed (HfApi handles its own commit serialization). 2. Build the pending row: metadata fields + status="pending" + failure_reason=null + submission_blob_url + null score fields, matching the "Pending" regime in cadgenbench-submissions/schema.md. 3. Acquire _HUB_LOCK, download current results.jsonl (treat EntryNotFoundError as empty), append one JSON line, re-upload. Lock held only for the RMW cycle (~1-2s), not for eval time; concurrent submitters serialise here, not in the worker. If step 1 fails the submitter sees a clean "Hub…1748c76 -
refactor: split leaderboard read path into its own module
Michael RabinovichMirrors the shape adyen/DABstep uses on their reference Space (UI assembly in app.py, business logic in a sibling module). app.py drops from 224 lines to 158 by extracting the env-driven repo constants, the LOCAL_RESULTS_PATH + LEADERBOARD_COLS schema, the Hub/local row loaders, and load_leaderboard() into leaderboard.py. The submit handler stays in app.py for now; it'll move to a third module (submit.py) when Step 6 (E) lands the real validation + async eval. No behaviour change: load_leaderboard is imported back into app.py and wired into the same gr.Dataframe / refresh button pair as before. Smoke-tested locally (cadgenbench-space env): app module imports cleanly, load_leaderboard() returns a dataframe with the expected columns.b5ad973