VIDRAFT_LAB

Results from the greedy first-commit run. MMLU-Pro, 250 questions (200 from deciles 6 to 10 on the symmetric cut, 50 from deciles 1 to 5 as a control), T=0, 131K cap, parent UD-Q4_K_XL vs POCKET, both 250/250. Tokens counted with the parent's tokenizer (identical IDs to llama-tokenize).

First commit = the first point the trace states a final option letter (rule and per-question index linked below).

Deciles 6 to 10 (200 questions), parent to POCKET:

• tokens before the first commit: 3,245 to 3,287 (POCKET slightly longer)
• tokens after the first commit: 5,053 to 3,536 (-1,517)
• share of the total saving that falls after the first commit: 112% [67, 215] (above 100% because the pre-commit part is slightly longer in POCKET)
• first commit sits at a median 0.62 vs 0.67 of the trace, inside the thinking block 90.5% vs 92.0%
• wait / verify / double-check type phrases per question: 14.3 to 11.0
• first commit equals final answer: 88.5% vs 89.0%; same first-commit letter in both builds: 82.9%
• accuracy: 76.5% vs 80.0% (single greedy pass, for reference only)

Control, deciles 1 to 5 (50 questions): 304 to 324 tokens, 54 to 59 after commit, same first commit on all 50.

So your re-verification hypothesis holds: both builds take the same road to their first answer, and POCKET spends less time re-checking it afterwards. Short answers rarely re-check, which is why the bottom half does not move.

One assumption did not hold: the traces do not run together for long. They diverge early (median first divergent token 26), so "divergence before or after the commit" does not discriminate here; the before/after-commit lengths carry the result.

Summary: https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF/blob/main/analysis/first_commit/fc_mmlu_analysis_2026-10-04.md
Per-question index (250 rows): https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF/blob/main/analysis/first_commit/fc_mmlu_commit_index.jsonl
Commit rule and token counting: https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF/blob/main/analysis/first_commit/fc_mmlu_commit_rule.txt

SuperGPQA is in. Same 1,000 held-out questions, 4 samples each, parent UD-Q4_K_XL vs POCKET, same settings, 16K generation budget. The only bytes that differ are the 300 Q8_0 tensors.

Accuracy:

• mean of 4: parent 59.10%, POCKET 61.55%, paired difference +2.45 [+1.54, +3.36]
• single sample: 58.70 vs 62.20
• majority of 4: 63.60 vs 65.90 (over answered samples; ties go to the answer that appears first in sample order, seeds 21 to 24)

The budget matters here: 798 of the parent's 4,000 samples reached the 16K cap, vs 675 for POCKET. Much of the gain is POCKET finishing within the budget where the parent runs out. Same budget, more answers delivered.

Length: mean 6,716 to 6,051 tokens (-10% in total), geometric-mean per-question ratio 0.868 [0.848, 0.888].

Symmetric deciles (length ratio / accuracy parent vs POCKET):
D1 0.91 (82/82), D2 0.91 (76/76), D3 0.83 (80/80), D4 0.80 (74/75), D5 0.79 (81/83), D6 0.81 (74/74), D7 0.86 (62/63), D8 0.86 (44/54), D9 0.95 (19/27), D10 1.00 (1/2)

Two differences from MMLU-Pro: the bottom deciles also shrink, probably because even the short SuperGPQA traces are already past the ~700-token step; and D9 to D10 sit at the cap in both builds, so their ratios are compressed toward 1.0. D8 and D9 are where finishing earlier turns into the accuracy gap.

SuperGPQA is in. Same 1,000 held-out questions, 4 samples each, parent UD-Q4_K_XL vs POCKET, same settings, 16K generation cap. The only bytes that differ are the 300 Q8_0 tensors.

Accuracy (mean of 4):

• parent 59.10%, POCKET 61.55%
• paired difference +2.45 points [+1.50, +3.35]
• single sample 58.70 vs 62.20, majority of 4 63.40 vs 65.40

The cap matters here, so up front: 806 of the parent's 4,000 samples hit 16K, vs 688 for POCKET, and most capped samples give no answer (796 vs 675). On the 667 questions where no sample in either build hit the cap, the difference is +0.64 [0.00, +1.31].

So most of the gain is POCKET finishing within the budget where the parent runs out. Same budget, more answers delivered.

Length: mean 6,716 to 6,051 tokens (-9.9%), geometric-mean ratio 0.868 [0.848, 0.888]; on the uncapped questions the token-weighted ratio is 0.842.

Symmetric deciles (ratio / accuracy parent vs POCKET):
D1 0.91 (82/82), D2 0.91 (76/76), D3 0.83 (80/80), D4 0.80 (74/75), D5 0.79 (81/83), D6 0.81 (74/74), D7 0.86 (62/63), D8 0.86 (44/54), D9 0.95 (19/27), D10 1.00 (1/2)

Two differences from MMLU-Pro. First, the bottom deciles also shrink (about 9%), probably because even the "short" SuperGPQA traces here are 500 to 1,000 tokens, past the ~700 step we saw. Second, D9 and D10 sit at the cap in both builds, so their ratios are compressed toward 1.0, and D8 and D9 are where POCKET's earlier finish turns into the accuracy gap.

The greedy first-commit run is next; we will post it here.

That is a clean way to split it, and the re-verification hypothesis fits what we see: short answers barely change, long ones drop by a step.

We will run it as you describe: greedy decoding on the same prompts, parent vs POCKET, on a sample from deciles 6 to 10 plus a small control from deciles 1 to 5. For each question we will report where the first divergent token sits relative to the first answer commit, whether both builds commit to the same first answer, and how many tokens follow that commit in each.

Results will go up here together with SuperGPQA.

Good question. From the symmetric cut, deciles 6 to 10 (geometric-mean ratio, roughly 700 to 78K tokens):

D6 0.872, D7 0.867, D8 0.857, D9 0.741, D10 0.876

So it looks more like a step than a slope. The ratio drops from about 1.00 to about 0.87 around 700 tokens, then stays in the 0.74 to 0.88 range without a steady decline; the longest decile (D10) comes back up to 0.876. D9 is the one dip, and with ~200 questions per decile we would not read a trend into a single bin.

We will check whether the same step shows up on SuperGPQA, where many more traces run past 700 tokens, and include the per-decile table with the cap counts.

Thanks for reconciling the cells. Here are the deciles, with one caveat about how to cut them.

Cut by parent length, as you asked, the top decile does fall: POCKET/parent ratio 0.756 [0.687, 0.829] (geometric mean 0.611), vs 1.008 for the bottom nine combined.

But that cut is biased. Sorting on one build's length puts the questions where that build happened to run long into the top decile, so the ratio there is pulled down by regression to the mean. Cut the same data by POCKET length and the top decile flips to 1.027 (geometric 1.177).

So we sorted on the geometric mean of the two lengths, which treats both builds the same:

• Deciles 1 to 5 (up to ~700 tokens): ratio 0.97 to 1.03, essentially unchanged
• Deciles 6 to 10 (~700 tokens and up): 0.74 to 0.89, i.e. 11 to 26% shorter
• Top decile vs bottom nine: +0.088 [-0.009, +0.187] (mean ratio), -0.041 [-0.12, +0.043] (geometric). No separate drop in the top decile.

So it is neither a top-decile effect nor a flat ~15% shrink. Short answers stay the same length; answers past roughly 700 tokens get 11 to 26% shorter. The typical per-question reduction (geometric mean ratio) is 8.7% [6.2, 11.2]; the 14.5% headline is the token-weighted total, carried by the long questions. Accuracy tracks the parent within ±2 points in every decile.

For SuperGPQA we will report both the parent-length and symmetric cuts, plus how many samples hit the 16K cap in each build, since capping can shrink the ratio. Results when both builds finish.

Thanks, these are the right questions. All numbers are from the same 2,000 MMLU-Pro questions, paired, parent UD-Q4_K_XL vs POCKET, same settings (T 1.0, top-p 0.95, top-k 20, seed 7, 131K cap, llama.cpp b11048).

Q1. POCKET used fewer tokens on 1,088 questions, more on 903, same on 9. Mean difference -628 tokens [-871, -383]. Median difference -16 [-22, -7]. So most short answers barely change; the saving comes from the long reasoning traces.

Q2. Split by correctness (parent / POCKET):

• both right, 1,708 questions: 2,926 to 2,530 tokens, -395 [-589, -195]
• parent right, POCKET wrong, 51: -773 [-3,396, +1,890], not significant
• parent wrong, POCKET right, 45: -2,808 [-6,992, +1,087], not significant
• both wrong, 196: -2,117 [-3,421, -866]
  The shortening is mostly in questions both builds got right. There is no sign that POCKET gets shorter by giving up on the ones it misses.

Q3. Zero runs hit the 131K cap in either build, so capped runs do not move the mean. One parent run produced no extractable answer, none for POCKET.

So on this set: same accuracy, and the same correct answers reached with fewer tokens, with the reduction concentrated in long traces. The SuperGPQA R0 vs R3 comparison on the 1,000 held-out questions is running now; we will post it here when both builds finish.

Thanks for verifying the graft byte by byte. That is exactly the check we hoped someone would run.

You were right about shard 1's metadata. It now reads "POCKET-Darwin-180B" on both Hugging Face and ModelScope (only that 10 MB metadata file changed, the three weight shards are untouched).

On your question: the parent UD-Q4_K_XL build scored 87.95% on the same 2,000 questions, so on MMLU-Pro the parent, R3 BF16 (87.65%) and POCKET (87.65%) are all within sampling noise. The ~100 flips you estimated are that noise floor: at temperature 1.0, any two of our builds flip 85 to 98 of the 2,000 questions, including R1 vs R3 in BF16.

MMLU-Pro is close to saturated for this family, so the RSI delta shows on held-out SuperGPQA (+1.03, lower bound +0.05). Your framing of POCKET vs the parent build as a clean R0 to R3 test, with the quantized experts fixed byte for byte, is a good one. We are running exactly that now on the same 1,000 held-out SuperGPQA questions (4 samples each) and will post the numbers here.

One thing the MMLU-Pro run already shows: at the same accuracy, POCKET answers in 3,694 tokens on average vs 4,322 for the parent build, about 15% shorter.

Thanks! "Pocket" here means it fits on one personal machine instead of an 8-GPU server. For smaller laptops we already have POCKET-35B-GGUF (669K downloads) and POCKET-26B-GGUF (296K downloads), and a smaller Darwin RSI build in POCKET form is on our list. We will post here when it is up.