Jonna Matthiesen

Agreed on the numbers, they're in the article. The table lists 36 blocks × 2048 vs 64 blocks × 1536, and the depth-pinning is called out explicitly: "because the expert shadows the backbone block for block, its depth is not a free parameter." The post one-liner compresses that to "remains at 2.3B," which is a parameter-count claim, not a shape claim. Fair to note 2.28B → 2.42B is a ~6% drift under the rounding.

One thing was held fixed though: the attention interface. 16 × 128 = 2048 in both generations, sized to the backbone's KV head geometry rather than the expert's own hidden width, see the dashed overhang in Figure 2. It's also the part relevant to your question. Width-prune the backbone (hidden, FFN, query heads) and the expert carries over unchanged; depth-prune and architecture breaks. The deployment section draws exactly that line.