kanaria007 PRO

One broader observation from this exchange.

I think the failure pattern we have been uncovering here is structurally very close to a familiar reproducibility-crisis pattern, especially in domains where research and commercial evaluation overlap.

The issue is not only whether someone can rerun code and recover the same headline score.

A stronger question is whether an external reader can reconstruct enough of the evidence-producing path to understand why that score exists:

• how the evaluation surface was constructed,
• how labels and outcome classes were derived,
• what was fixed before outcomes were known and what was selected afterward,
• what apparently separate evidence surfaces share upstream dependencies,
• what comparator was actually available,
• and what error structure was compressed into the aggregate.

A result can therefore be mechanically repeatable while still being difficult to independently interpret, challenge, or audit as an evidential claim.

The research/commercial boundary adds an interesting asymmetry.

Commercial systems can sometimes make these failures easier to falsify than in pure research. There may already be an operational incumbent, production pipeline, failure logs, and a concrete replacement claim against which the published result can be tested.

The resolver here is a good example. Once the incumbent is treated as an eligible operational comparator rather than background infrastructure, it becomes a very strong external falsifier.

But the same commercial context can make correction harder after a claim has propagated into product positioning, customer expectations, internal roadmaps, investment narratives, or other institutional commitments.

So one possible structural failure mode is:

falsifiability increases while correctability decreases.

I do not mean that as a claim about the motives of this particular publisher. It is a more general institutional risk that this example makes unusually visible.

The broader SI work already separates the corresponding responsibilities.

274 handles the benchmark-publication surface: comparability, disclosure, material evidence dependency, and claim boundaries.

The Continuous Audit and Evidence Pipeline Supplement handles preserved evidence, verifier handoff, archive linkage, and replayable audit-pipeline lineage.

The Assurance, Claim Publication, and External Reliance Supplement handles the downstream claim lifecycle: honest benchmark disclosure, bounded external reliance, claim status, re-certification, narrowing, suspension, and revocation.

The research-specific work applies the same structure more directly. 049 treats research as a traceable design → preregistration → execution → analysis → replication lineage, while 569/570 extend that into replication, negative results, claim retirement, correction, retraction, and downstream correction propagation.

So the label-reconstruction issue you found is real, but I see it as the point where the benchmark-publication problem meets the more general evidence-reconstruction problem, rather than as another benchmark-specific rule that 274 itself should absorb.

What I find especially useful about this case is that it makes those older abstractions much less abstract.

We started with a 90.0% headline and a trivial-baseline question. Following the evidence backward exposed the operational comparator, then shared evidential dependency, then a label-derivation gap.

That is very close to the core reproducibility lesson:

the published number is only the end of the chain; what has to remain inspectable is the chain that made the number defensible.

Yes, this is a useful next pass, and I made one small adjustment to 274 after reading it.

Your rerun makes the specific dependency much sharper. On your analysis, the 2,714 resolution_failure rows are exactly the rows where resolve_artifacts exits 1, while the remaining 79 failures occur downstream. So I agree that, in this artifact, the 97.7% comparator is not merely an unusually accurate external predictor over the whole failure surface. Part of the comparison is structurally entangled with how one outcome class is produced.

I did not, however, turn fp = 0 or that exact stage ladder into a new 4.5 detection rule.

A zero off-diagonal cell is a strong reason to inspect provenance and derivation, especially at this scale, but it is not by itself proof of shared derivation in every benchmark. Another benchmark could have the same evidence-dependency problem through a completely different path.

So I kept 4.5 at the more general level:

identify material provenance and dependency, and do not count coupled evidence surfaces as independent corroboration unless their provenance supports that interpretation.

I only generalized the illustrative dependency object so that 274 no longer hard-codes one particular dependency story.

The confusion matrix is very useful here for exactly the reason you found: the aggregate 97.7% hides an error structure that makes the dependency inspectable. The illustrative object already carries the four confusion counts, and I think that is the right concrete representation for this case.

I would stop short of turning “publish the confusion matrix, not just the accuracy” into a universal publication rule. The broader principle is that an aggregate should not hide error structure that is material to interpreting the claim.

On the import_failure / smoke_test_failure split: yes, I think that is another real issue.

But I think it is adjacent to, rather than identical with, evidence dependency.

The question becomes:

can an external reader reconstruct how the published outcome label was derived from the published artifact and its explicitly bound evidence references?

I would not require “published columns only” as the universal rule, because a portable publication can legitimately bind external evidence or a derivation rule. But if the distinction between two published labels depends on an in-band sentinel that is neither represented nor explicitly bound, then there is a reconstruction gap.

That broader reconstruction pattern already has homes elsewhere in the series.

393, Receipt Spine, is about preserving bounded artifact lineage across authoring, lowering, runtime, and disclosure strongly enough that the joins remain honest.

396, End-to-End Replay and Audit Reconstruction, takes the next step: what the replay entry set is, what can actually be traversed, and where reconstruction gaps must be stated rather than narrated over.

Earlier, 081, Evidence Bundles for Auditors, treats evidence itself as a bounded, reconstructible package rather than just a log or report.

So I would keep the boundary here:

274: what was measured, what is comparable, what dependencies materially affect the evidential reading, and how far the publication claim may go.

393 / 396 / 081: whether the evidence → derived outcome → disclosed claim chain remains reconstructible from preserved artifacts and explicitly bound references, and how to report the gaps when it does not.

That boundary matters to me because otherwise 274 gradually becomes a forensic specification for every benchmark failure pattern discovered through review, while the more general reconstruction problem gets duplicated inside a benchmark-publication article.

And yes — the 79 rows are doing a lot of work here. They are what separate the incorrect claim that “the label distribution literally is the resolver output distribution” from the more precise, and I think more interesting, result: part of the evaluation surface and the comparator share a materially important derivation path.

Yes, I think this found a real missing axis, and I updated 274 for it.

I ended up giving it a separate 4.5 rather than stretching 4.4 further, because the issue is broader than baselines:

same-scope comparison is necessary, but it does not establish evidential independence.

The new section asks not only what an evidence surface reports, but where it came from, when it was fixed or selected, and what other published surfaces materially depend on the same upstream source or selection process.

So in your example the relevant questions are now explicit:

where did the split composition come from?

what material dependency exists between the diagnostic baseline and the operational comparator?

I also generalized the failure mode beyond this split. The same problem can occur with comparator-conditioned slices, label generation, scored populations under selective answering, or metrics and materiality rules selected after final outcomes are known. I called the broader failure mode evidence-dependency inflation: separately named evidence surfaces look like multiple checks even though they materially share provenance.

One arithmetic correction, though. 2,714 / 3,432 is about 79.1%, not 81.4%. The 81.4% constant-FAIL baseline comes from the 2,793 / 3,432 true failures. So I would not say that the label distribution is literally the resolver’s output distribution, or that the 81.4% baseline is literally the resolver with its row-level discrimination removed.

But I think your structural point survives that correction.

Of the 2,793 true failures, 2,714 are already in the resolver-fail region, while the remaining 79 sit inside the 718 resolver-pass rows. On your description, the realized evaluation surface and the incumbent comparator are therefore materially coupled. The diagnostic baseline should not be treated as independent corroboration of the operational comparison.

I would also stop slightly short of calling the 81.4% diagnostic worse than uninformative. It still tells us something real: a constant FAIL prediction gets 81.4% on this realized split. What changes is its evidential role. Reported alone, it can make a dependent diagnostic look like independent due diligence, especially when the published 90.0% clears it while still losing to the 97.7% operational comparator.

The update therefore does not prohibit dependent evidence. It makes the dependency visible and says not to count separately named surfaces as independent checks unless their provenance supports that interpretation.

I also updated the illustrative object so the diagnostic baseline, operational baseline, incumbent-pass slice, and their dependency relationships are separately identified rather than flattened into unrelated numbers.

So yes: “where did this split composition come from?” is now explicitly part of the publication discipline. I think that is the right generalization of the failure mode you found.

This is a very useful follow-up. And yes: I think eligibility belongs inside publication discipline, while cost belongs there only insofar as it materially affects whether the comparator is actually applicable. Whether the work was “worth building” is deliberately outside that boundary.

4.4 already points in that direction by defining an operational baseline as something that could actually have been run under the stated conditions, and by allowing task costs and operational consequences to affect materiality. So I do not think this needs a new posture rule.

The publication question is:

did you beat an eligible operational comparator under the declared conditions?

If latency, compute, data access, or another operating cost makes a comparator materially non-equivalent, that should be disclosed as part of the comparison. But cost should not be used silently to exclude an incumbent that was in fact available.

The next question

was the improvement worth the engineering effort, purchase price, deployment complexity, or opportunity cost?

is a separate value and decision surface. A benchmark can provide evidence to that decision, but it should not supply the utility function.

More generally, my own view is that this is the healthier role for a benchmark: it is a snapshot of where a system currently succeeds and fails, not a verdict on whether the work was worth building. Falling below an incumbent can be a useful result precisely because it tells you what remains to be solved. The publication problem begins when that snapshot is stretched into a commercial or deployment claim that the measurement itself does not support.

Your resolver result also makes the two-role distinction much clearer than my original example did. The 81.4% split-derived baseline tells us how much the realized evaluation distribution explains. The 97.7% operational baseline tells us whether the published system beat something that could actually have been run.

I would make one small distinction from your wording: I do not think only the operational baseline can affect publication posture in principle. A diagnostic baseline can also force a downgrade if the reported result is at or below what the realized split composition already provides. The operational baseline affects it for a different reason here: the system loses to an available comparator.

And with the 0.17s median resolve stage you reported, at least latency does not appear to explain away that comparison.

Thanks again for rerunning this against the revised section. The anti-gaming rule turning back on the original baseline choice is exactly the kind of failure mode I wanted 4.4 to expose.

You’re right. 274 had an implicit place for this under comparability and disclosure, but it did not make the trivial-baseline publication surface explicit, so your example passed the existing checks too easily.

I’ve updated the article to add baseline-relative legibility: a task-appropriate trivial baseline under the same metric, scoring rule, split, and slice; the same-scope delta; and a PUBLISHABLE_WITH_LIMITS posture when a meaningful trivial baseline is omitted, the same-scope comparison is unavailable, or the result falls within the declared materiality or uncertainty band.

I also separated two cases that are easy to conflate:

• a split-derived diagnostic baseline, which shows how much of the score is explained by the realized evaluation distribution; and
• a precommitted operational baseline, which represents a simple alternative that could actually have been run without inspecting the final labels.

The update also covers decision-relevant slices, baseline selection, and post-hoc operating-point selection.

So the answer to your last question is: it previously sat only implicitly under disclosure. It now has an explicit slot.

Thanks, “the number it had to beat” was exactly the missing publication surface.