When Agreement Is Zero: How a Claim Auditor Failed Its Gold.

We built a deterministic claim auditor, wired it into a research apparatus, and almost treated engineering quality as measurement quality. The package passed its gates. The CLI worked. Coverage was high. Frozen rules hashed cleanly. Then we ran a blind human calibration and got near-zero agreement. The machinery was fine. The signal was wrong.

That result is the point of this apparatus lesson. Topic overlap is not support. A citation box is not a verdict. And an automated auditor is another system making claims, not a neutral instrument you can trust because the tests are green.

Engineering quality is not measurement quality

I separate two questions teams often merge.

Engineering readiness asks whether the package runs, freezes its rules and config, emits replayable traces, and stays independent of the generator or bundler under audit. Measurement quality asks whether the verdicts actually track human judgment on support, not just whether the code is deterministic.

Claim Audit Lab v0.2.0 (public tag at commit a673535) cleared the first bar. It is a Python package with a frozen rules file (cal-rules-v1.2.0), a six-value verdict vocabulary, per-claim YAML audit units, apparatus-contracts integration, and the usual verification gates: mypy-strict, ruff, high branch coverage, clean wheel install. From an engineering standpoint, it looked like the right shape for an audit instrument.

The second bar is what blind calibration tests. We scored 98 claims from PILOT-001 v2 audit cells against a blind, single-coder human gold. Exact agreement was 4/98. Cohen’s κ was ≈ −0.006, effectively chance. CAL rated 0/98 claims supported. The human coder rated 80/98 supported.

Those numbers are not a model leaderboard. They are an instrument falsification. The evaluator failed its gold.

What established practice already covers

Measurement theory has long separated reliability from validity and precision from accuracy. Inter-rater agreement tells you whether two scorers are stable relative to each other. It does not tell you whether either scorer is correct against a reference you care about.

NLI and entailment research stress the same gap IR people know: finding a relevant passage is not the same as establishing that the passage warrants the claim. Fact-checking pipelines, legal citation practice, and regulated SME review all train reviewers to ask whether the record answers the exact question, not merely whether a record exists.

The mistake we almost made was familiar in a different costume. We had a rigorous-looking scorer with tests, traces, and frozen rules. We had not yet asked whether its support signal could reach the top of its own verdict ladder on real text.

What we did

PILOT-001 v2 gave us 98 claims across 15 audit cells (four conditions × four models in the calibration design). Human labels were blind and packet-relative: the coder judged each claim against the evidence bundle as supplied, using a fixed rubric. CAL ran with frozen cal-rules-v1.2.0 and no peeking at the gold.

CAL’s architecture was explicit: extract claims, classify type with regex patterns, score each claim against passages with a bag-of-stemmed-terms matcher plus a lexical numeric check, then apply deterministic rules to emit a verdict on the six-value contract (supported, partially_supported, unsupported, contradicted, overstated, not_checkable).

Independence from the Evidence Bundler was already mostly in place. Bundler links could influence admission but not the verdict ladder directly. Calibration packets were blind. The apparatus was designed to catch exactly this kind of failure. It did.

The result in numbers

Metric	Value	How to read it
Blind set size	98 claims (PILOT-001 v2 audit cells)	Instrument check, not a model bake-off
Exact agreement	4/98	Near-total mismatch
Cohen’s κ	≈ −0.006	Chance-level
CAL supported	0/98	Top of ladder never reached
Human supported	80/98	Single coder; packet-relative
Starved at support 0.00	55/98	Matcher returned no usable signal
Of those, supporting passage in bundle	41/55	Evidence was present; matcher could not see it
Non-numeric term-score ceiling vs supported threshold	~0.45 vs 0.80	Structural unreachability
Offline bi-encoder recall on those 41	41/41 at cosine ≥ 0.4	Topic recovery works
Offline NLI probe (support vs non)	~0.97 vs ~0.16 entailment	Directional separation, not published accuracy

I do not read this as “CAL is useless.” I read it as “CAL v0.2’s lexical signal cannot measure support, and the calibration proved it before anyone treated the scores as authority.”

Root cause: lexical support as a category error

The failure was structural, not a tuning miss we could patch with synonyms.

The bag-of-stemmed-terms matcher had a non-numeric term-score ceiling around 0.45, but the rules required 0.80 to emit supported. Text claims could not reach the top of the ladder by design. Numeric claims had a separate path. That is why CAL rated zero supported claims while a human, reading the same bundles, found support in most cases.

Of the 55 claims starved at 0.00 support, 41 had a supporting passage in the supplied bundle that the matcher could not admit. Median lexical coverage against the ideal span was 0.43; none of the 41 cleared the 0.40 admission gate. Synonym and stemming expansion recovered 0/41 in offline probe. The gap was semantic, not lexical.

Offline probes pointed the redesign direction without becoming product claims. A bi-encoder recovered all 41 starved cases at cosine ≥ 0.4, which confirms topic recovery. Cosine similarity alone cannot separate support from mere relevance: supported and unsupported claims overlapped heavily on cosine. An NLI-style entailment probe separated much more cleanly (~0.97 median entailment on supported vs ~0.16 on non-support cases in the directional probe). Entailment is the right family of signal for the verdict layer. Retrieval and entailment are different jobs.

This is the same distinction as A Citation Is Not Support: nomination and topical match live at layer two; support lives at layer three.

What we kept

The falsification would have been worse if the engineering shell had been sloppy. Several choices survived because they made the failure legible.

Frozen rules with byte-identical example regeneration on every change. A hashable rules file means you can replay a verdict and know which policy produced it. The six-value vocabulary stayed: the output space was not the problem. Deterministic numeric tolerance checks were sound in principle; the extractor was the weak point, not the rule shape. Apparatus-contracts integration and the public CLI entry points (claim-audit audit, claim-audit audit-bundle) remain the right boundary. Independence between verifier and bundler is non-negotiable. Blind, packet-relative calibration packets are how you catch a scorer that looks rigorous and measures the wrong thing.

IQ/OQ/PQ-inspired validation structure also stays. Treating an audit instrument like regulated software, with intended use and traceability, was correct even when the underlying signal was wrong.

What we changed

v0.2 retired as a measurement instrument. v1 redesign follows a retrieve → entail → rules stack:

1. Retrieve with a bi-encoder (or equivalent) to fix recall starvation.
2. Entail with an NLI model to separate support from topical overlap.
3. Apply deterministic rules for numeric checks, strength modifiers, negation and absence families, and the glass-box rationale.

Bundler links become corroboration or weak-link signals, not inputs to the verdict. The fuzzy step is confined to candidate admission; the verdict stays inspectable. audit_config becomes the single source of tunables, with config hash recorded in every trace.

I am not claiming v1 is fully measured in this article. Track B work against sealed gold remains separate from whether the tool can run as a first-pass auditor under human review.

Working model: two tracks

The project ADR from 2026-07-28 separates two release tracks that must not share labels.

Track A: tool usable under human review. Engineering is green. Independence is preserved. Known failure modes are documented. A human owns disposition on consequential claims. Sealed gold is not required for permission to run, test, or describe intended use.

Track B: research instrument measured. You have a preregistered, sealed (or otherwise declared) human gold comparison with explicit metrics and limits. This track is for version comparisons, public agreement statistics, and any claim that CAL is a calibrated scorer.

Human gold is for measurement, not permission. DEV sets like PILOT-001 remain adaptation and regression material. They are not confirmatory validation. I will not invent multi-coder ceilings or GxP validation claims here.

Regardless of track, a human (or an explicit adjudication policy) still owns “we will act as if this claim is true.” CAL does not close that loop.

Before trusting an automated claim auditor

Use this checklist before treating any automated support scorer as authority:

1. What claim does each score or verdict express?
2. What signal produces it (overlap, entailment, LLM judge, rules)?
3. Can the top of the verdict ladder actually be reached on real text?
4. Is the auditor independent of the generator or bundler under test?
5. Are rules, config, and model version hashed into every trace?
6. What is the human disposition step for consequential claims?
7. If you have gold: is it adaptation (DEV) or sealed fresh measurement?
8. What known failure modes are documented?

If question 3 fails, you do not have a tuning problem. You have a category error.

Current conclusion

Near-zero agreement was the honest result. It was also the apparatus working as designed: a verifier with frozen rules and blind gold caught a signal that could not measure support. Engineering readiness let us trust the replay. Measurement quality demanded we distrust the verdicts until the signal changed.

The intended-use model follows from that honesty. Claim Audit Lab is an independent first-pass auditor, not ground truth. It produces claim-level verdicts and a replayable trace over supplied evidence. A human still owns disposition on consequential claims. Engineering readiness and independence let us test and use the tool under that rule. Sealed human gold is for measuring the instrument when we want agreement statistics or version comparisons. It is not a license to run, and it is not a substitute for per-claim human review.

An auditor that requires perfect gold before it may run is being treated as an oracle. An auditor that always requires a human on the decision is being used correctly.

Open questions and limits

This lesson reports a single-coder blind pilot, not a multi-coder ceiling study. Human labels carry their own variance. I have not published v1 Track B agreement statistics here. Packet aggregation strategy (max entailment vs multi-passage fusion), negation and absence backstops, and threshold calibration against CAL’s own gold remain open engineering questions.

The NLI probe numbers are directional offline measurements, not product accuracy claims. I do not claim CAL is GxP-qualified or that lexical v0.2 should have shipped as an oracle.

Related work and version notes

Pairs with A Citation Is Not Support and The Evaluator Is Another System Making Claims. Source material includes CAL v0.2 retired prototype record, PILOT-001 v2 calibration, and the 2026-07-28 intended-use ADR (Track A vs Track B). Public engineering reference: v0.2.0 at commit a673535.

Commercial bridge

Evaluator audits, claim-support reviews, and apparatus design work apply this frame when a team needs to know whether its scorer measures support or merely decorates answers with citations. Claim Audit Lab is part of the apparatus I use internally; it is not automatic ground truth. Scoped application to a client’s pipeline remains engagement work.