Adjudication

After this lesson, the reader should understand:

• 01Separate detecting a divergence from judging it.
• 02Explain why outcome evidence, not the divergence itself, determines harm.
• 03Apply a defensible standard before asserting a causal claim — and recognise when to withhold one.

The tightening that saved nothing

Officers escalate at $48,000 against a documented $50,000. Is the drift harmful?

The divergence alone cannot answer it. Escalating earlier is more cautious, and caution is not self-evidently correct: it consumes review capacity, delays suppliers, and may deliver no measurable reduction in loss. The question is empirical — join the outcomes to the decisions, estimate the difference the tightening actually made, and adjust for the covariates that plausibly drive both the escalation and the outcome^[1]. Two disciplines apply. First, when many rules are examined at once, some will appear significant by chance alone, and the threshold must be corrected to account for that^[2]. Second, when the effect is small and the sample thin, the honest verdict is inconclusive — a category that must remain available, because a method that always returns an answer is not measuring anything. The NOMOS reference implementation runs precisely this sequence: it joins realised outcomes to the decisions that produced them, picks the covariates most associated with the outcome (Pearson |r| ≥ 0.05) as confounders, and estimates the effect with a causal model rather than a difference in means. Every estimate is also put through a refutation suite — placebo treatment, a random common cause, a data subset — recorded as provenance on the finding, not used to gate the result^[3]. Significance requires both adequate power for the observed effect size and a Benjamini–Hochberg FDR-adjusted q-value at or below 0.10 across all rules tested in the run, so the verdict is corrected for multiple testing rather than read off a single p-value. Where power is short or the effect is null, the classification is inconclusive, not a forced call. That verdict's calibration is measured against held-out outcomes and gated: the study fails if expected calibration error exceeds 0.06 (recent runs sit near 0.025). The strongest objection is that none of this can establish harm. Observational data, however carefully adjusted, cannot rule out a confounder nobody measured — perhaps the officers who escalate early also handle the riskiest suppliers, for reasons recorded nowhere — and only a randomised trial settles causation. The objection is correct about the ceiling and wrong about the conclusion. Adjustment removes the confounding one has measured; refutation tests check that the estimate does not collapse under perturbation; neither reaches what is unrecorded. The verdict is therefore causal under stated assumptions, and it should say so. What makes it useful is sensitivity analysis: asking how strong an unmeasured confounder would have to be to explain the estimate away^[4]. A finding that only a confounder of implausible strength could erase is worth acting on; one that a modest hidden factor could erase is a hypothesis, and should be reported as one. Randomising procurement thresholds to test them is rarely possible; quantifying how fragile the observational answer is always is. The covariate rule deserves the same scrutiny. Choosing adjustment variables by their association with the outcome is a practical heuristic, not a guarantee. The reference implementation excludes the rule's own fields and identifiers, but a variable that is itself affected by the escalation decision — the time a review took, say — can be strongly associated with the outcome and still be the wrong thing to adjust for, because it sits on the causal path rather than before it^[5]. Which variables are legitimate controls is a question about the causal structure of the process, and it is answered by knowing the process, not by correlation alone.

• What standard of evidence should precede telling an institution its practice is harmful?
• Who decides which outcome counts as the bad one?
• Is a beneficial divergence a reason to amend the document?
• What is the cost of reporting inconclusive — and the cost of not having the option?
• How strong would an unmeasured confounder have to be before you stopped believing a finding of harm?

No responses yet. Be the first to answer one of the questions above.

Set the standard before you look.

1. 01Take one divergence you have already found.
2. 02Name the outcome that would make it harmful, in the institution's own terms.
3. 03List the covariates that could produce a spurious association.
4. 04State, in writing and in advance, the evidence threshold you will accept.
5. 05Decide in advance what result would make you report inconclusive.
6. 06For each covariate, say whether it is fixed before the decision or could be changed by it — and drop the latter from adjustment.