Skip to main content

Conformal Coverage

What it is. A distribution-free, finite-sample guarantee that a regulator can verify offline.

The theorem​

For any α ∈ (0, 1) and any predictor f, given exchangeable calibration data, the conformal procedure constructs a prediction set C(x) such that:

P(Y ∈ C(X)) ≥ 1 − α

This holds without distributional assumptions, without model assumptions, and with finite-sample validity (no asymptotics). Reference: Vovk, Gammerman, Shafer (2005); Angelopoulos & Bates (2023).

This is the only confidence construct in our entire system that comes with a theorem rather than a heuristic.

What we actually compute​

For each (organization, agent, decisionType) tuple:

  1. Compute nonconformity scores from the calibration corpus: s_i = |rawScore_i − outcome_i|
  2. Take the (1 − α)(n + 1)/n empirical quantile of {s₁, …, s_n}. This q̂_α is the conformal radius.
  3. For a new query score s, the prediction set is {y : |s − y| ≤ q̂_α}. For binary outcomes this resolves to {0}, {1}, {0,1}, or rarely ∅.
  4. Verify empirical coverage on a held-out test set: count how often the true outcome falls inside the prediction interval. Should be ≥ (1 − α).

The badge​

Adjudon publishes an embeddable SVG badge at:

GET /api/calibration/{orgId}/agents/{agentId}/badge.svg?alpha=0.1

The badge is publicly cacheable for 24 hours — designed to be embedded as <img> on your compliance / trust page. It reports:

  • α (target miscoverage)
  • Empirical coverage (verified on past 90 days)
  • Verification date (when last computed)

Color-coded:

  • Green — coverage meets or exceeds (1 − α)
  • Amber — coverage within 5 percentage points below target
  • Red — coverage significantly below target (drift / refit overdue)

Caveat​

Per Angelopoulos & Bates 2023 §1.3: the coverage guarantee is marginal, not conditional. With a fixed calibration set, coverage will fluctuate from one batch to the next. Customers should not interpret "α = 0.1 coverage" as a per-decision guarantee — it is an average property over many decisions.

The Adjudon white paper (in preparation, target FTML 2026) explicitly documents this distinction for regulators.

Plan tier access​

The conformal coverage API (GET .../conformal) is gated to Scale plan and above (auditLog feature flag). The embeddable SVG badge endpoint is publicly cacheable — no authentication required on the URL — so the badge can be embedded as <img> on your compliance/trust page.

The badge data exposed (organization passes coverage X% at α=Y, with sample size Z) is intentionally public. Cardinal Rule #1 (cross-org data isolation) is preserved because each badge URL contains its own organization id and renders only that organization's coverage stat.