uta / conformance runner-tests /
The runner is the tested thing
Three oracles, one suite
Until v1.3.2 the reference scorer was our code: a stranger could run it, but had to trust it — the same asymmetry this suite exists to kill. These tests turn the runner from a trusted component into a tested component.
| Oracle | What it pins | Where it lives |
|---|---|---|
| Bytes | sha256(score-runner.mjs) — the exact runner bytes | answer-key.json, anchored in Rekor (entries #2 and #3) |
| Behavior | The 8-runner separation matrix + the reference-mode verdict, row by row, failure lists included | answer-key.json — recorded 2026-09-09, valid through 2027-08-19 |
| Teeth | 10 known-bad runner variants — each must DIVERGE from the key | mutants.json — deterministic byte patches, digests pinned in the key |
A key that nothing can fail is not a test. Every mutant here is caught — rebuild any of them and run the suite: it flags it.
v1.3.3 — what the second bug report forced
The two fixes, landed
- premature-atc (vector #14): a properly-signed, anchored, active card whose only defect is a FUTURE
issued_at(2030-01-01). The exact mirror ofexpired-atc; runners checking only the upper bound accept it. - Two-sided window:
issued_at ≤ NOW < expires_at— enforced by generator and reference runner alike. - Derived ground truth: generated-card expectations derive from card bytes and pinned anchors, never from a sidecar default; a lying sidecar is a hard FATAL.
# from the conformance directory node runner-tests/runner-tests.mjs # regenerate the key from a runner's live behavior node runner-tests/runner-tests.mjs --record
- README.md — full methodology
- answer-key.json · mutants.json · runner-tests.mjs
- ← conformance suite root