CA incident 2026-09-08

Rotation mn-ca-002 → mn-ca-003 · postmortem public · every claim links its registry · 2026-09-25

Exposure: CONFIRMED Third-party misuse: none observed Revoked same-day · fail-closed 4 Rekor anchors

On 2026-09-08 MarketNow rotated its production CA key mn-ca-002 to mn-ca-003 and published the revocation the same day. This page is the postmortem the incident deserves: what happened, what it affected, what the response was, and what it proved about the revocation machinery. Nothing here is a claim without a link.

Detection

During a key-hygiene review of public surfaces on 2026-09-08, private key material for the CA key mn-ca-002 (active since 2026-08-12) was found committed to a public repository. The revocation registry records the finding in the key's own words — this is the live OCSP response, fetched from production:

"Private key material was found committed to a public repository. Any signature that only verifies under this key must be treated as untrusted."

Source: GET /api/ocsp?kid=mn-ca-002 → status KEY_COMPROMISE, recommendation DENY (fail-closed).

Detection was manual review, not automated scanning — raw Ed25519 private keys are not covered by default secret-scanning patterns. That fact is lesson #1 below.

Timeline

2026-07-16 — ca-key-001 active (first generation CA key).
2026-08-12 — scheduled rotation: ca-key-001 retired, mn-ca-002 activated.
2026-09-08 (detection) — key-hygiene review finds mn-ca-002 private key material committed to a public repository. Exposure treated as confirmed (worst-case).
2026-09-08 (same day) — mn-ca-002 revoked (KEY_COMPROMISE), mn-ca-003 activated, revocation published, OCSP fail-closed answers DENY.
2026-09-08T21:03:33Z — first Rekor transparency anchor integrated for the rotation digests.
2026-09-09T17:24:25Z — Rekor entry #4 anchors the revocation registry itself (logIndex 2771735480, entry UUID 108e9186e8c5677a…).
2026-09-16/17 — registry signing key rotated mn-revoc-001 → mn-revoc-002 during scheduled CRL renewal (the CA key never signs revocations — delegated by design).
2026-09-25 — this postmortem published; repository wording aligned with the registry record.

Scope

Item	Status	Detail
`mn-ca-002` (CA key)	REVOKED	Active window 2026-08-12 → 2026-09-08. Any ATC that only verifies under this key must be treated as untrusted (fail-closed).
ATCs issued under mn-ca-002	Re-issued	Production ATCs re-issued under mn-ca-003. Historic superseded ATCs (3, from July) remain in the CRL with reason SUPERSEDED.
`mn-ca-003` (successor)	ACTIVE	Active since 2026-09-08. Never published, never committed.
Test CAs (`ca-test-1`, `ca-test-2`)	Unrelated	Separate throwaway keys for the conformance suite — intentionally public, labeled TEST ONLY, never production CAs.
Revocation registry key	Rotated	mn-revoc-001 → mn-revoc-002 (2026-09-17). The registry is signed by a delegated key, so CA compromise cannot forge revocations.

Evidence (all public, all verifiable)

Artifact	What it proves	Where
Signed CRL (MNR-CRL-1.0)	mn-ca-002 status `retired-compromised` since 2026-09-08; Ed25519 signature over RFC 8785 JCS payload; verification steps included	/api/crl · crl.json
Live OCSP responder	`kid=mn-ca-002` → KEY_COMPROMISE / DENY / fail-closed, nonce anti-replay	/api/ocsp?kid=mn-ca-002
Rekor anchors ×4	Rotation digests + the CRL itself anchored in Sigstore's public append-only log (logIndex 2762061972, 2764017355, 2764479676, 2771735480)	anchors · rekor.sigstore.dev
Registry key + history	mn-revoc-002 public key, SPKI base64, rotation history	registry-key.json
Repository documentation	Revocation semantics, states, fail-closed policy	revocations README

Revocation mechanics exercised

This was not a drill: the incident exercised the full revocation path that roadmap v5.1 shipped. The CRL is append-only and signed by a delegated registry key (mn-revoc-002), not by the CA — so a compromised CA key cannot forge or suppress revocations. The OCSP responder resolves per-subject status (card_id or kid) against the signed CRL, embeds the CRL signature so clients can verify independently, and answers UNKNOWN → DENY (fail-closed) for anything it cannot resolve. Inside the rotation window, a correctly implemented verifier therefore denied mn-ca-002 signatures instead of trusting them — the safe failure.

Impact

Exposure: confirmed. Private key material for mn-ca-002 was committed to a public repository (registry record, quoted above under Detection).
Third-party misuse: none observed. No signatures verifying under mn-ca-002 were observed after rotation. The response nonetheless treated the exposure as a full compromise — worst-case handling, per the fail-closed design.
Bounded blast radius. mn-ca-002's active window was 2026-08-12 → 2026-09-08; signatures from that window that only verify under this key are untrusted by policy. Successor mn-ca-003 was never exposed.
Zero buyer/seller impact. The CA key signs agent trust cards, not marketplace transactions; the catalog, payments and scans operated continuously.

Resolution

mn-ca-003 active since 2026-09-08; production ATCs re-issued under it.
mn-ca-002 revoked with reason KEY_COMPROMISE, published same-day in the signed CRL and served live by OCSP.
Rotation and revocation anchored in Sigstore Rekor (4 log entries) — third parties can verify the response, not just take our word for it.
Registry signing key rotated on schedule (mn-revoc-001 → mn-revoc-002), proving CA-key compromise and registry integrity are decoupled.
Repository wording aligned with the registry record (2026-09-25): earlier README phrasing said "suspected compromise (no confirmed leak)" — the registry's record is the authority and says exposure was confirmed; the README now matches it.

Lessons learned

1 — Secret scanning does not catch raw Ed25519 keys. Detection came from a manual key-hygiene review. Default scanner patterns target tokens/PEM, not bare key material. Mitigation: treat any private-key material visible in a public context as compromised — rotate first, investigate second. That is exactly what happened here, and it is the correct default for trust infrastructure.

2 — The machinery worked under a real incident. CRL publication, OCSP fail-closed resolution and delegated registry signing all ran in production, same-day, under a real KEY_COMPROMISE — not in a test harness. Every trust claim about the revocation system now has an incident behind it.

3 — Transparency anchoring converts incidents into evidence. Because the rotation and the CRL are anchored in Rekor, the response itself is independently verifiable by strangers. An external auditor does not need to believe the postmortem — they can check the log entries.

4 — Postmortems must use the registry's language. The README softened the event to "suspected compromise (no confirmed leak)" while the registry recorded a confirmed exposure. Both surfaces now say the same thing. For a trust company, wording drift is an incident class of its own — the fix is to make the signed registry the single source of truth for incident wording, and human prose must quote it.

Verify this postmortem yourself

# Live OCSP status of the revoked key
curl -s 'https://marketnow.site/api/ocsp?kid=mn-ca-002' | jq '{status, recommendation, evidence}'

# The signed CRL (status of every CA key)
curl -s https://marketnow.site/api/crl | jq '.crl.ca_key_status'

# Rekor entry #4 — the revocation registry, anchored
# log: https://rekor.sigstore.dev · logIndex 2771735480
# local verification: /uta/conformance/anchors/ → verify-rekor.mjs