Sentinel benchmark

Decision-level TP/FP/F1 computed from public, verifiable inputs · 2026-09-25 · machine version

P 0.75 R 1.00 F1 0.857 n = 18 (small — read the limitations)

This benchmark measures Sentinel where it matters for trust: decisions (quarantine vs. no-quarantine), not raw rule hits. Every positive is a signed ledger record or a documented incident; every negative is a first-party package you can download yourself. No number in the matrix is simulated — each links to its evidence.

0.75

Precision
TP / (TP + FP)

1.00

Recall
TP / (TP + FN)

0.857

Total labeled decisions

Confusion matrix

Pred: quarantine

Pred: pass

Actual: malicious

True positives

False negatives

Actual: benign

False positives

True negatives

Positives — every record verifiable

Case	Label	Outcome	Class	Evidence
Prompt-injection submission	malicious	quarantined	TP	qd_2026_08_15_001.json (L1.7/L1.8/L1.9 fail, sha256-signed)
Crypto wallet with hardcoded mnemonic	malicious	quarantined	TP	qd_2026_08_16_002.json (L1.6 fail)
Weather API — API key in test fixtures	benign (fixture)	quarantined pending review	FP	qd_2026_08_17_003.json (self-declared false positive, appealable)
Trojan in nested zip (issue #9, July 2026)	malicious	caught by L1.7 + sandbox	TP FN pre-L1.7	lib/sentinel-l17.mjs — the incident predates L1.7; current pipeline has the regression vector

Negatives (TN=14): MarketNow first-party npm packages — audited, 0 vulnerabilities, published from source. None was quarantined. Verify per-package results at /api/certification-scans.json (own=true).

Flag-level analysis on the benign corpus

Scanning known-benign packages is where false-positive discipline shows. Of 14 first-party packages, error-severity findings fired on 4:

Package	Rule	Honest classification
`[email protected]`	MCP-TP-002	False positive — UTF-8 BOM (EF BB BF) at the start of package.json reads as a zero-width char. Confirmed in the published tarball. Remediable.
`@marketnow/[email protected]`	MCP-TP-002	False positive — same BOM class.
`@marketnow/[email protected]`	MCP-CI-001 / AC-002	True pattern, benign context — real `execSync(openssl…)` in build code and detection regexes shipped in dist. The scanner is right about the pattern; the context is our own build.
`@marketnow/[email protected]`	PI/AC family	True positive by design — the package contains the detection signatures themselves ("ignore previous instructions" is rule PI-001's corpus).

Error-severity FP rate on the benign corpus: 2/14 = 14.3% — published, not hidden. WARNING-level rules (SSRF localhost heuristics, readFileSync) are high-sensitivity by design.

How to reproduce

# 1. Fetch the ledger (SHA-256 per record)
curl -s https://marketnow.site/api/quarantine

# 2. Fetch raw L2 scans (2,839 packages, per-rule findings)
curl -s https://marketnow.site/api/certification-scans.json | jq '.stats'

# 3. Per-package verdict for any flagged own package
curl -s 'https://marketnow.site/api/certification?package=@marketnow/trust-core'

# 4. Rules (29): npm @marketnow/sentinel-rules or repo
#    uta-monorepo/packages/sentinel-rules/rules/rules-lite.json

Limitations — read before quoting this benchmark

This is a transparency artifact, not statistical power.

n = 18 decisions: confidence intervals are wide. The benchmark proves the evidence chain, not scanner superiority.
Corpus bias: negatives are first-party packages (cleaner than npm average).
Recall = 1.0 only over known, public incidents — a novel attacker is not represented. The July trojan was a real FN of the pre-L1.7 pipeline; L1.7 exists because of it.
Lite engine: 22 exact regex + 7 structural approximations; full AST runs in Semgrep (rules published).