Evaluating With Leakage Gates
Evaluate an OpenMed de-identification or clinical NER model against the leakage-first...
Install
npx skills add maziyarpanahi/openmed --skill evaluating-with-leakage-gates -g -a claude-code -y- Repository
- maziyarpanahi/openmed5.8k
- Category
- Engineering
- Updated
- 4d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 1,829 tokens
- Runs scripts
- No, instructions only
What it does
Evaluate an OpenMed de-identification or clinical NER model against the leakage-first release gates G1a through G8, which gate releases on residual PHI leakage rather than on F1. Use when the user wants to run the OpenMed eval harness on a synthetic golden set, decide whether a de-id model is RELEASABLE or QUARANTINED, enforce direct-identifier recall floors, require zero critical leakage, fit calibration thresholds, or produce a signed gate report. Trigger on \"release gate\", \"leakage\", \"is this model safe to ship\", \"G1a\", \"G3\", \"quarantine\", \"recall floor\", or \"calibration thresholds\" in an OpenMed de-id context.
Evaluating with Leakage Gates
OpenMed's release gates answer one question: did any PHI leak? A de-id model
with a beautiful F1 can still leak a single SSN, and that one leak is a HIPAA
breach. So openmed.eval gates on residual leakage and per-label recall
floors, not on aggregate F1. The candidate is either RELEASABLE or
QUARANTINED; there is no partial credit.
When to use this skill
- You have a candidate de-id or PII model and need a ship / no-ship decision.
- You want to run the benchmark harness over a synthetic golden suite.
- You need to enforce direct-identifier recall floors and
critical_leakage == 0. - You need calibration thresholds (
thresholds.json) before the gate will pass. - You want a signed, reproducible gate report for governance.
This is the flagship eval skill. For a pure NER scorecard see
benchmarking-clinical-ner; for CI wiring see gating-deid-leakage.
The gates (G1a, G8)
| Gate | Checks | Floor / rule |
|---|---|---|
| G1a | Direct & quasi identifiers (PERSON, EMAIL, PHONE, SSN, ID_NUM, DATE_OF_BIRTH, ...) | recall ≥ 0.990 (v1.6) / 0.995 (v2.0); strict-no-leak policies raise the floor |
| G1b | Structured secrets (API_KEY, ACCOUNT_NUMBER, CREDIT_CARD, IBAN) | recall ≥ 0.995 |
| G2 | Free-text names/locations/dates | recall ≥ 0.980 (v1.6) / 0.990 (v2.0) |
| G3 | Critical leakage (SSN, CREDIT_CARD, CVV, API_KEY, PIN, IBAN, ...) | count must be exactly 0 |
| G4 | Quantized recall delta vs fp parent | within INT8 / INT4 limits |
| G5 | Latency & RAM vs device tier budget | p50/p95/RAM under tier budget |
| G6 | p50/p95 latency documented | must be present and finite |
| G7 | Baseline regression | recall drop ≤ 0.002/label; leakage ≤ soft ceiling 0.005 and ≤ steward target; no leakage regression vs last-green |
| G8 | Span integrity | predicted spans validate (no overlaps/out-of-range) |
