# Evaluating With Leakage Gates

Evaluate an OpenMed de-identification or clinical NER model against the leakage-first...

- Repository: https://github.com/maziyarpanahi/openmed (5755 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-10-31
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add maziyarpanahi/openmed --skill evaluating-with-leakage-gates -g -a claude-code -y`
Codex: `npx skills add maziyarpanahi/openmed --skill evaluating-with-leakage-gates -g -a codex -y`
Cursor: `npx skills add maziyarpanahi/openmed --skill evaluating-with-leakage-gates -g -a cursor -y`
Gemini CLI: `npx skills add maziyarpanahi/openmed --skill evaluating-with-leakage-gates -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add maziyarpanahi/openmed`

## Description

Evaluate an OpenMed de-identification or clinical NER model against the leakage-first release gates G1a through G8, which gate releases on residual PHI leakage rather than on F1. Use when the user wants to run the OpenMed eval harness on a synthetic golden set, decide whether a de-id model is RELEASABLE or QUARANTINED, enforce direct-identifier recall floors, require zero critical leakage, fit calibration thresholds, or produce a signed gate report. Trigger on \"release gate\", \"leakage\", \"is this model safe to ship\", \"G1a\", \"G3\", \"quarantine\", \"recall floor\", or \"calibration thresholds\" in an OpenMed de-id context.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Evaluating with Leakage Gates

OpenMed's release gates answer one question: **did any PHI leak?** A de-id model
with a beautiful F1 can still leak a single SSN, and that one leak is a HIPAA
breach. So `openmed.eval` gates on *residual leakage* and *per-label recall
floors*, not on aggregate F1. The candidate is either `RELEASABLE` or
`QUARANTINED`; there is no partial credit.

## When to use this skill

- You have a candidate de-id or PII model and need a ship / no-ship decision.
- You want to run the benchmark harness over a **synthetic** golden suite.
- You need to enforce direct-identifier recall floors and `critical_leakage == 0`.
- You need calibration thresholds (`thresholds.json`) before the gate will pass.
- You want a signed, reproducible gate report for governance.

This is the flagship eval skill. For a pure NER scorecard see
`benchmarking-clinical-ner`; for CI wiring see `gating-deid-leakage`.

## The gates (G1a, G8)

| Gate | Checks | Floor / rule |
| --- | --- | --- |
| **G1a** | Direct & quasi identifiers (PERSON, EMAIL, PHONE, SSN, ID_NUM, DATE_OF_BIRTH, ...) | recall ≥ 0.990 (v1.6) / 0.995 (v2.0); strict-no-leak policies raise the floor |
| **G1b** | Structured secrets (API_KEY, ACCOUNT_NUMBER, CREDIT_CARD, IBAN) | recall ≥ 0.995 |
| **G2** | Free-text names/locations/dates | recall ≥ 0.980 (v1.6) / 0.990 (v2.0) |
| **G3** | Critical leakage (SSN, CREDIT_CARD, CVV, API_KEY, PIN, IBAN, ...) | count **must be exactly 0** |
| **G4** | Quantized recall delta vs fp parent | within INT8 / INT4 limits |
| **G5** | Latency & RAM vs device tier budget | p50/p95/RAM under tier budget |
| **G6** | p50/p95 latency documented | must be present and finite |
| **G7** | Baseline regression | recall drop ≤ 0.002/label; leakage ≤ soft ceiling 0.005 and ≤ steward target; no leakage regression vs last-green |
| **G8** | Span integrity | predicted spans validate (no overlaps/out-of-range) |

The rest: https://github.com/maziyarpanahi/openmed/blob/HEAD/skills/evaluating-with-leakage-gates/SKILL.md