# Benchmark Pii Recall

Benchmark an OpenMed PII model with synthetic gold spans and report label-aware...

- Repository: https://github.com/maziyarpanahi/openmed (5755 stars)
- Category: AI and agents
- License: Apache-2.0
- Last updated: 2026-10-31
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -g -a claude-code -y`
Codex: `npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -g -a codex -y`
Cursor: `npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -g -a cursor -y`
Gemini CLI: `npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add maziyarpanahi/openmed`

## Description

Benchmark an OpenMed PII model with synthetic gold spans and report label-aware exact-span and grapheme recall without emitting identifier surfaces. Use when an agent must compare a model, threshold, backend, or quantized artifact and enforce a recall floor before release.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Benchmark PII recall

Measure PII recall before optimizing F1, size, or latency. A missed direct
identifier is a privacy failure even when aggregate F1 improves.

## Procedure

1. Build synthetic fixtures with exact offsets and canonical PII labels.
2. Include direct identifiers, boundary cases, languages/scripts, and the
   target device or quantization.
3. Run `extract_pii` at the candidate threshold.
4. Normalize prediction labels and score each document separately.
5. Aggregate counts only; do not persist raw text or identifier surfaces.
6. Fail the release when the recall floor or zero-critical-leak requirement is
   not met.

## Runnable synthetic benchmark

Install the model runtime first with `python -m pip install "openmed[hf]"`.

```python
from openmed import extract_pii
from openmed.core.labels import normalize_label
from openmed.eval import compute_character_recall, compute_exact_span_f1

MODEL = "OpenMed/OpenMed-PII-SuperClinical-Small-44M-v1"
RECALL_FLOOR = 0.99
FIXTURES = [
    {
        "text": (
            "Call the synthetic clinic at 212-555-0198 or email "
            "demo.patient@example.test."
        ),
        "spans": [
            ("PHONE", "212-555-0198"),
            ("EMAIL", "demo.patient@example.test"),
        ],
    },
    {
        "text": (
            "The synthetic callback number is 415-555-0136 and the contact "
            "address is sample.user@example.test."
        ),
        "spans": [
            ("PHONE", "415-555-0136"),
            ("EMAIL", "sample.user@example.test"),
        ],
    },
]

true_positives = false_positives = false_negatives = 0
covered_graphemes = total_graphemes = 0

for fixture in FIXTURES:
    text = fixture["text"]
    gold = []
    for label, surface in fixture["spans"]:
        start = text.index(surface)
        gold.append(
            {"start": start, "end": start + len(surface), "label": label}
        )

The rest: https://github.com/maziyarpanahi/openmed/blob/HEAD/skills/benchmark-pii-recall/SKILL.md