# Benchmarking Clinical Ner

Score an OpenMed clinical or biomedical NER model against a user-supplied gold corpus...

- Repository: https://github.com/maziyarpanahi/openmed (5755 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-10-31
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add maziyarpanahi/openmed --skill benchmarking-clinical-ner -g -a claude-code -y`
Codex: `npx skills add maziyarpanahi/openmed --skill benchmarking-clinical-ner -g -a codex -y`
Cursor: `npx skills add maziyarpanahi/openmed --skill benchmarking-clinical-ner -g -a cursor -y`
Gemini CLI: `npx skills add maziyarpanahi/openmed --skill benchmarking-clinical-ner -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add maziyarpanahi/openmed`

## Description

Score an OpenMed clinical or biomedical NER model against a user-supplied gold corpus with entity-level precision, recall, and F1, then break errors down per label. Use when the user wants a seqeval-style scorecard, strict vs partial (relaxed) span matching, a per-label confusion matrix, false-negative / false-positive examples, or to debug why a model misses entities. Trigger on \"evaluate NER\", \"entity-level F1\", \"seqeval\", \"precision recall F1\", \"confusion matrix\", \"error analysis\", \"strict vs partial match\", or \"score against gold\" in an OpenMed context. The gold corpus is user-supplied; OpenMed bundles no i2b2/n2c2/MIMIC data.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Benchmarking Clinical NER

This skill produces an honest entity-level scorecard for an OpenMed NER model:
precision / recall / F1 plus a per-label error breakdown. It scores **spans**,
not tokens, because clinical entities are multi-token ("type 2 diabetes
mellitus") and token-level accuracy hides boundary errors. Reported numbers are
**entity-level** in the seqeval tradition (CoNLL-2000 / SemEval-2013 families).

## When to use this skill

- You have a gold-annotated clinical corpus and an OpenMed NER model to score.
- You want strict (exact-boundary) and partial (relaxed-overlap) span F1.
- You need per-label numbers, not one aggregate, DRUG recall ≠ DISEASE recall.
- You need to *explain* the errors: what was missed, what was spurious, what was
  mislabeled.

For PHI de-id specifically, gate on leakage with `evaluating-with-leakage-gates`
instead of (or in addition to) F1.

## Match modes

| Mode | Counts a hit when… | Use for |
| --- | --- | --- |
| **Strict / exact** | predicted span boundaries **and** label match gold exactly | release scoring, boundary-sensitive tasks |
| **Partial / relaxed** | predicted span **overlaps** gold with the right label | recall-oriented triage, tokenizer-mismatch tolerance |

OpenMed exposes both: `compute_exact_span_f1` (strict) and
`compute_relaxed_span_f1` (partial), with the full bundle in
`compute_metrics_bundle`.

## Quick start

Run a model over a user-supplied gold fixtures file and print a scorecard:

```python
from openmed.eval import run_suite, error_report

# Fixtures: JSON list of {"id", "text", "gold_spans": [{start, end, label}, ...]}
report = run_suite(
    "eval/gold/clinical_ner.json",        # YOUR gold corpus, not bundled
    suite="golden",
    model_name="OpenMed/Disease-Detection",
    device="cpu",
)

m = report.metrics
print("exact F1 :", m["exact_span_f1"]["f1"])      # strict
print("relaxed F1:", m["relaxed_span_f1"]["f1"])    # partial
print("recall by label:", m["recall_slices"]["by_label"])

The rest: https://github.com/maziyarpanahi/openmed/blob/HEAD/skills/benchmarking-clinical-ner/SKILL.md