# Auditing Subgroup Fairness

Audit an OpenMed NER or de-identification model for performance disparities across...

- Repository: https://github.com/maziyarpanahi/openmed (5755 stars)
- Category: Writing
- License: Apache-2.0
- Last updated: 2026-10-31
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add maziyarpanahi/openmed --skill auditing-subgroup-fairness -g -a claude-code -y`
Codex: `npx skills add maziyarpanahi/openmed --skill auditing-subgroup-fairness -g -a codex -y`
Cursor: `npx skills add maziyarpanahi/openmed --skill auditing-subgroup-fairness -g -a cursor -y`
Gemini CLI: `npx skills add maziyarpanahi/openmed --skill auditing-subgroup-fairness -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add maziyarpanahi/openmed`

## Description

Audit an OpenMed NER or de-identification model for performance disparities across demographic subgroups (sex, age band, race/ethnicity when available) using openmed.eval.fairness_report. Use when the user wants per-subgroup recall and leakage, wants to check whether de-identification under-protects a group, wants to surface a documentation gap where subgroup data is missing, or needs equalized-odds-style disparity numbers for a clinical model. Trigger on \"fairness\", \"subgroup\", \"bias audit\", \"disparity\", \"equalized odds\", \"under-protected group\", \"per-group recall\", or \"STANDING Together\" for an OpenMed model.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Auditing Subgroup Fairness

An aggregate pass can hide a group the model fails. For de-identification that
failure has a name: **under-protection**, PHI that leaks more often for one
demographic group than another. `openmed.eval.fairness_report` slices leakage and
recall by gold-span group so disparities surface before deployment, not after a
breach.

## When to use this skill

- You want per-subgroup recall and leakage for a de-id or NER model.
- You suspect (or must rule out) that one group is under-protected.
- You need disparity numbers for a clinical AI governance review.
- You need to document *which* subgroups you couldn't evaluate (the data gap).

## What it measures

For each surrogate group `fairness_report` returns:

- **leakage_rate**, fraction of that group's gold PHI characters left exposed
  (the de-id harm metric).
- **recall**, fraction of that group's gold spans detected.
- **leakage_disparity**, `max - min` leakage across groups (the gap to close).
- **worst_group** / **worst_group_leakage**, the most-failed group.

Group membership comes from a `group` tag in each gold span's `metadata` (keys
`group`, `demographic_group`, or `surrogate_group`); ungrouped spans fall into
`unspecified`.

## Quick start

```python
from openmed.eval import fairness_report

# Gold fixtures must tag spans with a surrogate group, e.g.
#   {"start": 4, "end": 12, "label": "PERSON", "metadata": {"group": "female"}}
fair = fairness_report(
    "OpenMed/Privacy-PII-Detection",
    "golden",                 # named suite, or pass a list of fixtures
    device="cpu",
)

print("leakage disparity:", fair.leakage_disparity)
print("worst group      :", fair.worst_group, fair.worst_group_leakage)
for group, m in sorted(fair.per_group.items()):
    print(f"  {group:14s} recall={m.recall:.3f}  leakage={m.leakage_rate:.4f}")

The rest: https://github.com/maziyarpanahi/openmed/blob/HEAD/skills/auditing-subgroup-fairness/SKILL.md