# Building Gold Corpus

Scaffold a synthetic gold-standard annotation project for evaluating OpenMed NER and...

- Repository: https://github.com/maziyarpanahi/openmed (5755 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-10-31
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add maziyarpanahi/openmed --skill building-gold-corpus -g -a claude-code -y`
Codex: `npx skills add maziyarpanahi/openmed --skill building-gold-corpus -g -a codex -y`
Cursor: `npx skills add maziyarpanahi/openmed --skill building-gold-corpus -g -a cursor -y`
Gemini CLI: `npx skills add maziyarpanahi/openmed --skill building-gold-corpus -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add maziyarpanahi/openmed`

## Description

Scaffold a synthetic gold-standard annotation project for evaluating OpenMed NER and de-identification models, label schema, annotation guidelines, BRAT or Label Studio config, and disjoint train/dev/test splits. Use when the user wants to create eval fixtures, set up annotation, define a label set, write guidelines, configure an annotation tool, or build a held-out gold set for the OpenMed eval harness. Trigger on \"gold corpus\", \"annotation project\", \"label schema\", \"annotation guidelines\", \"BRAT\", \"Label Studio\", \"train dev test split\", or \"build eval fixtures\" for OpenMed. Committed gold must be synthetic; licensed (i2b2/n2c2/MIMIC) data is eval-only and never committed.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Building a Gold Corpus

You can't evaluate what you can't measure against. This skill scaffolds a
gold-standard annotation project whose output drops straight into the OpenMed
eval harness as fixtures. The hard rule: **anything committed to the repo is
synthetic**. Licensed clinical corpora (i2b2, n2c2, MIMIC) are DUA-gated, use
them at eval time from the user's own copy, never check them in.

## When to use this skill

- You need eval fixtures for `benchmarking-clinical-ner` or
  `evaluating-with-leakage-gates` and have none.
- You're standing up an annotation effort: schema, guidelines, tool config.
- You need disciplined train/dev/test splits with no leakage between them.
- You want a small **synthetic** golden set you can commit and gate on in CI.

## The OpenMed fixture shape (your target output)

Annotations must serialize to character-offset spans the harness understands:

```json
{
  "fixtures": [
    {
      "id": "synthetic-0001",
      "language": "en",
      "text": "Ms. Jane Roe (MRN 0000000) seen 2099-01-02 for type 2 diabetes.",
      "gold_spans": [
        {"start": 4,  "end": 12, "label": "PERSON"},
        {"start": 18, "end": 25, "label": "ID_NUM"},
        {"start": 32, "end": 42, "label": "DATE"},
        {"start": 47, "end": 62, "label": "DISEASE"}
      ]
    }
  ]
}
```

`openmed.eval.harness.load_fixtures` accepts a top-level list or a `{"fixtures":
[...]}` mapping. Offsets are character indices into `text`; labels are
OpenMed-canonical.

## Quick start, scaffold the project

```
eval/
  gold/
    guidelines.md            # annotation manual + edge-case decisions
    label_schema.json        # canonical labels + definitions + examples
    synthetic/               # COMMITTED synthetic fixtures (CI-gateable)
      train.json
      dev.json
      test.json
  external/                  # GITIGNORED: licensed DUA corpora, eval-only
    .gitignore               # *  (never commit i2b2/n2c2/MIMIC)
```

The rest: https://github.com/maziyarpanahi/openmed/blob/HEAD/skills/building-gold-corpus/SKILL.md