Building a Gold Corpus

You can't evaluate what you can't measure against. This skill scaffolds a gold-standard annotation project whose output drops straight into the OpenMed eval harness as fixtures. The hard rule: anything committed to the repo is synthetic. Licensed clinical corpora (i2b2, n2c2, MIMIC) are DUA-gated, use them at eval time from the user's own copy, never check them in.

When to use this skill

• You need eval fixtures for benchmarking-clinical-ner or evaluating-with-leakage-gates and have none.
• You're standing up an annotation effort: schema, guidelines, tool config.
• You need disciplined train/dev/test splits with no leakage between them.
• You want a small synthetic golden set you can commit and gate on in CI.

The OpenMed fixture shape (your target output)

Annotations must serialize to character-offset spans the harness understands:

{
  "fixtures": [
    {
      "id": "synthetic-0001",
      "language": "en",
      "text": "Ms. Jane Roe (MRN 0000000) seen 2099-01-02 for type 2 diabetes.",
      "gold_spans": [
        {"start": 4,  "end": 12, "label": "PERSON"},
        {"start": 18, "end": 25, "label": "ID_NUM"},
        {"start": 32, "end": 42, "label": "DATE"},
        {"start": 47, "end": 62, "label": "DISEASE"}
      ]
    }
  ]
}

openmed.eval.harness.load_fixtures accepts a top-level list or a {"fixtures": [...]} mapping. Offsets are character indices into text; labels are OpenMed-canonical.

Quick start, scaffold the project

eval/
  gold/
    guidelines.md            # annotation manual + edge-case decisions
    label_schema.json        # canonical labels + definitions + examples
    synthetic/               # COMMITTED synthetic fixtures (CI-gateable)
      train.json
      dev.json
      test.json
  external/                  # GITIGNORED: licensed DUA corpora, eval-only
    .gitignore               # *  (never commit i2b2/n2c2/MIMIC)