Mandela
Audit any eval, metric, experiment, or benchmark for leakage, does external...
Install
npx skills add LilMGenius/paperthin --skill mandela -g -a claude-code -y- Repository
- LilMGenius/paperthin1.4k
- Category
- Design
- Updated
- 3d ago
- License
- MIT
- Context cost
- Lightabout 668 tokens
- Runs scripts
- No, instructions only
What it does
Audit any eval, metric, experiment, or benchmark for leakage, does external ground-truth enter independently, or are the model, scorer, and designer just confirming a result no outside truth ever produced? Use before trusting any 'how we'll know it worked', an A/B, a holdout, a score, a validation, and whenever a result feels too clean or self-confirming. Walks an 8-pattern leakage taxonomy and returns only the patterns that fire, each with an independence fix. Read-only.
Audit a validation for leakage: does outside ground-truth actually enter, or is everyone confirming a result no one independently produced?
Goal
The name is the Mandela Effect, a whole population confidently remembers something that never independently happened; a leaky validation is the same shape. Walk the 8 patterns below. mandela checks one thing: whether a validation is independent, or whether the designer, model, and scorer are only confirming each other.
Workflow
- Identify the validation (eval / metric / experiment / holdout / "how we'll know"). Name its components, what plays model, scorer, designer, dataset.
- Ask the core question: does external ground-truth enter independently?
- Test the validation against all 8 patterns below (some apply only to certain components, a human subject, a scorer); report only the ones that fire, each by name.
- Give the independent-ground-truth fix for each hit.
The 8 leakage patterns
- Recall, not reason, a memorized answer recited instead of one actually derived; the system already knows the result it is supposedly computing.
- Wrong null hypothesis, an ablation that removes a surface label but not the underlying signal the system actually exploits, so the "control" still leaks.
- Shared hallucination, two components verifying each other; circularity reported as a number.
- Tautology, a scorer grading buckets it drew itself.
- Verifier = designer, a private, unreproducible recipe in a holdout's clothes.
- Shared-pool bias, train and holdout drawn from one labeler pool, so one bias enters both sides.
- Frame injection, a question that hands the subject the hypothesis.
- Demand characteristics, measured subjects who know they're being measured.
