Calibrate
Calibrates any eval against the principal's own taste: takes a task in plain words...
Install
npx skills add danielmiessler/LifeOS --skill Calibrate -g -a claude-code -y- Repository
- danielmiessler/LifeOS19.7k
- Category
- Writing
- Updated
- 8d ago
- License
- MIT
- Context cost
- Mediumabout 2,275 tokens
- Runs scripts
- Yes, review before use
What it does
Calibrates any eval against the principal's own taste: takes a task in plain words, finds the eval suites and targets that cover it and offers them as options, then shows N real outputs (images or text, default 16, any count like 4, 24 or 64) on a local review page, collects a 0, 10 score per output plus optional typed notes, measures how well the judge agrees, distills a taste profile the judge reads from then on, and rescores to prove the change. USE WHEN calibrate, calibrate 16, calibrate 64, calibrate social posts, calibrate the art judge, calibrate this eval, show me outputs to rate, rate these images, teach the eval what I like, tune the eval to my taste, what does the judge get wrong about my taste, calibration session. NOT FOR measuring a suite's judge against existing labels without a session (Assay Calibrate), recording one verdict on one delivered artifact (ArtDirection Calibrate), or producing the artifacts themselves (Art).
Calibrate
Contents: Ideal State · Intent Routing · Tool · How taste reaches the judge · Data & Config · Gotchas · Examples
A judge that disagrees with the principal is grading against someone else's taste. Calibrate closes that gap for anything that has an eval. Every Assay suite is a target with no setup: its stored run outputs are the pool and its judged asserts are the judge. Image intents join through a configured target. The principal names a task in plain words, discover ranks the targets and suites that cover it, and the principal picks one.
Outputs are shown blind (no judge scores on the page), and text outputs show the prompt that produced them. The principal scores each one 0, 10 on a slider, with a note when it matters; a score at or above the target's likeAt (default 7) counts as a like for pass/fail agreement, and the raw score is kept. The labels answer three things: how often the current judge agrees, where its cut-off should sit to match the principal, and what the principal actually values. The last becomes Taste.json, which the judge reads on every later run.
Ideal State
- The judge's agreement with the principal is measured on labels held out from the taste being tested and reported against the shared floors in
LIFEOS/TOOLS/lib/Calibration.ts. - Every label came from the principal's own submit on the review page, scoring their real published outputs blind on 0, 10.
- Every principle in a taste profile cites the labels behind it, and a suite or brief without a profile gets exactly the judge prompt it had before.
- No bar changes and no judge is promoted on calibration numbers without the principal's yes.
