Assay
Assertion-first eval framework: code asserts, calibrated judges, pass@k, A/B...
Install
npx skills add danielmiessler/LifeOS --skill Assay -g -a claude-code -y- Repository
- danielmiessler/LifeOS19.7k
- Category
- Design
- Updated
- 8d ago
- License
- MIT
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- Yes, review before use
What it does
Assertion-first eval framework: code asserts, calibrated judges, pass@k, A/B comparison, agentic evals, judgment-layer question systems and code-extract-decide-write pipelines. USE WHEN eval, evaluate, benchmark, judge, judge prompt, judge panel, regression test, assert, llm-rubric, calibrate judge, judged residue, pass@k, pass^k, grade output, compare prompts or models, A/B test, which model did we like, model retro, ablation, with/without, test agent, agentic eval, noul/choice/score question, glance row, judged miss, proof this, check the output, did it come out as asked, design judgment questions, question system, design a pipeline, extract values, Clairic, LLM-only baseline, judgment layer status/report/tune, is this caller ready to enforce. NOT FOR scientific-method framing (Science), property/mutation testing (Hardening), live UI verification (Interceptor), non-eval prompts (Prompting), TypeSafe apps outside LifeOS (typesafe-ai).
Assay, Assertion-First AI Evaluation
Contents: Ideal State · Intent Routing · The taxonomy · Choosing a check: tier first, then shape · Selection, acceptance, and unavailable evidence · The canonical path · Doctrine (from Anthropic; encode, don't restate) · The image lane (visual evals) · Harness integration · Judgment-layer question systems · Data & Config · Gotchas · Examples
An eval gives an AI an input, then applies assertions to its output to measure success (Anthropic's definition). A case is {id, prompt, assert:[...]}. Each assertion is either deterministic (code, free) or model-graded (a calibrated judge). Cases run multiple trials and report pass^k (every trial passes; the reliability metric and the regression gate) and pass@k (any trial passes). Proof applies the same tiers to one run's output (the film, image, page, email or document it made) against what was asked, before the principal sees it: Intents/Proof.md, with the whole system in LIFEOS/DOCUMENTATION/Assay/AssaySystem.md. CreateQuestions designs the fast tier itself: it turns a fuzzy decision into a measured judgment-layer question system (LIFEOS/DOCUMENTATION/Assay/AssaySystem.md § Judgment layer).
Sources: Demystifying evals for AI agents (2026-01), A Statistical Approach to Model Evaluations (2024-11), develop tests, the eval-awareness contamination post (2026-03), and the live claude plugin eval surface.
Freshness contract: "aligned to Anthropic's doctrine" is a live claim. When designing a new suite class or touching ## Doctrine, re-fetch the demystifying-evals post and flag where it has moved. Report divergence, never auto-adopt; an unreachable URL never blocks a run.
