# Assay

Assertion-first eval framework: code asserts, calibrated judges, pass@k, A/B...

- Repository: https://github.com/danielmiessler/LifeOS (19669 stars)
- Category: Design
- License: MIT
- Last updated: 2026-10-28
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add danielmiessler/LifeOS --skill Assay -g -a claude-code -y`
Codex: `npx skills add danielmiessler/LifeOS --skill Assay -g -a codex -y`
Cursor: `npx skills add danielmiessler/LifeOS --skill Assay -g -a cursor -y`
Gemini CLI: `npx skills add danielmiessler/LifeOS --skill Assay -g -a gemini-cli -y`

## Description

Assertion-first eval framework: code asserts, calibrated judges, pass@k, A/B comparison, agentic evals, judgment-layer question systems and code-extract-decide-write pipelines. USE WHEN eval, evaluate, benchmark, judge, judge prompt, judge panel, regression test, assert, llm-rubric, calibrate judge, judged residue, pass@k, pass^k, grade output, compare prompts or models, A/B test, which model did we like, model retro, ablation, with/without, test agent, agentic eval, noul/choice/score question, glance row, judged miss, proof this, check the output, did it come out as asked, design judgment questions, question system, design a pipeline, extract values, Clairic, LLM-only baseline, judgment layer status/report/tune, is this caller ready to enforce. NOT FOR scientific-method framing (Science), property/mutation testing (Hardening), live UI verification (Interceptor), non-eval prompts (Prompting), TypeSafe apps outside LifeOS (typesafe-ai).

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Assay, Assertion-First AI Evaluation

**Contents:** Ideal State · Intent Routing · The taxonomy · Choosing a check: tier first, then shape · Selection, acceptance, and unavailable evidence · The canonical path · Doctrine (from Anthropic; encode, don't restate) · The image lane (visual evals) · Harness integration · Judgment-layer question systems · Data & Config · Gotchas · Examples

An eval gives an AI an input, then applies **assertions** to its output to measure success (Anthropic's definition). A case is `{id, prompt, assert:[...]}`. Each assertion is either deterministic (code, free) or model-graded (a calibrated judge). Cases run multiple trials and report **pass^k** (every trial passes; the reliability metric and the regression gate) and **pass@k** (any trial passes). Proof applies the same tiers to one run's output (the film, image, page, email or document it made) against what was asked, before the principal sees it: `Intents/Proof.md`, with the whole system in `LIFEOS/DOCUMENTATION/Assay/AssaySystem.md`. CreateQuestions designs the fast tier itself: it turns a fuzzy decision into a measured judgment-layer question system (`LIFEOS/DOCUMENTATION/Assay/AssaySystem.md` § Judgment layer).

Sources: [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) (2026-01), [A Statistical Approach to Model Evaluations](https://www.anthropic.com/research/statistical-approach-to-model-evals) (2024-11), [develop tests](https://platform.claude.com/docs/en/docs/build-with-claude/develop-tests), the eval-awareness contamination post (2026-03), and the live `claude plugin eval` surface.

**Freshness contract:** "aligned to Anthropic's doctrine" is a live claim. When designing a new suite class or touching `## Doctrine`, re-fetch the demystifying-evals post and flag where it has moved. Report divergence, never auto-adopt; an unreachable URL never blocks a run.

## Ideal State

The rest: https://github.com/danielmiessler/LifeOS/blob/HEAD/LifeOS/install/skills/Assay/SKILL.md