metric-design, build the ruler (Phase 1)

Build the ruler before you measure anything with it. This skill explains Phase 1: how the app designs and proves a metric the keep-or-revert loop can trust. A loop optimizing a noisy or wrong ruler "wins" on fiction, and that is the dominant risk this app is built to eliminate, so Phase 2 does not start until Phase 1 has proven itself.

Calibration is backend code, not an agent task. It runs from the owner-only POST /api/apps/auto-improvement/calibrate route, and again as the pre-flight of every perf-track run. No agent tool calibrates or writes the ruler, and the discovery agent must never touch it; read it with the get_ruler MCP tool.

The output, the ruler record

Calibration writes ruler/ruler.json under the active repository+branch workspace (data/repos/<workspace-key>/ruler/): status (calibrated or canary_failed), the primary metric, the noise band, a baseline anchor (the median sample), the canary result and the raw samples. There is no separate metric-design document. The ruler the active target profile supplies:

• Primary metric, a low-variance number where the two measurement arms cancel as much shared cost as possible (label + unit + direction). Never hard-coded in the UI; it is read from the profile.
• Attributable sub-stages, so a win is pinned to a named stage rather than hand-waved as a whole-system improvement.
• Frozen anchors, reference measurements. Calibration records one, the baseline median, in ruler.json; the shipped GitHub profile declares no others.
• Guardrails, metrics that must not regress beyond a stated tolerance.
• Reward-hack guards, checks the build/test gate structurally cannot see, such as "no silent capability shrink" or "a held-out functional probe still passes".

Calibration, the trust gate