
Experiment Audit
Audit experiment integrity before claiming results.
Install
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-audit -g -a claude-code -y- Repository
- wanshuiyin/Auto-claude-code-research-in-sleep18.4k
- Category
- Writing
- Updated
- 4d ago
- License
- MIT
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- No, instructions only
What it does
Audit experiment integrity before claiming results. Uses cross-model review (external reviewer backend) to check for fake ground truth, score normalization fraud, phantom results, and insufficient scope. Use when user says \"审计实验\", \"check experiment integrity\", \"audit results\", \"实验诚实度\", or after experiments complete before writing claims.
Experiment Audit: Cross-Model Integrity Verification
Before each reviewer slot, follow Selectable Reviewers: resolve invocation →
.aris/reviewer.json→ Claude Code default (Codex MCP) withreviewer_route.pythrough the canonical helper resolver, preserving this slot's tier, fresh contexts, source/image inputs and deterministic gates. Codex examples/pins below apply only tobackend=codex; other routes use their own first-call/reply contract. Save the route and actual returned identity; unavailable selections areREVIEW_UNAVAILABLE. Classify acceptance from actual families/completion/scope. Forward invocation reviewer/model/effort overrides to reviewer-bearing children. Generation workers are unchanged.
🔒 Do not wrap this skill in
/loop,/schedule, orCronCreate. It is verdict-bearing, it judges experiment integrity. Re-running that verdict on a timer adds no new signal, and a loop that accepts its own output to decide when to stop crosses into self-acquittal (acceptance-gate.md). Schedule the external wait that precedes it, experiments done → then audit once. Seeshared-references/external-cadence.md.
Audit experiment integrity for: $ARGUMENTS
Why This Exists
LLM agents can produce fraudulent experimental results through:
- Fake ground truth, creating synthetic "reference" from model outputs, then reporting high agreement as performance
- Score normalization, dividing metrics by the model's own max to get 0.99+
- Phantom results, claiming numbers from files that don't exist or functions never called
- Insufficient scope, reporting 2-scene pilots as "comprehensive evaluation"
These are NOT intentional deception, they are failure modes of optimizing agents that lack integrity constraints. This skill adds that constraint.
