AI Evaluation Dataset Design

Goal

Produce an evaluation dataset and its supporting documentation that measure whether an AI system does its job, refuses what it should refuse, and behaves acceptably under pressure. The dataset is a durable customer artifact, so its scope, balance, and rationale are recorded rather than implied.

Flow

1. Run the scoping interview from the interview reference. Ask one question at a time and wait for the answer; do not batch the interview into a single prompt.
2. Present a structured summary of what you heard and obtain explicit confirmation before generating anything.
3. Derive the difficulty distribution from the confirmed scope, adjusting the defaults when the system's risk profile warrants it.
4. Generate the dataset against the contract template in both machine-readable forms.
5. Validate the JSON and CSV pair with uv run python scripts/validate_evaluation_dataset.py --json <dataset.json> --csv <dataset.csv> before treating machine facts as evidence.
6. Walk a representative sample through the user, gather consolidated feedback, and revise before finalizing the full set.
7. Produce one sectioned evaluation guide containing curation notes, metric selection with rationale, and tooling recommendations.
8. Route every durable write through the workstream's scan gate before it lands in a customer location.

Current Evaluator Research

Activate rpi-research only when the confirmed evaluation job needs current evaluator names, availability, preview state, platform compatibility, or prerequisites that the authoritative live source must establish. Provide the platform and product-version scope, system context, tool-use pattern, metric-plan decision, source and date boundaries, evidence criteria, non-goals, and supplied evaluation evidence. Use analysis or comparison mode and the default Research evidence root.