Purpose

Create traceable cleaned data and one reusable profile. Do not repeat the same data inspection separately for every candidate method.

Preconditions

• Problem parse and subquestion IDs exist.
• Raw files are available under workspace/data_raw/ or the workspace's documented legacy raw-data path.
• Required outputs and known field needs are available.

Stop rather than fabricate a missing attachment, unit, field meaning, or label.

Workflow

1. Map attachments before cleaning.

   • List each attachment with name, size, sheet names, headers, and a small preview.
   • Map it to Qx or mark it shared.
   • Ask the user only when two mappings remain materially plausible.

2. Preserve raw data.

   • Treat raw files as read-only.
   • Record hashes or stable file metadata when practical.
   • Write cleaned copies under workspace/data_clean/.

3. Audit structure and semantics.

   • Rows, columns, keys, types, units, categories, time granularity, and encoding.
   • Missing values, duplicates, impossible values, outliers, discontinuities, and leakage risks.
   • Field-to-subquestion and field-to-required-output mapping.

4. Compute reusable risk-profile statistics.

   • Effective sample size and rows usable per Qx.
   • Missingness by field and row.
   • Numeric distribution summaries and extreme-value rates.
   • Category/class counts, imbalance ratios, rare levels, and cardinality.
   • Time coverage, gaps, sampling interval, and chronological split constraints.
   • Correlation/redundancy warnings where relevant.
   • Target or score concentration indicators when a target exists.
   • Record facts; do not convert them into a final method verdict.

5. Plan and apply cleaning.

   • Separate safe normalization of representation from assumption-bearing imputations or removals.
   • Explain and record every assumption-bearing operation.
   • Keep reproducible cleaning code only when transformations are nontrivial.