Differential Audit
Compare two implementations of the same thing, a port (R↔Python↔Stata), a...
Install
npx skills add pedrohcgs/claude-code-my-workflow --skill differential-audit -g -a claude-code -y- Repository
- pedrohcgs/claude-code-my-workflow1.7k
- Category
- Testing
- Updated
- 1mo ago
- License
- MIT
- Context cost
- Lightabout 1,384 tokens
- Runs scripts
- No, instructions only
What it does
Compare two implementations of the same thing, a port (R↔Python↔Stata), a reimplementation, a replication package, a refactor, or a new version against the old, so that agreement means something. Freeze inputs first, inventory every expected output, test the comparator itself, compare every channel (not just the headline number), and give each divergence a stable ID and a smallest witness. Use for cross-language parity, replication, upgrade/regression gates, or whenever "the numbers match" is about to license a claim.
Make agreement mean something
Two implementations agreeing proves they satisfy a prespecified contract. It does not prove either is correct, and it never validates the method or its assumptions. Both can be wrong in the same way, especially when one was written by reading the other. Design the comparison so that agreement is informative and disagreement is legible.
Rule: freeze before you compare; test the comparator before you trust it.
1. State the claim and the reference
Write down: what is being compared, which side is the reference, and what agreement would and would not establish. "Matches the R package" is a conformance claim, not a correctness claim. Say so explicitly, so nobody later reads parity as validation.
2. Freeze the inputs before inspecting anything
Record and fix: data versions or hashes, code and package versions, random seeds or realized sample splits, options and defaults, the outputs to be compared, and the acceptance thresholds. Freezing after a first look invites tolerance drift toward whatever the run produced.
Do not compare defaults across systems as if only the language changed. Map the choices explicitly, a "default" is a substantive modeling decision that usually differs between implementations.
3. Declare tolerance classes, and make them binding
Do not carry a single fuzzy epsilon. Classify each output:
- EXACT, names, ordering, sample masks, counts, statuses, return/error codes, warning classes, option defaults. Byte-equal after documented normalization.
- Scalar numeric, deterministic estimates, standard errors, p-values, critical values. State absolute and relative tolerances and the justification.
- Matrix/vector, covariance matrices, influence summaries, weight vectors, plot data.
- Stochastic, must meet a prespecified error-rate criterion with uncertainty reported.
