Risky Changes

A filter once passed every test but disabled ~99% of the feature on live data. Tests check code correctness; research and live measurement check whether a change is useful.

When this applies

Use for changes where being wrong is expensive or customer-visible:

• Public API fields, filters, or response shaping
• Dropping, transforming, or reordering upstream data
• Billing, pricing, caps, or quotas
• Defaults, thresholds, or provider request parameters
• Unverified assumptions about external data or user behavior

If unsure whether a change qualifies, apply this process.

1. Identify assumptions

List the assumptions the change depends on. Mark which have evidence and which are still unverified.

2. Research before implementing

Use available research tools and reliable sources. Investigate each distinct question separately, covering at least:

• How do leading products handle this design decision?
• What does real-world data look like: frequencies, shapes, and edge cases?
• What do users or agents actually need?

If evidence contradicts an assumption, reconsider the design before coding. If research is unavailable or inconclusive, state the gap; do not treat the assumption as verified. Do not ship while material assumptions remain unverified.

3. Measure real behavior

Run 10, 20+ realistic cases against the real endpoint or provider:

• Base cases on real usage; vary topics, parameters, languages, and edge conditions.
• Define benchmarks per case: speed, quality, accuracy, and how often the new behavior occurs.
• Use hard numbers where possible. For subjective quality, use blind, criteria-based judging.
• Compare before and after when both can be measured.
• Read-only production analysis also counts as measurement.