Quantitative analysis

A wrong answer here is rarely an arithmetic error. It is a correct calculation on the wrong comparison, or on data that does not mean what the field name suggests.

Frame the question so that an answer changes something

Start from the decision. "How is retention doing" has no answer; "is the cohort we changed onboarding for retaining better than the one before it, enough to justify rolling it out" does.

Write down what you expect to find and what you would do in each case before you look. If every possible result leads to the same action, the analysis is not worth running, and knowing that in advance is worth more than the analysis would have been.

Choose the comparison before the metric

Almost every meaningful number is a comparison, and the choice of what to compare against does more work than the calculation.

• Against what it was, needs a period long enough to see through seasonality and noise.
• Against what it would have been, the strongest comparison and the hardest to construct. A holdout group, a matched segment, a pre-trend extended forward.
• Against a peer or a benchmark, only useful if the definitions genuinely match, which they usually do not.

Name the counterfactual explicitly. "Revenue rose after the campaign" is a comparison against nothing, and it is the single most common way credit is claimed for a trend that was already happening.

Interrogate the data before you trust it

Look at the raw rows. Check when collection started and whether the definition changed partway. Check null rates, duplicates, and test or internal accounts still in the set. Check whether recent periods are still filling in, partial data at the tail is what produces the "sudden decline" that resolves itself a week later.

A field's name is not its definition. Find out what actually writes it and under what conditions, especially for anything named status, type, active, or created.