# Fixing Flaky Tests

Guides an agent through reproducing, root-causing, fixing, and validating flaky tests...

- Repository: https://github.com/amplifthq/openmeld (1404 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add amplifthq/openmeld --skill fixing-flaky-tests -g -a claude-code -y`
Codex: `npx skills add amplifthq/openmeld --skill fixing-flaky-tests -g -a codex -y`
Cursor: `npx skills add amplifthq/openmeld --skill fixing-flaky-tests -g -a cursor -y`
Gemini CLI: `npx skills add amplifthq/openmeld --skill fixing-flaky-tests -g -a gemini-cli -y`

## Description

Guides an agent through reproducing, root-causing, fixing, and validating flaky tests in the PostHog monorepo. Use when a test fails intermittently in CI but passes on rerun or locally, when `hogli ci:insights` or the debugging-ci-failures skill classifies a failure as a flaky test, when given a GitHub Actions URL for a flaky job, when asked to check Trunk Flaky Tests for a test, PR, or master, or when asked to deflake, stabilize, or fix a flaky Jest, pytest, or Playwright test. Core discipline: reproduce locally before changing anything, fix the root cause (never mask it with sleeps, retries, or bigger timeouts), and prove the fix with an N-run validation loop sized to the observed failure rate. Stabilizing is not the only valid outcome, the skill also gates whether the test should exist, so deleting a test that catches nothing real, or re-leveling one that flakes because of the level it runs at, are first-class endings.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Fixing flaky tests

Before you propose a change to how the suite runs in CI, check [things already tried](../../../docs/internal/ci-things-already-tried.md). It records measured verdicts on test parallelism, sharding, and coverage-based selection, so a rejected approach is not rebuilt.

Three non-negotiables, in order:

1. **Reproduce before you fix.** A fix for a failure you never observed is a guess.
   Only fall back to analytical fixes when the escalation ladder below is exhausted.
2. **Fix the root cause.** Sleeps, raised timeouts, retries, and weakened assertions hide flakes; they do not fix them.
3. **Validate with an N-run loop.** One green run proves nothing about an intermittent failure.
   Size N to the observed failure rate.

Stabilizing the test is not the only valid ending.
Once you know why it flakes, step 5 asks whether it should exist.
A test that catches nothing real is worth deleting, and one that flakes because of the level it runs at is worth moving down a rung.

Before any of these: **measure, don't assume.** Flaky-vs-deterministic, and the rate, are facts to establish from verifiable GitHub run data (step 1), never inherited from a Slack alert, a teammate's guess, or a `ci:insights` label.

For triaging a red CI run (finding and classifying the failure), use the `debugging-ci-failures` skill first, this skill takes over once the failure is classified as a flaky test.
For writing new Playwright tests that aren't flaky, use the `playwright-test` skill.
`investigating-ci-failures` (green/red boundary) and `diagnosing-ci-and-merge-bottlenecks` (the `engineering-analytics-flaky-tests` tool's caveats) are product skills under `products/engineering_analytics/skills/`, not invocable here: read their `SKILL.md` at that path.

## 1. Measure the failure rate, from GitHub, not from a digest

The rest: https://github.com/amplifthq/openmeld/blob/HEAD/.agents/skills/fixing-flaky-tests/SKILL.md