# Test Flakiness

Find flaky tests from CI logs, aggregates pass rates, spots intermittent failures...

- Repository: https://github.com/Donchitos/Claude-Code-Game-Studios (32144 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-10-19
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness -g -a claude-code -y`
Codex: `npx skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness -g -a codex -y`
Cursor: `npx skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness -g -a cursor -y`
Gemini CLI: `npx skills add Donchitos/Claude-Code-Game-Studios --skill test-flakiness -g -a gemini-cli -y`

## Description

Find flaky tests from CI logs, aggregates pass rates, spots intermittent failures, recommends quarantine. After multiple runs.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

!`bash "${CLAUDE_SKILL_DIR}/../../hooks/yaml-helper.sh" resolve_config --keys automation`

**Automation mode**: Resolve `modes.automation` (`project.local.yaml` →
`project.yaml` → default `collaborative`). Every `AskUserQuestion` call and
every file write follows `.claude/docs/automation-modes.md`
(collaborative asks always · guided major-only · autonomous logs and proceeds;
`automation_always_ask` categories always prompt).

# Test Flakiness Detection

A flaky test is one that sometimes passes and sometimes fails without any code
change. Flaky tests are worse than no tests in some ways, they train the team
to ignore red CI runs, masking genuine failures. This skill identifies them,
explains likely causes, and recommends whether to quarantine or fix each one.

**Output:** Updated `tests/regression-suite.md` quarantine section + optional
`production/qa/flakiness-report-[date].md`

**When to run:**
- Polish phase (tests have had many runs; statistical signal is reliable)
- When developers start dismissing CI failures as "probably flaky"
- After `/regression-suite` identifies quarantined tests that need diagnosis

---

## 1. Parse Arguments

**Modes:**
- `/test-flakiness [ci-log-path]`, analyse a specific CI run log file
- `/test-flakiness scan`, scan all available CI logs in `.github/` or
  standard log output directories
- `/test-flakiness registry`, read existing regression-suite.md quarantine
  section and provide remediation guidance for already-known flaky tests
- No argument, auto-detect: run `scan` if CI logs are accessible, else
  `registry`

---

## 2. Locate CI Log Data

### Option A, GitHub Actions (preferred)

Check for test result artifacts:
```bash
ls -t .github/ 2>/dev/null
ls -t test-results/ 2>/dev/null
```

For Godot projects: GdUnit4 outputs XML results compatible with JUnit format,
under `reports/` (its default report folder, `res://reports/`). Check `reports/`,
and any `test-results/` you saved runs into, for `.xml` files.

The rest: https://github.com/Donchitos/Claude-Code-Game-Studios/blob/HEAD/.claude/skills/test-flakiness/SKILL.md