Flaky tests: finding and fixing tests that fail at random
Short answer: A flaky test passes and fails on the same code without any change, so the team stops trusting the test suite and starts clicking "re-run" by habit. Track which tests are flaky, quarantine them so they don't block every merge, then fix the usual causes: timing assumptions, shared data between tests, dependence on test order, real network calls, and dates or random values. Use automatic retries only as a temporary safety net, and make sure retried passes are reported, not hidden.
Why flaky tests are worse than no tests
A test suite is only useful if a red result means something is broken. When some tests fail at random:
- Developers re-run failed jobs until they go green, wasting minutes or hours a week.
- Real failures get dismissed as "probably flaky" and reach production.
- People stop writing tests because the suite feels like a lottery.
GitHub Actions and other CI systems make re-running easy. GitHub's documentation describes re-running only failed jobs from a workflow run. That convenience is exactly how flakiness becomes normal. The goal is to make re-runs rare again.
Step 1: find out which tests are flaky
You can't fix what you haven't identified. Ways to spot flaky tests:
- Use your test runner's reporting. Playwright Test, for example, classifies results after retries as "passed", "flaky" (failed on the first run but passed when retried) or "failed" (failed every attempt), according to its retries documentation. Other runners and CI tools have similar reports or plugins.
- Track failures over time. Record which tests fail on the main branch, where the code should be good. A test that fails there intermittently is flaky by definition.
- Repeat suspicious tests. Run a single test many times in a loop locally or in CI. Intermittent failures show up quickly.
Keep a simple list, in an issue tracker or a shared document, with each flaky test, when it was spotted and who owns the fix.
Step 2: quarantine, don't ignore
A flaky test that blocks every merge holds up the whole team. But deleting it loses coverage. Quarantine is the middle path:
- Move the test into a separate group or tag, such as
@flaky, that still runs but doesn't block merges. - Set a deadline to fix or delete it.
- Review the quarantine list regularly so it doesn't grow forever.
Step 3: fix the common causes
Most flaky tests come from a small set of problems.
Timing assumptions
Tests that wait a fixed time, such as "sleep 2 seconds then check", fail when the machine is slower than usual. Replace fixed waits with waiting for a condition: an element to appear, a request to finish, a job to complete. For browser tests, tools like Playwright auto-wait: its documentation describes actionability checks, such as visible, stable, enabled and receiving events, that it performs before actions, so tests act when the page is ready rather than after a guessed delay. Use assertions that retry until a condition is met, rather than checking once.
Shared state between tests
If one test creates a customer and another expects the database to be empty, the result depends on which runs first. Each test should set up the data it needs and not rely on leftovers. Use unique identifiers, such as generated emails, so parallel tests don't collide, and reset or isolate state between tests.
Order dependence
Tests that only pass in a certain order often reveal shared state. Running tests in random order, which many runners support, exposes these quickly.
Real network calls
Calls to real third-party APIs, such as payment, maps or email, fail for reasons outside your control. Stub or mock external services in most tests, and keep a small, clearly labelled set of integration tests that use sandboxes and are allowed to be slower.
Time, dates and randomness
Tests that depend on "today" break at midnight, at month-end, across time zones or on daylight saving changes. Freeze or inject the clock in tests. Seed random generators so results are repeatable, and print the seed when a test fails.
Resource limits in CI
Tests that pass locally but fail in CI may be hitting memory, CPU or parallelism limits. Check whether failures cluster when many tests run in parallel, and adjust worker counts or timeouts accordingly.
Step 4: use retries carefully
Automatic retries can stop the occasional infrastructure hiccup from blocking a release. Playwright, for instance, lets you configure retries in its configuration file. But retries hide flakiness if nobody looks:
- Allow a small number of retries, such as one or two, in CI only.
- Make sure retried passes are reported as flaky, and review them.
- Never use retries as the fix.
Step 5: make failures easy to diagnose
Flaky failures are hard to reproduce, so capture evidence the first time:
- Save logs, screenshots, traces or videos for failed tests.
- Record the random seed, test order and environment details.
- Keep artefacts from CI for long enough to investigate.
Make it a team habit
Flakiness comes back if nobody owns it. Agree that whoever spots a flaky test logs it, that the quarantine list is reviewed in a regular team meeting, and that new tests are run several times before merging. A short note in your contributing guide about waiting for conditions and isolating data prevents many new flaky tests being written.
A worked example
Consider a hypothetical Saskatoon event-ticketing team whose CI pipeline failed roughly every few runs, almost always on browser tests, and developers re-ran jobs several times a day. They turned on retry reporting to list flaky tests, quarantined the worst offenders, replaced fixed waits with condition-based waits, gave each test its own generated customer and event, and froze the clock in tests that checked event dates. Within a few weeks the quarantine list was empty, and re-runs became rare.
FAQ
Is a test that fails once in a hundred runs worth fixing?
Yes, if the suite has many such tests. Together they make failures common.
Should flaky tests be deleted?
Only if they don't test anything important or can't be made reliable. Otherwise, fix and keep them.
Do unit tests get flaky too?
Less often than browser tests, but yes, usually because of time, randomness or shared state.
If your team re-runs CI more than it should, ask us to help stabilise your test suite.