Flaky tests
How ReplayQA spots tests whose results flip without any change, where to find them, and the most common ways to make them reliable again.
A flaky test passes and fails against the same version of your site. A few flaky tests are enough to make people ignore red results, so ReplayQA tracks them for you.
How ReplayQA spots them
ReplayQA tracks how often each test's result flips between runs of the same approved version. A test that flips often gets a Flaky badge with its flip rate, for example Flaky · 18% flip rate.
When a flaky test fails, its run page says so: treat a single failure from it with care before reporting a bug.
Find your flaky tests
- On Home, press Review flaky tests.
- On Tests, tick Flaky only to filter the list.
Fix the common causes
| Symptom in the run | Likely cause | What to do |
|---|---|---|
| A step fails because something "isn't visible yet" | Slow page or animation | Add a Wait for step for the text that appears when the page is ready |
| The wrong item is clicked in a list | Several elements have the same text | Make the target more specific in the editor (press Change) |
| "Already exists" or missing data | Tests share or use up data | Use data each run can reuse, or reset the test site regularly |
| Fails only in Play All | Tests interfere with each other | Lower Max parallel, or use separate test accounts |
| Fails around midnight or month-end | Dates | Use fixed dates far in the future. See Dates, dropdowns and popups |
| Random failures on a slow staging server | Server under load | Lower Max parallel on the suite |
After a fix, the flip rate recovers as new runs pass.
Should I just retry?
Retry N failed after a Play All is a good way to confirm a failure is flaky rather than real. But a test that needs retries is a test to fix: it slows down every run and hides real failures.
For a deeper guide, read Flaky end-to-end tests: 10 causes and how to fix each one.