Investigating failures
How to work out why a test failed, whether a real bug, an intended change or a moved element, and how to accept a suggested fix as a new draft.
A failed run means the site didn't behave the way the video showed. Your job is to decide which of three things happened:
- A real bug. Report it, with the run page as evidence.
- An intended change. The site changed on purpose, so update the test.
- The element moved or was renamed. The flow still works, but the test can't find a button. Accept a fix.
Start on the run page
The failed step is highlighted with its error. Compare the screenshot and replay video with what the step expected. Often the answer is obvious: an error message on the page, a button with new wording, or a page that never loaded.
If the answer isn't obvious, press Investigate failure.
The investigation page
The investigation page puts two images side by side:
- Reference: the frame from the original recording, showing what should have happened.
- Actual: the page at the moment of failure.
ReplayQA classifies the failure so you know where to look:
| Classification | Usually means |
|---|---|
| Functional | The app behaved differently, which is likely a bug or an intended change |
| Locator | The element is there but looks different (renamed, moved or restructured) |
| Timing | Something loaded slower than usual |
| Environment | The site was down, slow or misconfigured |
| Data | The data the test relies on changed or is missing |
Press Explain with AI for a plain-English explanation of what went wrong. This uses AI, so it counts toward your plan's AI usage. It's only run when you ask.
Accepting a suggested fix
When the evidence suggests an element only changed, for example "Checkout" became "Go to checkout", the page shows Evidence suggests the element changed with a proposed way to find it.
- Press Create draft with this fix to make a new draft version with the change.
- Press Review the repair draft to check it, then validate it.
- Or press Dismiss if the suggestion is wrong.
Accepting never changes the approved test directly; the new draft only takes over once it's approved.
Your site changed on purpose
If the flow itself changed, for example a new step in checkout:
- Small change: open the test, press Edit (new version), adjust the steps, and validate. See Reviewing and editing steps.
- Big change: record a new video of the flow and upload it. Archive the old test when the new one is approved.
A failure that doesn't repeat
If the test passes when you run it again without anything changing, it's flaky. See Flaky tests.