When a test fails in CI, understanding what happened during the test run is critical to debugging a root cause. Test in CI can fail for many reasons:
- Issues with the test run environment
- An error thrown by the test runner
- A slow application under test causing a timeout
- A regression introduced by a recent change
Replay provides a recording of everything that happened in your browser during a test run. We also provide context about the test run so you can diagnose failures.
Viewing a test run
GitHub Action comment
When using a Replay GitHub Action, a link to the test run in the Test Suite Dashboard is included in the comment on the originating pull request.
Links to replays are also included in the GitHub comment. If you want to jump right in to debugging a specific test, click on a replay link directly.
Click the comment link to navigate to your team’s Test Runs view with the specified run selected.
Test Suite Dashboard
You can also visit app.replay.io and select your team to view all latest test runs, then select the run to debug.
Once you are in the run, Replay shows you contextual information from CI, including:
- Commit message
- User triggering run
- Workflow trigger
- Branch name
- Time since test run
- Run pass/fail status
- How many tests failed (if failures)
- How many tests ran (if no failures)
This information can help you make critical triage decisions and offer insights for debugging. For example:
- A test failure triggered by a release may be user-impacting
- A failure on a work-in-progress branch may be deprioritized
- If there is a subsequent passing run for the same branch, the failure may have been resolved
- If all tests failed, it may point to an issue in the environment or with the test runner
- The commit message may give context, like “trying something, may break”
Test run details
Clicking on a run opens a sidebar with details of the test run, including the pass/fail status and replays generated for all tests recorded within a run. The name of the test file is also shown.
For Cypress, replays and pass/fail status is based on spec file, not tests. This is because the same browser session is used for the entire spec file.
Click the play icon next to open the replay of that test from the run. From here, you can jump right into debugging.
Click the test name to navigate to the test results view for a given test.
Viewing test results
The test results view shows all recent runs of a given test. For each run, see:
- Test pass/fail status
- Branch
- Time since test run
- User triggering run
Clicking on a result will open the replay for that test. Use test history view to:
- Identify how often a test fails
- Identify if this is the first failure
- View a replay of a passing run for debugging comparison