Skip to main content

Flaky test analysis

A test is called flaky when, across several runs, it sometimes passes and sometimes fails. The code under test, the tests themselves, and the environment configuration do not change; the test run is simply repeated.

Flakiness most often comes from the following causes:

  • concurrent code in the software under test or in the tests themselves, for example a data race
  • the application under test, an external service, or a database does not meet expectations, for example it handles only 10 requests per minute while the tests send 20 requests per minute
  • the tests are not isolated from each other, for example the first test passes when run alone, but fails when the whole suite including that test is run

Our plugin helps you identify the problem and suggests ways to fix it. In the coming months the plugin will also pass all the collected information to the agent so that it can fix your code automatically.

How to use​

  1. Next to the flaky test, click the Explyt icon and select Explyt: Analyze If Flaky.
  2. The plugin starts running this test and the other tests in the class with code instrumentation to determine the cause of the flakiness.
    • Note: the tests will be run many times, which may take a fairly long time. This is required in order to reproduce the flakiness.
    • The plugin reports the approximate number of runs required (Approximately 6 runs of...). This number is not exact, but you can use it as a rough guide. As a rule, far fewer runs are needed than the plugin reports.
  3. If the runs do not reproduce the flakiness, the plugin reports that the test was not recognized as flaky: is not recognized as flaky with rerun. In that case the plugin still offers to pass the code, the tests, and the collected data to the AI to identify the problem (go to step 5).
  4. If the runs do reproduce the flakiness, the plugin shows the problematic instruction and the stack trace leading to it. You can study them yourself or go straight to the next step.
    • Open instruction if you want to check the line at which the behavior of the successful and the failing run starts to differ. The plugin jumps to that line in the code or in the test. Check which data or state could produce different results.
    • Open stack trace if a single line is not enough and you need to understand how execution reached it. The plugin opens the stack in a separate read-only file; follow the links in the stack and trace the call chain down to the code that produces the differing state.
  5. Click Continue, and the plugin passes all the collected information to the AI, which explains the cause of the problem and suggests possible solutions.
  6. Change the code or the tests to fix the problem, or contact your colleagues if the problematic component of the system is outside your area of responsibility.
  7. After making the fix, you can confirm that the flakiness is gone. To do that, repeat everything starting from step 1. If the fix worked, you will finish at step 3.