TL;DR: Flaky tests in CI/CD are automated tests that pass and fail on the same code without any change, and the worst thing you can do is paper over them with retries. Retries hide the failure signal, quietly inflate pipeline time, and let root causes compound. Benchmark data across roughly 378 million test records shows framework and design choices, not luck, drive flakiness rates (Currents, 2024). Fix the cause, quarantine what you cannot fix yet, and never retry a red build into green.
Definition: Flaky test A flaky test is a test that produces both passing and failing results across runs of the same code and the same environment, with no change to the code under test. The industry calls flakiness the silent killer of CI/CD velocity because a flaky suite erodes trust in every green checkmark it produces. Google’s own research team found flakiness common enough that a meaningful share of tests fail intermittently, with flaky runs accounting for roughly 1.5% of test executions across its codebase (Google Research).
Quick answers
What causes flaky tests? Most flakiness traces to five patterns: improper waits that race async UI updates, test interdependence and leaked shared state, unstable environments or non-deterministic data, network and third-party call variability, and animation or transition timing. The test is rarely broken at random. Something non-deterministic is hiding in the timing, the data, or the environment.
Do retries fix flaky tests? No. A retry masks the symptom and lets the build turn green, but the root cause is still there and now invisible. Retries inflate CI time and cost, delay real bug detection, and turn your pass rate into a number you cannot trust for release decisions.
How do you fix flaky tests in CI/CD? Detect and label flakiness automatically, quarantine the offending tests into a non-blocking suite, classify each failure by root cause, then fix the cause with proper auto-waiting, test isolation, and deterministic data. Return each test to the blocking suite only after it proves stable.
What causes flaky tests in CI/CD? The five-class taxonomy
Before you can fix flaky tests in CI/CD, you have to name what you are looking at. Flakiness feels random, but it almost never is. In practice it clusters into five well-documented causes, and each has a real fix that does not involve a retry loop. I keep coming back to this taxonomy because it turns a vague “the pipeline is flaky again” into a specific, fixable engineering problem.
| Root cause | What happens in the pipeline | The real fix (not a retry) |
|---|---|---|
| Improper waits and timing | The test asserts before an async UI update lands, so it races the app and loses intermittently | Web-first assertions and true auto-waiting tied to real browser events, never fixed sleeps |
| Test interdependence and shared state | One test leaves data or a logged-in session behind, and the next test passes or fails depending on order | Fresh state per test, no shared global fixtures, isolated sessions and storage |
| Unstable environment and data | Non-deterministic seed data or a shared staging box under concurrent load returns different results per run | Synthetic, seeded, deterministic data and isolated environments per run |
| Network and third-party calls | A real API or payment sandbox is slow or rate limited, and the test times out at random | Mock or stub external calls, control timeouts, and gate on your code, not theirs |
| Animation and transition timing | A UI test clicks an element mid-transition, so it hits a moving or not-yet-interactive target | Wait on the settled state or disable animations in the test build |
Notice the pattern. Every fix in that right column removes a source of non-determinism. A retry does the opposite. It accepts the non-determinism and gambles that the next run lands on the lucky side of the race. That is the whole problem with treating flakiness as a scheduling issue instead of a design issue. For a deeper walk through each cause with code-level detail, our companion piece on what a flaky test is and how to fix it breaks down the selector and concurrency cases in more depth.
That benchmark is worth sitting with. Currents analyzed close to 378.5 million test records across 894 real-world applications. Playwright projects averaged 0.72% flakiness against Cypress at 0.83%, so the averages are close. The tail is where it gets loud. Cypress showed a P99 of 4.2% versus Playwright at 2.56%, meaning Cypress projects had far more extreme instability outliers. The lesson is not “Cypress bad.” It is that your framework’s waiting model and your team’s isolation discipline set your flakiness ceiling. If you are weighing runners, our breakdown of Playwright vs Selenium vs Cypress gets into the auto-wait differences that show up in this data.
Do retries fix flaky tests? Why the band-aid makes it worse
Retries are the default reflex for flaky tests in CI/CD because they work, right up until they do not. You add a rerun rule, the red builds go green, the Slack noise stops, and everyone moves on. Here is the honest problem. You did not fix anything. You bought quiet by throwing away information, and that information was the whole point of running tests.
Think about what a retry actually does to each of the failure signals you care about. It hides the failure, so you lose the signal that something is non-deterministic. It burns extra minutes and compute on every rerun, so your pipeline gets slower and more expensive in a way that is easy to miss. And it lets root causes stack up silently, because nobody feels the pain that would push them to fix the cause. A suite that has been retried into green is a false-confidence trap. When it finally lets a real regression through on the lucky retry, you ship the bug and trust the checkmark.
There is a narrow, legitimate use for retries against flaky tests in CI/CD: as a temporary safety valve while you quarantine and fix, or to smooth over genuinely external outages you do not control. Even then, an unlabeled retry is dangerous. If you must rerun, the rerun should be flagged loudly in your reporting so a flaky pass never counts as a clean pass. The moment a retry becomes the fix rather than the flag, you have started lying to yourself about release readiness.
| Dimension | Blind retries | Quarantine and root-cause fix |
|---|---|---|
| Failure signal | Hidden, discarded on the passing rerun | Preserved, captured, and classified |
| CI time and cost | Grows silently with every rerun | Drops as real causes get removed |
| Release confidence | False green, real regressions slip through | Trustworthy green, blocking suite stays clean |
| Root causes over time | Compound and multiply | Shrink toward zero |
| Developer trust | Erodes, people ignore red | Restored, red means red |
How to fix flaky tests in CI/CD, cause by cause
Now the concrete part. Each of the five causes has a fix you can ship this sprint. None of them is a retry. Work through them in order of how often they bite.
1. Replace fixed waits with web-first auto-waiting
Timing races are the single biggest source of flakiness, and the culprit is almost always a hard-coded sleep or a manual wait that guesses how long the app needs. Delete them. Use web-first assertions that poll for the real condition and auto-wait tied to actual browser events. This is where framework choice shows up in the Currents data. Playwright’s auto-wait model, built around actionability checks and events, tends to produce a shorter unstable tail than Cypress’s automatic command retrying, which is one reason Playwright’s P99 sits well below Cypress’s. The general rule holds across tools: assert on the settled state, never on a timer.
2. Isolate every test so order never matters
Test interdependence is the sneakiest cause because the failing test is usually not the guilty one. A test that mutates shared state, a global fixture, a logged-in session, a database row, leaves a landmine for whatever runs next. Fix it with real isolation: fresh state per test, no shared global fixtures, a clean session and storage for each case. If you can shuffle your test order and the suite still passes, you have earned isolation. If shuffling breaks it, you have found hidden coupling that retries would have masked forever.
3. Make test data deterministic
Shared mutable staging data is a flakiness factory. Two pipelines hit the same records, a seed script runs non-deterministically, and the same assertion sees different data on different runs. The fix is synthetic, seeded data generated per run rather than a shared mutable environment that many jobs fight over. Deterministic data also makes failures reproducible, which is the difference between a fifteen minute fix and a two day ghost hunt.
4. Control the network and third-party calls
If your test depends on a live third-party API, a payment sandbox, or a rate-limited service, your pass rate now depends on someone else’s uptime. Mock or stub those calls so the test gates on your code, not their infrastructure. Keep a small, separate integration suite for the real thing, run less often, clearly labeled, so a vendor outage never turns your main pipeline red.
5. Quarantine, do not retry, what you cannot fix today
You will not fix every flaky test the day you find it. That is fine. The move is to quarantine, not retry. Flag the flaky test, move it into a separate non-blocking suite so it stops holding up releases, and keep running it there so it still generates signal. Crucially, quarantine is a queue, not a graveyard. Each quarantined test carries an owner and a deadline, gets classified by root cause, fixed, and returned to the blocking suite only after it proves stable across many runs. The difference from a retry is everything: a retry hides the test, a quarantine surfaces it and schedules the fix.
Why flaky tests in CI/CD quietly wreck delivery speed
Zoom out to the delivery system and flakiness stops being a test annoyance and becomes a velocity tax. The DORA framework, the most cited research on software delivery performance, ties elite performance to fast, trustworthy feedback loops and low change-failure rates. Flaky tests attack both. They slow the loop with reruns and investigations, and they corrupt the signal that is supposed to catch change failures before they reach production.
The cost of flaky tests in CI/CD compounds in a way that is easy to underestimate. Google’s research on flaky tests exists precisely because, at scale, even a low per-test flake rate turns into a firehose of false failures that engineers must triage. When roughly 1.5% of runs flake and you run tests thousands of times a day, that is a lot of humans staring at logs for failures that are not bugs. Every one of those investigations is time not spent shipping.
And it teaches an even more expensive habit, which is that people start ignoring red, because red has cried wolf too many times. Once a team stops trusting its own pipeline, continuous delivery is over in everything but name. This is the same reason we argue, in our guide to continuous testing in DevOps, that a trustworthy suite matters more than a large one. A team that has learned to route around its own tests has already lost the value of automation.
The honest limitations
Let me be straight about what this approach cannot do, because anyone selling you a zero-flakiness guarantee is selling something. First, some flakiness is genuinely external. A cloud region hiccups, a shared dependency goes down, and no amount of test hygiene prevents that. The goal there is containment, isolate those tests so external noise never blocks your main pipeline, not elimination.
Second, quarantine can become a dumping ground if you let it. Without owners and deadlines, a quarantine suite quietly grows into a second graveyard of tests nobody runs, which is just retries with extra steps. Third, deep root-cause fixes take engineering time that is hard to prioritize against feature work, which is exactly why teams reach for retries in the first place. Fighting flaky tests in CI/CD is as much an organizational discipline as a technical one, and naming that tradeoff honestly is the only way to win the argument for doing it right.
Where AI-native testing attacks flakiness at a different layer
Everything above is framework-agnostic engineering discipline, and you should do it regardless of tooling. That said, a large chunk of the flaky tests in CI/CD that teams actually face comes from one specific place: brittle locators that break the instant a developer renames a class or moves a button. This is where an AI-native platform like ContextQA changes the shape of the problem, not by retrying, but by removing the fragility at the source.
ContextQA’s self-healing test automation identifies each element with multi-layered fingerprinting across visual, accessibility, DOM, and text signals rather than a single fragile selector. When the DOM shifts, the test re-locates the element from the other signals instead of failing, which kills off the entire class of selector-driven flakiness that would otherwise land in your quarantine queue. That is a genuinely different mechanism from a retry. A retry reruns the same brittle test and hopes; self-healing repairs the locator so the test keeps testing what it was meant to test.
The second half of the story is triage. ContextQA’s root-cause analysis classifies every failure as a real bug, a test issue, an environment problem, or a flake. That classification is the exact thing retries destroy. Instead of a wall of red that you rerun blind, you get a labeled failure that tells you whether to file a bug, fix the test, or check the environment. As ContextQA CEO and founder Deep Barot puts it, a green build bought with retries is borrowed confidence you pay back with interest at the worst possible time, in production. The platform’s job is to make the honest signal cheap enough that nobody feels the need to hide it.
Proof this works at scale
This is not theory for us. In a validated engagement documented in an IBM case study, ContextQA migrated roughly 5,000 test cases and removed the flakiness that had plagued the original suite, using watsonx.ai natural language processing to rebuild tests on the self-healing fingerprinting model. Removing flakiness was not a side effect. It was the point, because a migrated suite that inherits the old flakiness inherits the old distrust.
On the independent side, ContextQA holds a 4.8 out of 5 rating on G2, where the self-healing and stability of runs come up repeatedly in reviews. I point to these not as a victory lap but because the argument of this whole piece is about trusting your signal, and third-party validation is one more signal worth trusting more than a self-graded checkmark. If you want the deeper mechanics, our explainer on how self-healing test automation works shows the fingerprinting in action.
Where this runs: web, mobile, API, and your existing pipeline
Fixing flaky tests in CI/CD only helps if the fix lives inside the pipeline you already run. ContextQA’s continuous testing platform plugs into Jenkins, CircleCI, GitHub Actions, and GitLab, so self-healing and root-cause classification happen on every commit, not in a separate tool you have to remember to open. Coverage spans web, mobile, and API testing from one place, which matters because flakiness does not respect layer boundaries. A flaky API stub can sink a UI test, and a shared environment can flake all three at once.
The through line is that stability is a platform property, not a per-test heroics problem. When element identification, failure classification, and cross-layer coverage are built in, the quarantine queue shrinks on its own, because a large slice of what used to be flaky simply stops flaking. That frees your team to spend its root-cause budget on the genuinely hard cases instead of the thousand paper cuts from brittle selectors.
It also changes who can act on a failure. When every run comes back with a plain-language classification instead of a raw stack trace, a product engineer who is not a test-automation specialist can look at a red build and know whether it is their bug, a stale test, or an environment blip. That is the quiet win most teams miss when they fight flaky tests in CI/CD one script at a time. The goal is not a heroic QA engineer who can decode any failure. The goal is a pipeline whose signal is clear enough that decoding is not required, so the whole team keeps trusting the green.
Get root-cause classification on your worst flaky suite
Rather than guessing which of the five flakiness classes above is behind your worst offenders, ContextQA’s root-cause analysis identifies the actual failure pattern automatically, and self-healing fixes the selector-driven cases outright instead of leaving them for a retry to paper over.
Book a ContextQA DemoDo this now: a flaky-test cleanup checklist
- Turn off blind auto-retries in CI (15 min). Or at minimum, flag every rerun loudly so a flaky pass never counts as a clean pass. You cannot fix what you cannot see.
- Add flaky detection and labeling (30 min). Track pass/fail history per test so the pipeline tells you which tests flake and how often, instead of you guessing.
- Stand up a quarantine suite (30 min). Create a separate non-blocking job for flaky tests, with an owner and a deadline attached to each entry.
- Hunt hard-coded sleeps (20 min). Grep the suite for fixed waits and replace them with web-first assertions on the settled state.
- Shuffle your test order (10 min). Randomize execution order once. Anything that breaks is hidden coupling to fix with isolation.
- Pilot self-healing on your flakiest suite (20 min). See how much of your quarantine queue is really just selector fragility with ContextQA self-healing.
- Book a working session (10 min). Walk your worst flaky pipeline through root-cause classification with our team via a ContextQA demo.
The bottom line
Flaky tests in CI/CD are a signal problem, not a scheduling problem, so the fix is never a retry. Retries buy you a quiet Slack channel and cost you the one thing tests exist to give you: a green build you can trust. Name the cause, quarantine what you cannot fix yet, and repair the cause with proper waits, isolation, and deterministic data. Where brittle locators drive the flakiness, self-healing removes the fragility outright, which is why ContextQA’s engagement documented with IBM migrated roughly 5,000 tests and took the flakiness with it. Stop retrying red into green. Start trusting your pipeline again, and book a ContextQA demo to see root-cause classification on your own suite.
Written by Deep Barot.
