Why AI-Generated Playwright Tests Are Flaky (and How to Fix Them)

Isometric illustration of a browser test window where an unstable signal becomes a steady line with a green check

TL;DR: AI-generated Playwright tests are often flaky for predictable reasons: brittle CSS selectors copied from the page, fixed sleeps instead of real wait conditions, shared test data, and assertions that check that a page loaded rather than that the right thing happened. Fix it by giving the agent a behavior contract before it writes code, requiring user-facing locators and web-first assertions, isolating test data, and banning retries and sleeps as fixes. 45% of developers say debugging AI-generated code is more time-consuming (Stack Overflow Developer Survey, 2025), and flaky tests are a large part of that cost.

Ask Claude Code, Codex or Cursor to "write Playwright tests for checkout" and you'll get specs in seconds. Run them in CI twenty times and some will fail at random. The agent isn't careless. It's copying patterns that look right in one run on one machine. This guide covers why that happens, how to prompt and review so it doesn't, and how to diagnose the flaky tests you already have.

Quick answers

Why are AI-written Playwright tests flaky? Mostly because the agent writes what worked once: selectors tied to the current DOM, timing guesses, and data that another test can change. These pass on the agent's run and fail when timing, data or parallel workers change in CI.

Can I fix flakiness by adding retries? No. Playwright reports a test that fails and then passes on retry as "flaky", but that label doesn't say why (Playwright retries). Retries hide the first failure, which may be a real race condition in your product.

How does ContextQA help? ContextQA runs tests on the deployed build with full evidence for each run: video, screenshots, network logs and a Playwright trace. Its failure investigation helps separate product defects, test defects and environment problems. Self-healing repairs changed locators but keeps the business assertion.

What are the most common causes?

Pattern in the generated testWhy it flakesRequire instead
CSS or XPath selectors copied from the DOM, such as div:nth-child(3)Breaks when layout or class names changegetByRole, getByLabel or getByText, or test IDs you control
page.waitForTimeout(2000)Too short on a slow CI runner, wasted time on a fast oneWeb-first assertions that wait for the expected state
Asserting a URL or page title onlyPasses even when the transaction failedAssert the visible outcome and, where needed, the saved state
Hard-coded accounts and records shared across testsParallel workers change each other's dataA unique user or record per test or worker
Test order assumptionsPasses in sequence, fails when shardedEach test sets up its own state
Local time, locale or fontsWorks on a laptop, fails in a CI containerA controlled clock and fixed locale in config
Patterns AI agents produce, why they flake, and what to require instead

Playwright's own guidance lines up with the right-hand column. It recommends user-facing attributes and explicit contracts over implementation details, and isolated tests that don't depend on each other (Playwright best practices, Playwright locators).

Why do agents produce these patterns?

Three reasons, all fixable with better instructions.

They optimize for a green first run. An agent that runs its test once and sees it pass will stop. Nothing in that loop tests stability under parallel workers or a slower machine.

They read the DOM, not the requirement. Given a page, an agent finds the nearest selector that works now. Given a requirement, it can target what users actually see, such as a button named "Place order".

They fill gaps with guesses. If the prompt doesn't say what "success" looks like, the agent asserts something easy to check, such as the URL changing. The ContextQA Playwright guide warns against exactly this: a test that clicks "Generate" and only checks that the page still exists will miss a wrong result, a failed request or an unauthorized record (ContextQA Playwright testing with AI agents).

Software developer looking frustrated while debugging code at a computer

How should you prompt the agent?

Give it a behavior contract and rules before it writes code. A good instruction looks like this:

  • Read our Playwright config, existing fixtures and the component under test.
  • Propose scenarios and the expected result for each before writing any spec.
  • Use getByRole, getByLabel or our data-testid attributes. No CSS chains or XPath.
  • Use web-first assertions such as toBeVisible and toHaveText. No waitForTimeout.
  • Create unique test data per test and clean it up.
  • Assert the business outcome, not just navigation.
  • Don't add retries, raise timeouts, weaken assertions or skip tests to get a green run.
  • Run each new spec with --repeat-each=10 --retries=0 and report every result.

Put the rules in your CLAUDE.md or AGENTS.md so every session follows them. Playwright's web-first assertions retry until the expected condition is met, which removes most timing guesses on their own.

Want AI-written tests checked on the real build before they gate a release? Ship by ContextQA analyzes each pull request and runs the journeys it affects: start free, no credit card needed.

How do you diagnose a flaky test you already have?

Start from the first failure, not the retry. The ContextQA flaky test guide recommends saving the original attempt's commit, browser version, worker count, test data and trace, then repeating the narrow case without retries (ContextQA: find and fix flaky tests in CI):

  • npx playwright test tests/checkout.spec.ts --repeat-each=10 --retries=0 --workers=1 --trace retain-on-failure

Then match the evidence to a fix:

  1. Element visible before data is ready: wait for the ready state and expected value, not a fixed time.
  2. Fails only with parallel workers: give each test its own user and records, then rerun with the original worker count.
  3. Session expired or wrong role: recreate the intended login state and confirm the identity.
  4. Works locally, fails in CI: compare browser version, resources, locale, timezone and base URL.
  5. Backend sometimes returns the wrong result: that's a product bug. A retry can't make it acceptable.

The Playwright trace viewer shows each action, network call and DOM snapshot, which usually makes the cause obvious. Ten clean runs are a starting sample, not a guarantee. For the full playbook, see the flaky test playbook and fixing flaky tests in CI without retries.

Should you let an AI agent fix the flaky test?

Yes, with limits. Agents are good at reading a trace and proposing a fix. They're also tempted to "fix" flakiness by making the test weaker. Give the agent the first-failure artifacts and this boundary from the ContextQA docs: separate facts from hypotheses, propose one narrow reproduction, and don't suppress errors, weaken assertions, add sleeps or increase retries to get a green check. After the fix, the original assertion must still catch the defect it was written for.

Playwright now ships test agents, including a healer that repairs failing tests (Playwright test agents). Review any automatic repair the same way: did it keep the business assertion, and does it still fail when the product is broken?

How does ContextQA reduce flakiness for AI-written tests?

ContextQA adds three things that are hard to build yourself:

  • Evidence for every run. Each execution produces video, screenshots, network and console logs and a Playwright trace, so you can see the first failure instead of guessing.
  • Failure investigation. Through MCP, an agent can pull the step results and evidence for a specific failed run and work out whether it's a product, test or environment problem, without triggering a new run by accident.
  • Reviewed self-healing. When a control's locator changes but the business flow still works, ContextQA proposes a repair. You choose whether repairs are approved automatically or reviewed, at organization or test level, and the assertion stays intact (ContextQA self-healing docs).

If your team wants the tests in its own repository, ContextQA can export reviewed test cases as a Playwright project (ContextQA export and portability). For when to use Playwright MCP directly and when to use ContextQA, see Playwright MCP vs ContextQA MCP.

Bottom line

AI agents write flaky Playwright tests for the same reasons rushed humans do: brittle selectors, timing guesses, shared data and weak assertions. Give the agent a contract first, enforce user-facing locators and web-first assertions, isolate data, and judge every test by repeated runs without retries. When a test does flake, start from the first failure and fix the cause, never the symptom.

Running a large AI-generated suite that won't stay green? Book a ContextQA demo and bring your flakiest spec.

Sources

Share the Post:

Author

Deep Barot

CEO @ ContextQA | Agentic AI for Software Testing | Context-aware Testing

Deep Barot is the Founder and CEO of ContextQA, the only AI testing platform that understands context. He brings decades of experience across DevOps, full-stack engineering, cloud systems, and large-scale platform development.
AI Insights
Real User Intelligence Platform

Turn live sessions into test coverage. No prompts, no manual design - just pointed at your URL and generating suites within minutes.

Minutes
From URL to generated test cases
Zero
Prompts or manual test design needed
40%+
Average coverage increase after first run
100%
Based on real user behavior, not guesses

Frequently Asked Questions

Usually because something differs between the two: browser version, CPU speed, worker count, locale, timezone, fonts, base URL or test data. Compare those first. Fixed sleeps and shared accounts are the most common causes in AI-written specs.
A small retry count can keep a pipeline moving while you investigate, as long as the first failure is recorded and reported. Retries should never be the fix. A test that only passes on retry needs a root cause, and that cause may be a real bug in the product.
User-facing ones: getByRole, getByLabel and getByText, or test IDs your team controls. Avoid CSS chains, nth-child selectors and XPath copied from the current page. Put this rule in your repository instructions so every agent session follows it.
Yes, if you give it the first-failure trace and clear limits. Ask it to reproduce the failure narrowly, explain the cause, and fix it without adding sleeps, raising retries or weakening assertions. Then confirm the test still fails when the product is broken.
It's designed not to. Healing targets changed locators when the business flow still works. A wrong amount, missing feature, permission problem or server error still fails the test. You can require human review of every proposed repair at organization or test level.
Related Blogs