TL;DR: AI-generated Playwright tests are often flaky for predictable reasons: brittle CSS selectors copied from the page, fixed sleeps instead of real wait conditions, shared test data, and assertions that check that a page loaded rather than that the right thing happened. Fix it by giving the agent a behavior contract before it writes code, requiring user-facing locators and web-first assertions, isolating test data, and banning retries and sleeps as fixes. 45% of developers say debugging AI-generated code is more time-consuming (Stack Overflow Developer Survey, 2025), and flaky tests are a large part of that cost.
Ask Claude Code, Codex or Cursor to "write Playwright tests for checkout" and you'll get specs in seconds. Run them in CI twenty times and some will fail at random. The agent isn't careless. It's copying patterns that look right in one run on one machine. This guide covers why that happens, how to prompt and review so it doesn't, and how to diagnose the flaky tests you already have.
Quick answers
Why are AI-written Playwright tests flaky? Mostly because the agent writes what worked once: selectors tied to the current DOM, timing guesses, and data that another test can change. These pass on the agent's run and fail when timing, data or parallel workers change in CI.
Can I fix flakiness by adding retries? No. Playwright reports a test that fails and then passes on retry as "flaky", but that label doesn't say why (Playwright retries). Retries hide the first failure, which may be a real race condition in your product.
How does ContextQA help? ContextQA runs tests on the deployed build with full evidence for each run: video, screenshots, network logs and a Playwright trace. Its failure investigation helps separate product defects, test defects and environment problems. Self-healing repairs changed locators but keeps the business assertion.
What are the most common causes?
| Pattern in the generated test | Why it flakes | Require instead |
|---|---|---|
| CSS or XPath selectors copied from the DOM, such as div:nth-child(3) | Breaks when layout or class names change | getByRole, getByLabel or getByText, or test IDs you control |
| page.waitForTimeout(2000) | Too short on a slow CI runner, wasted time on a fast one | Web-first assertions that wait for the expected state |
| Asserting a URL or page title only | Passes even when the transaction failed | Assert the visible outcome and, where needed, the saved state |
| Hard-coded accounts and records shared across tests | Parallel workers change each other's data | A unique user or record per test or worker |
| Test order assumptions | Passes in sequence, fails when sharded | Each test sets up its own state |
| Local time, locale or fonts | Works on a laptop, fails in a CI container | A controlled clock and fixed locale in config |
Playwright's own guidance lines up with the right-hand column. It recommends user-facing attributes and explicit contracts over implementation details, and isolated tests that don't depend on each other (Playwright best practices, Playwright locators).
Why do agents produce these patterns?
Three reasons, all fixable with better instructions.
They optimize for a green first run. An agent that runs its test once and sees it pass will stop. Nothing in that loop tests stability under parallel workers or a slower machine.
They read the DOM, not the requirement. Given a page, an agent finds the nearest selector that works now. Given a requirement, it can target what users actually see, such as a button named "Place order".
They fill gaps with guesses. If the prompt doesn't say what "success" looks like, the agent asserts something easy to check, such as the URL changing. The ContextQA Playwright guide warns against exactly this: a test that clicks "Generate" and only checks that the page still exists will miss a wrong result, a failed request or an unauthorized record (ContextQA Playwright testing with AI agents).

How should you prompt the agent?
Give it a behavior contract and rules before it writes code. A good instruction looks like this:
- Read our Playwright config, existing fixtures and the component under test.
- Propose scenarios and the expected result for each before writing any spec.
- Use
getByRole,getByLabelor ourdata-testidattributes. No CSS chains or XPath. - Use web-first assertions such as
toBeVisibleandtoHaveText. NowaitForTimeout. - Create unique test data per test and clean it up.
- Assert the business outcome, not just navigation.
- Don't add retries, raise timeouts, weaken assertions or skip tests to get a green run.
- Run each new spec with
--repeat-each=10 --retries=0and report every result.
Put the rules in your CLAUDE.md or AGENTS.md so every session follows them. Playwright's web-first assertions retry until the expected condition is met, which removes most timing guesses on their own.
Want AI-written tests checked on the real build before they gate a release? Ship by ContextQA analyzes each pull request and runs the journeys it affects: start free, no credit card needed.
How do you diagnose a flaky test you already have?
Start from the first failure, not the retry. The ContextQA flaky test guide recommends saving the original attempt's commit, browser version, worker count, test data and trace, then repeating the narrow case without retries (ContextQA: find and fix flaky tests in CI):
npx playwright test tests/checkout.spec.ts --repeat-each=10 --retries=0 --workers=1 --trace retain-on-failure
Then match the evidence to a fix:
- Element visible before data is ready: wait for the ready state and expected value, not a fixed time.
- Fails only with parallel workers: give each test its own user and records, then rerun with the original worker count.
- Session expired or wrong role: recreate the intended login state and confirm the identity.
- Works locally, fails in CI: compare browser version, resources, locale, timezone and base URL.
- Backend sometimes returns the wrong result: that's a product bug. A retry can't make it acceptable.
The Playwright trace viewer shows each action, network call and DOM snapshot, which usually makes the cause obvious. Ten clean runs are a starting sample, not a guarantee. For the full playbook, see the flaky test playbook and fixing flaky tests in CI without retries.
Should you let an AI agent fix the flaky test?
Yes, with limits. Agents are good at reading a trace and proposing a fix. They're also tempted to "fix" flakiness by making the test weaker. Give the agent the first-failure artifacts and this boundary from the ContextQA docs: separate facts from hypotheses, propose one narrow reproduction, and don't suppress errors, weaken assertions, add sleeps or increase retries to get a green check. After the fix, the original assertion must still catch the defect it was written for.
Playwright now ships test agents, including a healer that repairs failing tests (Playwright test agents). Review any automatic repair the same way: did it keep the business assertion, and does it still fail when the product is broken?
How does ContextQA reduce flakiness for AI-written tests?
ContextQA adds three things that are hard to build yourself:
- Evidence for every run. Each execution produces video, screenshots, network and console logs and a Playwright trace, so you can see the first failure instead of guessing.
- Failure investigation. Through MCP, an agent can pull the step results and evidence for a specific failed run and work out whether it's a product, test or environment problem, without triggering a new run by accident.
- Reviewed self-healing. When a control's locator changes but the business flow still works, ContextQA proposes a repair. You choose whether repairs are approved automatically or reviewed, at organization or test level, and the assertion stays intact (ContextQA self-healing docs).
If your team wants the tests in its own repository, ContextQA can export reviewed test cases as a Playwright project (ContextQA export and portability). For when to use Playwright MCP directly and when to use ContextQA, see Playwright MCP vs ContextQA MCP.
Bottom line
AI agents write flaky Playwright tests for the same reasons rushed humans do: brittle selectors, timing guesses, shared data and weak assertions. Give the agent a contract first, enforce user-facing locators and web-first assertions, isolate data, and judge every test by repeated runs without retries. When a test does flake, start from the first failure and fix the cause, never the symptom.
Running a large AI-generated suite that won't stay green? Book a ContextQA demo and bring your flakiest spec.