Vibe Coding Testing: How to Verify AI-Written Code

TL;DR: Vibe coding testing means checking code that an AI assistant wrote from plain language prompts before it merges. Code that compiles and looks right can still break real behavior. Write tests from the requirement, not from the generated code, and make sure each test fails when you plant a known defect. Ship, the ContextQA autonomous quality engineer, tests every pull request with impact analysis and targeted regressions before merge.

Vibe coding testing confirms that AI generated code does what your ticket says. This guide shows how to write tests from the requirement, prove they catch a known defect, and gate every pull request.

Quick answers

How is testing AI written code different from normal test automation? Normal automation assumes a human wrote the code and the tests against an agreed spec. With AI written code, the developer may never have read every line. Every assertion has to trace back to the requirement and fail against a known defect, such as a reverted fix or a mutated boundary value.

How do I stop an AI coding agent from writing tests that just confirm its own bugs? Give the agent the ticket as the contract and have it propose test cases before it writes code. Review those cases for boundaries, invalid inputs, and failure states. After the tests pass, inject a known defect and confirm at least one test goes red. A test that cannot fail proves nothing.

How can a small team verify every AI generated pull request without hiring more QA engineers? Run targeted checks on each pull request instead of relying only on the full suite or manual review. Ship, the ContextQA autonomous quality engineer, runs impact analysis and targeted regressions on every pull request before merge, and reproduces bugs reported in Jira, Linear, or Slack. It is self serve: get started free.

What is vibe coding, and when does it need testing?

Vibe coding means you describe what you want in plain English and accept the code an AI assistant writes, often without reading every line. It needs dedicated testing once changes reach a shared branch, real users, or real data. Code that compiles and demos well can still hide failures nobody reviewed.

The defects sit where a happy path demo never goes:

• Missing edge cases: empty inputs, null fields, time zones, and pagination past page one. The ContextQA unit testing guide covers boundary and invalid input assertions.
• Weak authorization: routes that skip the ownership check on /api/orders/:id.
• Silent regressions: a prompt to fix one screen rewrites a shared helper and breaks another.
• Committed config: API keys or localhost URLs hardcoded into source.
• Tests that agree with the bug: AI written assertions that copy the implementation, so they pass either way.

The last one does the most damage because it hides the others. The ContextQA guide to testing with AI coding agents says to add a known defect and confirm the tests fail before you trust a generated suite.

What do you need before you start?

You need four things: the code in a git repository with pull requests turned on, one test command that already passes (for example npm test), a seeded staging or preview environment, and a short written list of behaviors that must never break. Without these, generated tests have nothing stable to check against.

Then decide where verification happens. Most teams use more than one layer:

• Your coding agent writing repository tests: No new tools needed. The agent writes tests in your current framework, and you plant a known defect to confirm they fail. The risk is that one model writes both the code and the assertions.
• A hand maintained end to end suite: You get full control, but every UI change the agent ships can break selectors, and your team owns that upkeep. See how AI based self healing repairs broken locators for the maintenance problem in detail.
• Ship, the autonomous quality engineer: It runs impact analysis and targeted regressions on every pull request before merge, and hands engineers root cause evidence for reported bugs.

Start with agent written tests proven to fail on a known defect. Add pull request checks once the agent ships changes faster than your team can review them.

How do you do vibe coding testing step by step?

The workflow has five steps: write down the expected behavior before you prompt, have the agent write tests in your existing framework, prove those tests fail on a known bug, run end to end checks against a preview URL, and gate every pull request in CI.

1. Pin the contract. List the inputs, outputs, and failure states in a TESTING.md file that the agent reads before it writes code.
2. Generate tests in your current framework. Ask the agent for Playwright specs. The ContextQA docs have client specific guides for Claude Code, Cursor, and Codex. If your code came from a browser based app builder, sync it to a GitHub repository first.
3. Break the code on purpose. Remove one validation check and confirm that a test fails.
4. Run end to end tests against a preview deploy:

import { test, expect } from '@playwright/test';

test('signup rejects a duplicate email', async ({ page }) => {

await page.goto(`${process.env.PREVIEW_URL}/signup`);

await page.getByLabel('Email').fill('existing@example.com');

await page.getByRole('button', { name: 'Create account' }).click();

await expect(page.getByRole('alert')).toContainText('already registered');

});

1. Gate every pull request. Run npx playwright test in CI and make the job a required check. Ship adds impact analysis and runs targeted regressions before merge.

What are the most common mistakes?

The most common mistakes are trusting tests the AI wrote for its own code, merging without a security review, skipping regression runs on "small" prompts, and letting CI accept whatever the agent produces. Each one lets plausible looking code ship with untested edge cases or leaked secrets.

• Self graded tests: The agent writes the code and the assertions from the same assumptions. Inject a known defect and confirm the test fails.
• No security checklist: Review every AI diff for hardcoded keys, missing input validation, SQL built by string concatenation, permissive CORS, and dependencies nobody asked for. Run npm audit or pip-audit in the pipeline.
• Ungoverned merges: Require branch protection, a passing test job, and human review on every AI generated pull request.
• Unreproduced bugs: Patching from a vague Slack thread just restarts the prompt loop. Ship reproduces bugs reported in Jira, Linear, or Slack and hands engineers root cause evidence.

Treat AI generated code as untrusted input, and make CI prove it works before anyone merges it.

How do you know it is working?

Your testing is working when three signals improve together: fewer defects reach production, fewer merged changes get reverted or rewritten within two weeks, and every test fails when you plant a known bug. A coverage percentage alone proves nothing, because AI written tests often just assert whatever the generated code already does.

Set a baseline first, because the risk is documented. Google's DORA 2024 Accelerate State of DevOps report estimated that a 25% increase in AI adoption comes with a 7.2% drop in delivery stability. GitClear's 2024 analysis looked at 153 million changed lines and projected that code churn in 2024 would double compared with 2021, before AI assistants were common. Churn is code rewritten soon after it was written, an early sign of technical debt.

• Planted defect check: Break one assertion target on purpose and confirm the suite goes red.
• Churn on AI written files: Measure how many lines get rewritten within 14 days of merge.
• Flaky rate: Count tests that both pass and fail on the same commit.

A regenerated test is still a changed test, so review the diff before you trust it.

How do you automate vibe coding testing with ContextQA?

Point Ship, the ContextQA autonomous quality engineer, at your pull requests so it tests every AI generated change before merge. Ship runs impact analysis on the diff, then targeted regressions. When a bug is reported in Jira, Linear, Slack, or a service desk, Ship reproduces it and gives engineers the root cause evidence.

Ship fits into your existing workflow:

1. Generate the change with your coding agent and open a pull request as usual.
2. Ship scopes the change with impact analysis and runs targeted regressions before merge.
3. Read the evidence Ship returns. Merge only when the affected flows pass.
4. When a user reports a defect, Ship reproduces it and points to the likely root cause, so your next prompt starts from evidence instead of a guess.

If AI written code reaches your main branch faster than you can verify it, Ship tests every pull request before merge. Get started free with Ship, no credit card needed.

What should you do right now? A checklist

Before you merge another AI generated change, make tests a gate: write down expected behavior first, have the agent run the tests it writes, prove those tests fail when the code breaks, and run targeted regressions on every pull request. These are the core best practices for vibe coding.

• Add your exact test command (for example npm test -- --run) to CLAUDE.md so Claude Code runs it after every edit.
• Ask the agent for test cases (happy path, boundaries, invalid input) before it writes any code.
• Reject tests that only check that a mock was called or that a value is truthy.
• Break the code on purpose and confirm the new test fails.
• Run new tests three times in CI and quarantine any test that flips between pass and fail.
• Run your tests in GitHub Actions and make them required checks with branch protection, so merges are blocked until they pass.
• Use Ship to test every pull request and to reproduce bugs reported in Jira, Linear, or Slack.
• Keep agent test logs, Ship results, and exported test code as separate, traceable artifacts.

Bottom line

Verify AI written code like any other code: start from the requirement, review every assertion, and prove each test fails when a known defect is present. Vibe coding speeds up writing code, but your team still has to show it works. Run the AI generated code testing checklist before merge, and let Ship run targeted regressions on every pull request.

Sources

These primary sources back the delivery stability and code churn figures in this article.

• Google DORA, 2024 Accelerate State of DevOps report
• GitClear, 2024, Coding on Copilot: AI's downward pressure on code quality
• ContextQA, Testing with AI Coding Agents