How to Build a Verification-Led AI Software Factory with Claude Code and ContextQA

Isometric factory line where a robotic arm assembles parts, an inspection station with a magnifying glass approves them, and a person watches the control panel

TL;DR: To build an AI software factory you can trust, add an independent verification stage to the coding loop. A coding agent such as Claude Code writes the change. ContextQA reproduces the bug or turns the spec into tests, runs the affected user journeys on the deployed pull request build, and returns evidence. A human approves the release. Trust is the bottleneck: 46% of developers actively distrust the accuracy of AI tools (Stack Overflow Developer Survey, 2025).

Most teams already have the first half of an AI software factory. An agent picks up a ticket, writes code and opens a pull request. The second half is missing: proof that the change works for users and didn't break anything else. This guide shows how to build that second half with Claude Code, the ContextQA MCP server and Ship, step by step.

Quick answers

What makes an AI software factory "verification-led"? Every change the coding agent produces is checked by a system that didn't write it. The verifier runs real user journeys against the build that will ship, compares results with expected behavior written before the code, and sends failures back to the agent. Merge waits for that evidence and a human decision (ContextQA AI software factory docs).

Can I build one with Claude Code today? Yes. Connect Claude Code to the hosted ContextQA MCP server with claude mcp add --transport http contextqa https://mcp.contextqa.com/mcp, sign in, and Claude Code can create, run and read ContextQA tests. Add Ship to analyze each pull request and run the journeys it affects (Claude Code testing guide).

Does this remove humans from the loop? No. Agents build and verifiers check, but a person still owns the release decision, retry limits and anything ambiguous. You choose which steps run automatically and which need approval, and you can widen autonomy as evidence builds up.

What is a verification-led AI software factory?

It's a delivery pipeline where AI agents do most of the building and an independent system proves the result. If you're new to the idea, start with what an AI software factory is. This post is the build guide.

The key word is independent. A coding agent that checks its own work is grading its own exam. Claude Code's own guidance says to give the agent a check it can run, because reading real output is the only evidence you have for a session you weren't watching (Claude Code best practices). A verification-led factory makes that check a separate stage with its own tests, its own environment and its own record.

The 2025 DORA report describes AI as an amplifier of a team's existing strengths and weaknesses. A factory without verification amplifies the weakness: more code, merged faster, checked less.

What are the parts of the factory?

You need five stages. Each one answers a different question, and mixing them up is the most common design mistake.

StageQuestion it answersTypical tool
IntakeWhat should change, and how will we know it worked?Jira, Linear or Slack ticket with acceptance criteria
BuildWhat code implements it?Claude Code, Codex or Cursor
ContextWhich screens, journeys and tests does this change touch?ContextQA application crawl, code analysis and PR impact analysis
VerifyDoes the deployed build behave as expected, with no new regressions?ContextQA browser, API and mobile runs, through MCP or Ship
DecideIs this safe to merge and release?A human release owner, with the run evidence
Source: ContextQA AI software factory documentation, learning.contextqa.com

ContextQA's docs are careful about one distinction you should copy into your process: a generated test is proposed coverage, an impact report is a proposed scope, and a completed run is execution evidence. Only the last one proves anything about the build.

Software engineer reviewing code on two monitors while a colleague looks on in a brick-walled office

How do you build it step by step?

Start small. One application, one repository, one important journey. These steps follow the setup in the ContextQA AI software factory guide.

  1. Pick a bounded outcome. Choose one journey that matters, such as checkout or sign up. Write the expected results before anyone writes code, so the implementation can't quietly become the specification.
  2. Connect a reachable test environment. Give ContextQA the URL, test data and a test identity. The MCP connection doesn't make localhost reachable, so use a deployed preview build or an approved tunnel.
  3. Set up sign-in. Create and verify an auth profile before crawling or running tests that need a logged-in user. Keep secrets in managed fields, never in prompts.
  4. Build context. Run an application crawl and code analysis, then review the proposed coverage against your critical journeys, roles and negative paths.
  5. Record a baseline. Run the accepted tests against the current main build. Triage existing failures now, or you won't be able to tell new regressions from old ones later.
  6. Connect delivery signals. Link the repository and issue tracker, and set branch filters and reproduction triggers so each pull request is tested on its own build.
  7. Connect Claude Code. Add the MCP server and complete OAuth sign-in in /mcp. Make a read-only request first to confirm the tools and workspace.
  8. Run one complete cycle. Reproduce one real issue, have Claude Code fix it, deploy the pull request build, run the affected journeys and inspect the evidence before merging.
  9. Expand gradually. Add more journeys and repositories only after the baseline and failure classification are reliable.
Want to see this loop on your own app? Ship by ContextQA analyzes each pull request and tests the journeys it affects: start free, no credit card needed.

What should you tell Claude Code to do?

Give the agent an evidence contract, not just a task. The contract says what to read first, what to change and what to send back. The ContextQA docs suggest an instruction like this one:

  • Work on the specified issue in a new branch.
  • Read the reproduction evidence and expected business behavior first.
  • Make a focused fix and run the repository's relevant checks.
  • Open a pull request and identify the candidate commit and deployment.
  • Use ContextQA to analyze impact and run the affected journeys.
  • Separate new regressions, existing failures, test defects and blocked checks.
  • Return the pull request, tested commit, run links and unresolved findings.
  • Leave merging and release approval to the release owner.

The last line matters most. It keeps the agent inside its lane. If you use Codex instead of Claude Code, the same contract works (Codex end to end testing guide).

How much autonomy should the factory have?

Decide autonomy per stage, not for the whole factory. ContextQA's PR impact analysis has three modes for test maintenance:

  • Read: reports what should change in the tests but changes nothing.
  • Upsert: a person approves or dismisses each suggested test addition or update, and approved changes apply on merge.
  • Autonomous: suggestions above a confidence threshold are approved automatically. Lower-confidence ones wait for a person, and retiring a test always needs human approval.

Test execution timing is configured separately. A sensible starting point is Read mode, verification on every pull request, and human merge approval. Move to Upsert once your baseline is stable. Our guide to human in the loop vs human on the loop covers when each level of oversight fits, and autonomous software delivery levels maps the wider path.

Why does independent journey testing catch what code checks miss?

Because a change can be correct in the code and wrong on the screen. The ContextQA docs use a payments example: removing a duplicate amount conversion fixes the stored balance, but the confirmation screen still shows the wrong amount. Unit tests on the conversion pass. Only a journey test that checks the balance, the transaction history and the confirmation screen separately catches it (worked example pull request).

When the agent fixes the screen, that new commit must be tested again. Every retry is a new candidate. Set a retry limit and an escalation path so the loop stops when access, requirements or results stay ambiguous.

What are the most common mistakes?

  • Letting the code define "correct". If expected results are written after the code, the tests confirm whatever the agent built.
  • Testing the wrong build. A green run on yesterday's deployment proves nothing about today's pull request. Record the commit and build for every run.
  • Treating a confidence score as a result. Confidence explains an analysis. A release needs completed runs with meaningful assertions.
  • Skipping the baseline. Without one, every old failure looks like a new regression and people stop trusting the signal.
  • Automating merge first. Widen autonomy after verification is reliable, not before. See AI coding agent quality gates for gate design.

How do you know the factory is working?

Track outcomes, not volume. The ContextQA docs recommend measuring:

  • Time to first useful evidence on a pull request
  • Reproduction quality for reported bugs
  • Verified fixes, and regressions introduced by fixes
  • Impacted tests that were omitted from a run
  • Human interventions per change
  • Total credits and elapsed time per verified change

Don't count generated test cases as coverage. Fifty weak tests prove less than five that check the right outcomes. For the money side, see our ROI of AI-assisted software development framework.

Bottom line

An AI software factory without verification is just a faster way to ship untested code. Start with one journey, one repository and a written expectation. Connect Claude Code to the ContextQA MCP server, test every pull request on its own build, and keep a person on the release decision. Then widen autonomy as the evidence earns it.

Planning a factory across several teams or applications? Book a ContextQA demo and bring one real journey to test.

Sources

Share the Post:

Author

Deep Barot

CEO @ ContextQA | Agentic AI for Software Testing | Context-aware Testing

Deep Barot is the Founder and CEO of ContextQA, the only AI testing platform that understands context. He brings decades of experience across DevOps, full-stack engineering, cloud systems, and large-scale platform development.
AI Insights
Real User Intelligence Platform

Turn live sessions into test coverage. No prompts, no manual design - just pointed at your URL and generating suites within minutes.

Minutes
From URL to generated test cases
Zero
Prompts or manual test design needed
40%+
Average coverage increase after first run
100%
Based on real user behavior, not guesses

Frequently Asked Questions

The MCP server lets Claude Code create, run and read ContextQA tests during a session. Ship adds the always-on part: it analyzes each pull request, picks the affected journeys and runs them without someone prompting an agent. Many teams start with MCP and add Ship when they want every pull request covered.
Yes. The verification stage doesn't depend on which agent wrote the code. Codex and Cursor connect to the same ContextQA MCP server, and the same evidence contract applies: read the expected behavior first, make a focused change, and return the tested commit and run links.
No. Execution evidence shows whether user journeys work. Code review still covers architecture, security, maintainability and contracts. A journey can pass while a code-level problem sits outside its assertions, so keep both.
Not directly. The MCP connection gives Claude Code tools, but ContextQA's runners can't reach your localhost. Deploy a preview build per pull request or use an approved tunnel, and confirm the target environment before every run.
Automate verification before you automate merging. Run the affected journeys on every pull request and keep human approval for merge. Once your baseline is stable and failures are classified reliably, let approved test updates apply automatically and consider wider autonomy for low-risk changes.
Related Blogs