TL;DR: To verify AI generated pull requests before merge, run three checks on every PR. First, an impact analysis maps the diff to the tests it affects. Second, a targeted regression runs those tests. Third, a reproduction confirms any bug the PR claims to fix. The PR should carry run IDs, logs and the reproduction result as evidence. In the Stack Overflow 2025 Developer Survey, 46% of developers said they distrust the accuracy of AI tools. ContextQA Ship, the autonomous quality engineer, runs these checks on pull requests. A human still reviews any change that is low confidence or high risk.
Coding agents open pull requests faster than reviewers can read them, and a green build with a confident description from Claude Code, Codex or Cursor proves very little. This guide covers a merge gate: what to run, what evidence each PR should carry, and when a human has to review it.
Quick answers
What should a merge gate require before an AI generated pull request can merge? It should require three checks. An impact analysis maps the diff to affected tests. A targeted regression runs those tests on an approved non production build. If the PR claims a bug fix, a reproduction must fail before the change and pass after it. In ContextQA, you configure the analysis in Settings → PR Impact Analysis (docs).
What evidence should an AI generated pull request carry for a reviewer? The PR should carry the source signal (the ticket or bug report), the scope that ran, test results, and logs, screenshots or traces. A bug fix should also carry the likely cause and any regression coverage that was added. Ship treats this bundle plus a reviewer decision as its output, not a bare pass or fail (Run Ship Workflows).
When must a human review instead of trusting automated checks? A human must review when recommendations fall below your confidence threshold, or when the report flags fixed URLs or missing environment variables. The same applies when the change touches authentication, payments, data deletion or production configuration. Code analysis never approves or merges a pull request (Run Ship Workflows).
What is it, and when do you need to verify AI generated pull requests before merge?
To verify an AI generated pull request, you run a bounded check on each change a coding agent proposes and attach evidence that the change does what it says. You need this whenever code from Claude Code, Codex or Cursor can reach your default branch before a human has traced how it behaves.
Agent PRs often look finished, but a reviewer skimming the diff can miss a hallucinated API call or a weakened assertion. The question becomes "what ran, and what did it prove."
A practical gate has four layers:
- Impact analysis: Map the diff to affected, reusable and missing tests, so regression scope comes from code change evidence instead of test names (PR Impact Analysis).
- Targeted regression: Run those tests against the PR build before merge.
- Bug reproduction: If the PR claims a fix, rerun the original reproduction and confirm it now passes.
- Assertion audit: Confirm no test was deleted, skipped or loosened to make the build green.
What do you need before you start?
You need a narrowly scoped repository connection, a staging environment running the PR build, a disposable test account with known seed data, and the bug ticket the agent says it fixed. Without these, a green check proves very little.
- Repository access: Authorize GitHub or GitLab with the narrowest suitable scope, then confirm the repository, default branch and deployed application (Ship workflows).
- A target that matches the build: ContextQA PR Impact Analysis flags fixed URLs or missing environment variables that would send tests to the wrong build. The feature is in limited rollout, so confirm it is enabled for your workspace.
- Test identity and data: Use an isolated account, seeded records and a cleanup step for each run (end to end guide).
- The source ticket: Have the Jira, Linear or Slack report the PR claims to fix, so Ship can reproduce that bug.
How do you verify AI generated pull requests before merge, step by step?
Put each AI generated pull request through five gates: a dependency scan, impact mapping, a targeted regression, a replay of any claimed bug fix, and branch protection that blocks merge until every check passes and a human approves.
- Scan before testing. Run dependency review and `npm audit`. If the agent added a vulnerable package, the PR fails before any browser test runs.
name: ai-pr-gate
on: pull_request
jobs:
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/dependency-review-action@v4
with:
fail-on-severity: high
- run: npm ci && npm audit --audit-level=high
- Map the change. Ship's code analysis links the repository to flows. In ContextQA PR Impact Analysis (Settings → PR Impact Analysis), the Read tier produces a report and changes no tests.
- Run the affected tests, along with any missing coverage the report flags.
- Replay the reproduction of the reported bug. It should fail on
mainand pass on the branch. Review the run evidence as described in Ship dashboard and evidence. - Gate the merge. Make every job a required status check, and require human approval for changes to auth, payments and migrations.
What are the most common mistakes?
The most common mistakes are trusting the agent's summary, guessing the regression scope, accepting a fix with no reproduction, and treating green CI as approval. Each lets a PR merge with evidence that something ran, not that the change behaves correctly.
- Trusting the PR description: The agent's summary is only a claim. Ask for run identifiers, logs, screenshots or traces before you review.
- Guessing regression scope: Impact analysis maps a change to affected tests with a confidence level and code change evidence, and holds recommendations below your threshold (docs).
- Skipping the reproduction: If the PR says it fixes a bug, run the reproduction against the change and attach the result.
- Accepting a toast as proof: A success message does not prove data was saved. Check the stored state (end to end guide).
- Ignoring debt and licensing: Duplicated helpers, weakened assertions and incompatibly licensed code need a human and a license scanner. Impact analysis does not catch them.
How do you know it is working?
The gate is working when agent pull requests fail in production no more often than human ones, and every merged agent PR carries an impact report, a targeted run and, for fixes, a reproduction. Measure AI authored and human authored PRs separately so a quality drop cannot hide in a combined number.
The DORA 2024 Accelerate State of DevOps report estimated that a 25% increase in AI adoption was associated with a 7.2% decrease in delivery stability. In the Stack Overflow 2024 Developer Survey, only 43% of developers said they trust the accuracy of AI tools.
Track four signals:
- Change failure rate by author: Label agent PRs
ai-authored, then compare failed deploys and reverts for labeled and unlabeled PRs. - Escaped defects: Count bugs traced to a merged agent PR, and confirm each with Ship's bug reproduction workflow.
- Evidence coverage: Track the share of agent PRs that merged with an impact report and a passing targeted run.
- Held recommendations: Count impacted tests below the confidence threshold that a person resolved before merge.
How can you automate how you verify AI generated pull requests before merge with ContextQA?
ContextQA Ship turns each AI written pull request into a bounded quality workflow: impact analysis, a targeted regression before merge, and a reproduction of any bug the PR claims to fix. The PR then carries the source signal, scope, evidence, likely cause and a reviewer decision.
To set up the gate, follow the Ship workflows guide. For change based test selection, the ContextQA PR Impact Analysis docs describe the settings:
- Authorize GitHub or GitLab with the narrowest repository scope, then run code analysis so Ship has repository context.
- In ContextQA, open Settings → PR Impact Analysis. Choose report only, run after merge or run before merge, then pick the execution environment and deployment timing.
- Start on the Read autonomy tier, which produces an advisory report and makes no test changes.
- Set a Confidence threshold. Recommendations below it are held for a human to review.
Ship complements AI code review by checking how the change behaves when it executes. A human still signs off on every merge.
If your coding agent opens pull requests faster than you can review them, Ship runs impact analysis, targeted regressions and bug reproductions before merge. Get started free, no credit card needed. Enterprise teams on the ContextQA platform can book a demo.
What should you do right now?
Before you merge another pull request from Claude Code, Codex or Cursor, require an impact analysis that names the affected tests, a targeted regression on the intended build and, for bug fixes, a reproduction. AI review of the diff helps, but it never executes the changed behavior.
- Connect GitHub or GitLab to Ship with the narrowest repository scope that works.
- Start PR Impact Analysis in the Read tier so reports stay advisory.
- Route held recommendations and PRs touching auth, payments or data migrations to a human reviewer.
- For bug fix PRs, require the reproduction to fail on the base branch and pass on the PR branch.
- Attach the source signal, scope, run results, likely cause and reviewer decision to the PR.
- Add a test plan to CI/CD only after it passes a controlled manual run.
Bottom line
To verify an AI generated pull request before merge, ask for evidence, not a green check. Every agent PR needs an impact analysis, a scoped regression run and, for bug fixes, a passing reproduction. Low confidence or high risk changes need a human. See the AI generated code testing checklist and how Ship, the AI QA agent, tests pull requests.
Sources
Product steps come from ContextQA Ship and PR Impact Analysis documentation, CI configuration from GitHub and npm documentation, and statistics from the DORA 2024 report and the Stack Overflow 2024 and 2025 developer surveys.
- Run Ship Workflows, ContextQA docs
- PR Impact Analysis, ContextQA docs
- Ship dashboard and evidence, ContextQA docs
- GitHub dependency review action
- npm audit documentation
- GitHub protected branches and required status checks
- DORA 2024 Accelerate State of DevOps report
- Stack Overflow 2024 Developer Survey, AI
- Stack Overflow 2025 Developer Survey, AI
