TL;DR: In a software factory, "human in the loop" means a person approves specific actions before they run. "Human on the loop" means agents act on their own while a person watches and can step in. Most teams need both. Merges to production, schema changes, and anything that touches secrets should always wait for explicit approval. Routine test runs and bug reproduction can run under supervision. Ship, ContextQA's autonomous quality engineer, attaches run evidence to each change so reviewers can approve faster.
The choice between the human in the loop vs human on the loop software factory model decides how fast AI coding agents ship and how much risk reaches production. If you approve every action, reviews pile up. If you supervise loosely, a bad merge, schema change, or leaked secret gets through. This guide covers which actions always need approval, how verification evidence speeds up review, and what to change on Monday.
Quick answers
What is the difference between human in the loop and human on the loop in a software factory? In human in the loop, the agent pauses until a named person approves a specific action, such as merging a pull request or applying a schema change. In human on the loop, the agent keeps working while a person watches runs, evidence and alerts, and steps in when something looks wrong. Most teams use both.
Which AI coding agent actions should always require human approval? Merges to production branches, database schema changes, and any action that touches secrets, tokens or credentials should always need explicit approval. So should release decisions, deletions, and permission changes. Operate Ship Safely also keeps people responsible for generated tests, suggested fixes, and the evidence behind a release decision.
How does verification evidence help a reviewer approve agent changes faster? Reviewers approve faster when every run that feeds a decision records the endpoint, build, data revision, browser, region and test identity, and that evidence never changes. The reviewer checks one evidence package and doesn't reproduce the failure by hand. Ship's documented controls also report an unreachable tunnel as blocked, not as a pass or a fail.
What is the difference between human in the loop vs human on the loop software factory?
Human in the loop means an AI agent in a software factory stops and waits for a named person to approve a specific action. Human on the loop means agents act on their own while a person watches and can step in. The first model gives you control over each action. The second gives you speed, and it only works with good monitoring and evidence.
An AI software factory is a pipeline where coding agents turn tickets into code and pull requests, and automated checks decide what merges. There are three ways to place people in it:
- Human in the loop: the agent waits for a named person to approve a specific action.
- Human on the loop: the agent goes ahead, and a person monitors runs and evidence and can stop or roll back the work.
- Human out of the loop: nobody approves or watches live. People review outcomes afterward, if at all.
Most teams pick a model for each type of action, not one model for the whole agent. Ship may start and coordinate quality work, but a person reviews generated tests, suggested code changes and release evidence (Operate Ship Safely).
How do human in the loop and human on the loop approaches compare?
The two models differ in where the gate sits, what they cost in throughput, and how they fail. Human in the loop gates the action itself, so nothing irreversible happens without a sign off. Human on the loop gates the outcome: a reviewer watches the evidence and steps in on exceptions. Most software factories need both, matched to the risk of each action.
- Control point: In the loop gates actions such as a production merge, a schema migration or a secret change. On the loop lets the work run, and a reviewer stops the agent when the evidence looks wrong.
- Throughput: Every in the loop gate adds a wait. On the loop keeps the pipeline moving, but only if you set maximum actions, reruns, mutations and spend before the agent starts (QA agent workflow guide).
- Scope fit: On the loop fits work that is limited in scope and easy to undo, such as pull request regressions and bug reproduction in staging with disposable data.
- Failure mode: In the loop turns into rubber stamping when approvals arrive without evidence. On the loop fails silently when nobody has tested the stop conditions or the alert path.
When should you choose human in the loop?
Choose human in the loop when an action can't be undone, touches production, or changes something other systems depend on. The agent proposes the action and then waits for a person to approve it. This applies to merges to production branches, database schema migrations, secrets and credentials, and release decisions, whatever the agent's track record.
Score each action by how much it can break and how easily you can undo it. How confident the agent sounds shouldn't factor in:
- Irreversible side effects: Without review, an agent can drop a column, rotate a key or message customers, and it can repeat the change if it retries after a crash. The workflow graph guide requires a duplicate guard on every external mutation, such as "create defect."
- Critical code paths: Payments, authentication and data migrations can fail quietly when an agent's tests pass against its own assumptions.
- Generated coverage and fixes: Operate Ship Safely lists what a person reviews: generated tests, bug reproductions, root cause hypotheses, suggested code changes and release evidence.
- Privileged access: The same page says not to grant payment, messaging, deletion or integration access unless the scenario requires it and safeguards are in place.
Rule of thumb: if undoing an action takes more than reverting a commit, a person approves it first.
When should you choose a human on the loop software factory?
Choose supervision when agent actions can be reversed, stay out of production, and leave evidence a reviewer can audit later. Agents keep working without waiting for a sign off, and a person watches the results and steps in on exceptions. Merges to production, schema changes, and secrets still sit behind explicit approval in this model.
- Reversible, sandboxed work: Ship's security controls recommend staging or QA over production, isolated test identities, and disposable data that is reset on a documented schedule.
- Bounded loops: A loop with set limits on actions, elapsed time, reruns and mutations can run without pausing for approval (workflow graph guide).
- Evidence on every run: Each decision bearing run records its endpoint, build, data revision, browser, region and identity. Reviewers check that record in the Ship dashboard.
- Audit obligations: SOC 2, ISO/IEC 42001, and the EU AI Act human oversight article (Article 14) all expect oversight you can demonstrate. Supervision only meets that bar when the record shows who reviewed what, and when.
How do you move between them?
Move from human in the loop to human on the loop one action class at a time. Don't flip a switch for the whole agent. Start with a person approving every Ship action in one repository and track how often reviewers overrule the evidence. Then move proven, low risk actions to supervision, and keep production merges, schema changes and secrets behind approval.
- Scope one evaluation repository. Grant selected repositories, one Jira or Linear project and one controlled Slack channel, and point Ship at staging with disposable data (Operate Ship Safely).
- Approve everything first. A person reviews generated tests, change impact confidence, bug reproductions and suggested fixes. This phase is slow, but it's where you learn what the agent gets wrong.
- Require evidence on decision bearing runs. Record endpoint, build, data revision, browser, region and identity so a reviewer approves from the record instead of rerunning locally.
- Promote action classes. When reviewers stop changing a type of output, such as targeted pull request regressions, move it to supervision with limits on actions, reruns, mutations and spend.
- Keep the hard gates. Merges to production, schema changes and secrets always need human approval.
- Redefine metrics. QA leads own review policy. Track override rate, approval latency and escaped defects, not test count.
If your AI coding agents merge code with no proof attached, see how Ship gathers verification evidence on each pull request so reviewers can approve faster. Get started free, no credit card needed.
Where does ContextQA fit?
Ship, the ContextQA autonomous quality engineer, works on the verification side of a software factory. It tests every pull request with impact analysis and targeted regressions before merge, reproduces reported bugs, and points to a likely root cause. People keep the approvals. Its documented controls send generated tests, suggested fixes and release decisions to a reviewer.
Ship's security and controls guide supports both oversight models:
- Approval for specific actions: A person reviews the repository boundary, the crawler's allowed actions and stopping conditions, generated tests, root cause hypotheses, suggested code changes and the release evidence package.
- Supervision through narrow scope: Supervision is safer when the blast radius is small: selected repositories, staging over production, isolated identities, and no tokens or private keys in prompts or tickets.
- Evidence that speeds review: Reviewers check the recorded run facts instead of rerunning the suite.
Bottom line
AI coding agents don't all need the same oversight model. Supervise bounded, reversible work, and require explicit approval for production merges, schema changes and secrets. Ship keeps the run evidence reviewers need to approve quickly. To go further, see how ContextQA works as an autonomous quality engineer, read about on premise testing for software quality assurance, and browse the ContextQA integrations list.
Sources
This article draws on ContextQA's public Ship documentation for product behavior. It cites the primary compliance and regulatory texts directly for the oversight and audit expectations that apply to AI driven delivery pipelines.
