TL;DR: An open, model-agnostic AI software factory lets you swap the coding agent or the model behind it without rebuilding your pipeline. The trick is to standardize the parts that shouldn't change (repository rules, the MCP connections, the CI gate and the verification evidence) and treat the model as a replaceable part. Open-source agents such as Aider, Cline and OpenHands, plus Claude Code through your own cloud account, all fit this pattern. ContextQA stays the same verification layer whichever model wrote the code, and it runs as SaaS or on-prem.
Teams in regulated industries, and teams that simply don't want lock-in, ask the same question: can we build an AI software factory on our own terms? The answer is yes, as long as you're clear about which parts are swappable and which parts need to stay fixed. This guide explains the architecture, the trade-offs and how to keep quality steady when the model changes.
Quick answers
What does "bring your own model" mean for a software factory? You choose where the model runs and who it's billed through: a vendor API, your own cloud account, a company gateway or a self-hosted open-weight model. The rest of the factory, including rules, tools, gates and evidence, doesn't depend on that choice.
Do I need open-source models to avoid lock-in? No. Most lock-in comes from pipelines built around one agent's quirks, not from the model. If your rules live in the repository, your tools connect over an open protocol like MCP, and your gate checks real execution results, you can change agents later with little rework.
Where does ContextQA fit? It's the independent verification stage. Any MCP-compatible agent can connect to the ContextQA MCP server, and ContextQA test plans run as CI gates regardless of which agent opened the pull request. For strict data needs, ContextQA is available on-prem as well as SaaS.
Why build a model-agnostic factory?
Four reasons come up again and again:
- Data control. Some code and test data can't leave your network or your cloud account.
- Cost control. Model prices and plans change. A swappable model lets you route simple tasks to cheaper models.
- Procurement and compliance. Many enterprises can only use models available through an approved cloud provider.
- Pace of change. The best model for coding changes every few months. A factory tied to one model ages fast.
Trust matters more than any of these. 46% of developers say they actively distrust the accuracy of AI tools, and only 3% say they highly trust them (Stack Overflow Developer Survey, 2025). Changing models without a fixed verification layer makes that worse, because every swap resets what you know about output quality.
Which parts should be swappable, and which should stay fixed?
| Layer | Swappable? | Keep it portable by |
|---|---|---|
| Model | Yes | Calling it through a gateway or your cloud provider, not hard-coding one API |
| Coding agent | Yes | Keeping instructions in AGENTS.md or CLAUDE.md in the repository |
| Tools the agent uses | Mostly | Connecting through MCP, an open protocol many agents support |
| Repository rules and definition of done | No | One shared rules file, reviewed like code |
| CI gate | No | Running the same required checks for every pull request |
| Verification and evidence | No | An independent test layer with stored run results |
| Release decision | No | A named human owner |
The bottom four rows are your factory. The top three are interchangeable workers.
What are the options for the coding agent and model?
You have three broad routes. Mix them if different teams have different constraints.
1. A commercial agent through your own cloud account. Claude Code can run through Amazon Bedrock or Google Vertex AI, so traffic and billing go through your cloud contract (Claude Code on Bedrock, Claude Code on Vertex AI). It can also route through a company LLM gateway (Claude Code LLM gateways). This is often the easiest path for enterprises that want a top model with their own controls.
2. An open-source agent with the model of your choice. Aider says it can connect to almost any LLM, including local models (Aider). Cline and OpenHands are open-source coding agents you can run and inspect yourself. The Codex CLI is also open source (Codex on GitHub).
3. Self-hosted open-weight models. Tools such as Ollama for local use and vLLM for serving at scale let you run models on your own hardware. Expect more setup and tuning, and test output quality on your own codebase before you rely on it.

How do you keep the rules portable?
Write them once, in the repository. AGENTS.md is an open format many coding agents read (agents.md), and Claude Code can read AGENTS.md as well as CLAUDE.md (Claude Code memory docs). Your rules file should hold:
- Install, build, lint and test commands.
- Folders and files agents must not change without a human request.
- The definition of done, including which tests must pass and what evidence to report.
- Things an agent never does, such as merging, disabling tests or weakening assertions.
When you swap the agent, the rules come with the repository. That's most of the migration done.
How do you keep tools portable?
Use MCP. The Model Context Protocol is an open standard for connecting agents to tools and data. If your test platform, issue tracker and docs are reachable over MCP, a new agent gets the same tools on day one.
ContextQA's MCP server works this way. In ContextQA, Settings → MCP shows setup for ChatGPT, Claude, Claude Code, Cursor and Codex, plus a generic method for other MCP clients (ContextQA integrations and MCP docs). Tests created by one agent are visible to the next, so changing agents doesn't mean losing your test library. For a wider look at the protocol, see MCP for test automation.
Want a verification layer that works with any coding agent? Ship by ContextQA analyzes each pull request and tests the journeys it affects: start free, no credit card needed.
How do you keep quality steady when the model changes?
Make verification independent of the model. A model that writes the code and the tests can make the same wrong assumption in both. ContextQA's docs use an export permission example: the implementation and its generated tests both forget that a report must belong to the user's tenant, so every test passes and the bug ships (ContextQA: testing AI-generated code).
The fix is a verification stage that doesn't change when the model does:
- Expected results written from the requirement, before code exists.
- A baseline run on the current main build, so you can tell new regressions from old failures.
- The same test plan as a required CI check on every pull request (ContextQA CI/CD release gates).
- Stored evidence per run: the commit, the build, step results, screenshots and traces.
Then run a simple swap test. Before changing a model for real work, give the old and new setups the same five tickets and compare verified pass rates, review findings and human interventions. Let the numbers decide.
What about data that can't leave your network?
Check each layer separately, because "self-hosted model" doesn't make the whole factory private:
- Model traffic: your cloud account, gateway or self-hosted model.
- Agent runtime: where the agent runs and what it can read.
- Test execution: where browsers run and what they can reach. ContextQA supports tunnels for private applications (ContextQA tunnels docs), and an on-prem deployment for teams that need the test platform inside their own environment.
- Evidence storage: where screenshots, videos and logs live, and who can see them.
For permissions and audit trails across agents, see our guide to enterprise AI coding governance.
What are the trade-offs?
- More setup. Self-hosted models need hardware, serving and updates. Budget engineering time for it.
- Uneven quality. Smaller open-weight models may need more retries and tighter instructions on complex changes.
- More to secure. Every gateway, runner and tunnel is something to patch and monitor.
- Harder comparisons. If teams use different models, you need shared metrics to compare them fairly.
None of these are reasons to avoid an open factory. They're reasons to keep verification fixed, so you can measure what each choice really costs.
Bottom line
An AI software factory doesn't have to belong to one model vendor. Keep the rules in the repository, connect tools over MCP, gate every pull request with the same independent tests, and leave the release decision to a person. Then the model becomes a part you can swap when a better or cheaper one arrives. Start with the AI software factory build guide, and read what an AI software factory is if you're new to the idea.
Need the verification layer inside your own environment? Book a ContextQA demo to discuss SaaS and on-prem options.