Engineering teams are writing outcome contracts for AI agents because a voice agent can sound right on every call while the booking, refund or record update never happens. Scoring the transcript alone misses that gap. This guide turns six clauses (intent, permitted actions, required state, deadline, escalation and evidence) into a release gate you can run on every agent revision.
TL;DR: An outcome contract for an AI agent defines six clauses for each workflow: intent, permitted actions, required state, deadline, escalation and evidence. Score the spoken promise, the tool response and the system of record separately. Block releases only on critical violations. Treat a call as resolved only when the authoritative record confirms it.
Quick answers
What should an outcome contract for an AI agent actually contain? Each workflow needs six terms: the caller's intent, the permitted actions, the record state required afterward, the deadline for that state, the escalation path, and the evidence for each term. Write the terms against the system of record. "An appointment row exists for the corrected vehicle ID before the call ends" is a contract term. "The agent said it's booked" is not.
How do I turn outcome contracts into a release gate without blocking every deploy? Split the terms into critical rules and quality measures. Critical rules, such as an unauthorized promise, a missing handoff or a wrong record, block the release. Quality measures are tracked as trends. Run a fixed set of synthetic scenarios against each agent revision and log the rubric revision too, so a failure points to a change and not to drift in the judge.
Why score the spoken promise, the tool response, and the backend record separately? Each layer can fail on its own. The agent can say "you're booked" while the tool returns an error, or the tool can succeed while the record holds the wrong date. Separate scores show whether to fix the prompt, the integration or the data. In Ship Voice, the Policy and Grounding models check what the agent says, and the Outcome model checks the record.
What is an outcome contract for an AI agent?
An outcome contract for an AI agent is a written specification of what one interaction must achieve. It covers the customer's intent, the permitted actions, the record state required afterward, the deadline, the escalation path and the evidence that proves it. It scores the result of the interaction, not the quality of the reply.
A traditional SLA (service level agreement) sets targets across many interactions, such as uptime, latency or ticket resolution time. An agent can meet all of them and still fail the customer, for example by saying "your appointment is booked" after the scheduling API returned an error. An outcome contract splits each verdict into three parts: the spoken promise, the tool response and the authoritative record.
The split also speeds up release triage. After a prompt or model change, a conversation regression, a broken tool call and a write that never landed each look different and belong to different owners. For background, see agentic testing vs AI agent testing and business outcome testing for AI agents.
What problems does an outcome contract solve?
An outcome contract turns a vague goal like "handle the reschedule call" into terms a test can check. Each clause becomes a pass or fail assertion, so reviewers stop making judgment calls on transcripts. The failures it exposes usually hide behind fluent speech.
- Promise versus result: "Your appointment is moved" fails if the booking record still shows the old slot.
- Actions outside policy: A list of permitted actions exposes any refund or discount the agent had no authority to offer.
- Tentative agreements: "Maybe Friday" is logged as tentative, not booked. The Ship Voice Conversational State model does this on every call.
- Escalation you can test: "I'll transfer you" counts only when a handoff record exists.

Why do outcome contracts for AI agents matter in 2026?
Outcome contracts for AI agents matter in 2026 because agents now change records, and a fluent transcript says nothing about the record. Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing unclear business value and inadequate risk controls.
Pricing adds pressure. When a vendor contract bills per resolved conversation or completed task, someone has to define "resolved" and "completed." If the definition is a label on a transcript, you are paying for promises. A verified record change is a billing unit both sides can audit.
Liability follows the same logic. Disputes over indemnification (who pays for a loss) and warranties start with hallucinations and wrong actions. A per call record of what was said, what the tool did and what the system of record shows gives legal and engineering teams the same facts.
How does outcome testing work, step by step?
Outcome testing defines what a finished interaction must leave behind, then checks it in layers. You write six clauses for each workflow, build synthetic scenarios that stress them, and score what the agent said, what the tool returned and what the system of record shows. Any critical violation fails the run, no matter how good the transcript reads.
- Name the intent. Write one sentence per contract, such as "reschedule an existing appointment," and list the inputs that identify the record.
- List permitted actions. Name the allowed tool calls (
lookup_order,update_appointment), the forbidden ones (cancel_appointment), and promises the agent must not make, such as a refund outside policy. - Define required state. Name the exact fields the authoritative record must show, for example
status = confirmedwith the corrected date. - Set the deadline and escalation. State when the record must reach that state (for example, before the call ends) and which verified handoff happens if it does not. Saying "I will transfer you" does not pass on its own.
- Attach evidence. Store the recording, transcript, tool response and record ID with every result.
- Rerun as regression. After every prompt or model change, rerun scenarios that include corrections, interruptions and requests outside policy.
Which approach fits your team best?
Choose based on what you need to prove. Transcript scoring works for prototypes. In house verifier scripts work for a single agent with a stable schema. A per call outcome layer fits once an agent books, updates or looks up records, because from that point a spoken promise no longer proves the task got done.
| Approach | Best for | Cost model | Notable tradeoff |
|---|---|---|---|
| LLM judge on transcripts | Prompt tuning, prototypes | Model tokens per scored call | Scores what was said, not what happened |
| In house verifier scripts | One agent, stable schema | Engineering time to build and maintain | Each new tool or record type needs new code |
| Ship Voice | Phone agents that write to a system of record | Starts with a demo or free signup | One more system to adopt |
Decision rule: define "resolved" as a verified record change, then pick the cheapest approach that can prove it.
How does Ship Voice verify the conversation, the action and the outcome?
Ship Voice is ContextQA's builder for AI voice agents that answer and make phone calls, and it checks the conversation, the action and the business outcome on every call. Four intelligence models run alongside the agent. The Outcome model confirms in the system of record, such as the booking system, that the problem was actually solved before a call is marked resolved.
The workflow on the product page lines up with the contract clauses:
- Intent: describe the agent in plain language. No code is needed.
- Permitted actions: connect knowledge and tool actions, such as looking up an order, checking a calendar or updating a record.
- Required state: the Outcome model checks the record before resolution.
- Evidence: every call keeps its recording, transcript and outcome.
The Policy model stops a promise that policy does not allow. The Grounding model catches a claim the sources do not support. The Conversational State model tracks what was agreed, rejected, corrected or left unresolved. The Ship Voice product page says these models can act within the same conversational turn. Write deadline and escalation clauses as assertions in your scenarios, checked against the record and the handoff log.
If your voice agent says "you're booked" but the booking system shows nothing, book a demo or start with Ship Voice to see the conversation, the action and the recorded outcome verified on every call.

What should you do on Monday? A checklist
Start with one high risk call flow and write its outcome contract before you choose tooling. Score the spoken promise, the tool response and the system of record separately, and gate releases only on critical clauses. Then monitor the contract in production, with a stated sampling rate and a named alert owner, before you add a second flow.
- Pick one flow, such as appointment rescheduling, and name its owner.
- Write the intent and every permitted action.
- Define the required record state, for example
status=confirmed, and how long it may take to appear. - List escalation triggers and what counts as a verified handoff.
- Build synthetic, anonymized scenarios: a happy path, a correction in the middle of the call and a request outside policy.
- Track verified task completion and critical failure rate, not average scores (production analytics guide).
- Report the sampling rate, missing data and measurement delay every time.
- Assign an alert destination and owner, then test what happens when ingestion fails.
Bottom line
An outcome contract makes "done" something you can test. For each workflow, write down the intent, permitted actions, required state, deadline, escalation and evidence. Then score the spoken promise, the tool response and the CRM, database or EHR record separately. A call that sounds resolved has not passed until the system of record agrees. On Monday, pick one high risk workflow, write its contract and gate your next release on it.
Sources
These sources support the statistic and the product capabilities described in this article. Ship Voice details come from its public product page. Measurement definitions, including verified task completion and critical failure rate, come from ContextQA's documentation on production analytics for AI agents, read in November 2026.
- Gartner (2025): Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027
- ContextQA: Ship Voice product page
- ContextQA Docs: AI Agent Production Analytics and Real-Time Monitoring