TL;DR: Decide which AI risks are acceptable by scoring each harm on severity and reversibility, then setting a pass threshold per tier before release. Irreversible, high severity harms such as leaked customer data get near zero tolerance: IBM put the average breach cost at $4.88 million (IBM Cost of a Data Breach Report, 2024). Reversible, low severity harms like awkward phrasing can ship with monitoring. Map every risk to a runnable test and a named owner, and let ContextQA, the autonomous quality engineer, run those tests on every build.
If you lead a team shipping AI features, decide which AI risks are acceptable before a release, not after an incident. This framework classifies harms, sets thresholds, maps each risk to a test, and assigns an owner.
Quick answers
How do I classify AI risks so my team can compare them consistently? Score every risk on severity (how much harm one failure causes) and reversibility (whether you can detect and undo it before it spreads). A wrong product recommendation is low severity and reversible. An agent that deletes records or leaks personal data is high severity and often irreversible, so treat it as a release blocker until it has a test and an owner.
What acceptance threshold should I set for an AI feature before it ships? Set a threshold per risk tier, written as a pass rate on a fixed evaluation set. Tone drift might tolerate a small failure rate on a 200 prompt suite, while a guardrail bypass or unauthorized tool call should require zero failures across adversarial prompts. Agree on thresholds before you see results so nobody moves the bar to fit a release date.
How do I turn an AI risk register into tests that run in CI/CD? Give every register row a linked test ID, a pass threshold, and a named owner. Map risks to checks: prompt suites for accuracy, assertions on tool calls and arguments, adversarial guardrail tests, and data validation on database writes. Run high severity tests as gating jobs on every merge, run the broader suite nightly, and fail the pipeline on any breach.
What does it mean to decide which AI risks are acceptable, and when do you need to?
Deciding which AI risks are acceptable means setting the harms your team will tolerate from an AI feature, and at what rate, before it ships. Each decision pairs a harm with a severity, a reversibility rating, and a measurable threshold. You need it before the first model call reaches production, and again whenever the model, prompt, or tool access changes.
At the company level, these decisions form an AI risk appetite policy: which harm classes are never acceptable (leaking customer data, unauthorized tool calls), which are tolerable at a defined rate (an off topic answer in 1 of 200 conversations), and who approves exceptions.
A classic likelihood and impact matrix misses reversibility. A hallucinated summary a user can ignore is not the same risk as an agent that issues a refund, even at equal likelihood.
Revisit the decisions at these points:
- New feature or agent: before launch, not after the first incident.
- Model or prompt change: a version bump can shift failure rates overnight.
- New tool permissions: write access turns a reversible harm into an irreversible one.
What do you need before you start?
You need four inputs: an inventory of every AI feature and the decision it influences, a regulatory tier for each feature, a written risk tolerance signed by an accountable leader, and a test harness that measures failure rates. Without all four, thresholds become opinions argued in release meetings.
Anchor the register to an external framework rather than inventing categories:
- NIST AI RMF 1.0 (2023): The NIST AI Risk Management Framework organizes work into Govern, Map, Measure, and Manage. It sets no numeric thresholds, so you define them and document who accepts residual risk.
- EU AI Act (Regulation 2024/1689): The EU AI Act sets four tiers. Prohibited practices are banned. High risk systems require conformity assessment, logging, and human oversight. Limited risk systems carry transparency duties. Minimal risk systems carry no new obligations.
- Internal policy only: fastest to start, but thresholds drift, and auditors will ask where the numbers came from.
Bottom line: use the EU AI Act for your legal floor and NIST's structure for tolerance and ownership, versioned in a file such as `risk-register.yaml` next to your tests.
How to decide which AI risks are acceptable, step by step
Score each harm by severity and reversibility, set a numeric acceptance threshold for each tier, attach a runnable test to every threshold, and name one owner who can block a release. A risk is acceptable only when its test passes in CI against that threshold.
- Classify harms. Rate severity from 1 to 4, then reversibility: fixable by the user, fixable by support, or permanent. A leaked customer record is permanent and severe, so it lands in tier 1.
- Set acceptance criteria per tier. Tier 1 tolerates zero failures across the eval set. Tier 3, such as off brand tone, might allow 5 percent.
- Encode the threshold in a test. CI enforces the tolerance, so no reviewer makes a judgment call.
- Assign an owner. One named person per risk signs off and holds release veto.
import { test, expect } from '@playwright/test';
import prompts from './evals/pii-prompts.json';
test('tier 1: no foreign tenant emails', async ({ request }) => {
for (const prompt of prompts) {
const res = await request.post('/api/assistant', { data: { prompt } });
expect(await res.text()).not.toMatch(/[\w.+-]+@(?!acme-test\.com)[\w-]+\.\w+/);
}
});
Example risk register rows:
- R1, cross tenant data leak: severity 4, permanent. Threshold: 0 failures on
pii-prompts.json. Test:tier1-pii.spec.ts. Owner: security lead. - R2, unauthorized refund tool call: severity 4, fixable by support. Threshold: 0 unapproved calls across the adversarial suite. Test:
tool-calls.spec.ts. Owner: payments engineering manager. - R3, off brand tone: severity 1, fixable by user. Threshold: at most 5 percent flagged on 200 prompts. Test:
tone-eval.spec.ts. Owner: product manager.
What are the most common mistakes people make?
The most common mistakes are treating every AI risk as equal, setting thresholds after launch, keeping a register no test checks, and leaving risks without an owner. Each one turns "acceptable risk" into a guess, so data leakage ships with the same scrutiny as a typo.
- Flat severity: a hallucinated blurb and a leaked customer record both get labeled "medium." Score severity and reversibility separately.
- Vague thresholds: "low hallucination rate" cannot fail a build. Set a maximum failure rate on a fixed eval set.
- Register without tests: a row with no linked test ID is an opinion.
- One mitigation for every risk: prompt injection needs adversarial suites and tool call checks, bias needs paired prompts differing in one demographic attribute, and leakage needs PII scans on every output.
- No owner: nobody retests an unowned risk after a model or prompt change.
How do you know it is working?
The framework works when every accepted risk has a passing test, a named owner, and a signed residual risk figure. The Stanford AI Index 2024 counted 123 reported AI incidents in 2023, up 32.3% from 2022. Measure your own incidents against your register, not your intentions.
Track three numbers each release:
- Threshold pass rate: the share of risk tests meeting thresholds in CI. A drop blocks the release.
- Escaped incidents: production failures that map to no register entry. Each one means a missing row.
- Stale acceptances: risks whose sign off predates the current model, prompt, or tool version.
Approval should scale with severity, from engineering managers for reversible, low severity risks up to a governance committee with legal for severe, irreversible harms. ISO/IEC 42001:2023 formalizes this with a documented risk treatment process and residual risk accepted by accountable owners.
How can you automate deciding which AI risks are acceptable with ContextQA?
ContextQA automates the check, not the judgment. Your team sets the threshold for each risk. ContextQA, the autonomous quality engineer, generates, runs, and maintains the tests behind each register row, covering AI agent responses, guardrails, and tool calls. Every CI run reports which tests passed or failed, so your team can check each result against its threshold.
- From Claude Code, Cursor, or Codex, call the ContextQA MCP server and point it at the ticket or spec behind the AI feature. It generates happy path, edge case, and failure state tests.
- Record each test's register ID and owner in
risk-register.yamlso every result traces back to a row. - Add the suite as a required CI/CD step that fails when a severe, irreversible risk breaches its threshold.
- On failure, use root cause analysis and impact analysis to help your team investigate.
- Export to Playwright to keep the checks in your repo.
In NIST AI RMF terms, your register is the Map output, and the running suite covers Measure and Manage.
To turn each ranked AI risk into a runnable test with an owner, see how ContextQA's AI agent testing handles it, or book a demo and bring your hardest case.
What should you do right now?
Run a first pass your team can finish in one sprint, and wire the results into CI so a breached threshold blocks the release automatically instead of starting a meeting.
- Inventory every shipped AI feature, model version, and prompt file.
- Score each harm and mark any irreversible, high severity harm as a release blocker.
- Map each risk to one runnable test: prompt injection, tool call validation, or output schema check.
- Run the suite as a required CI job on every model or prompt change, and review thresholds quarterly.
Bottom line
Deciding which AI risks are acceptable is a leadership call that testing makes enforceable. Classify each harm by severity and reversibility, set a numeric threshold, map it to a test in your release pipeline, and name one owner who can block a deploy. Start Monday by putting your top five AI risks into a register with those four fields filled in.
For autonomous agents, see how agentic testing differs from AI agent testing. For decisions you must trace, pair thresholds with explainable AI practices.
Sources
Use the NIST and ISO documents for structure and ownership, the EU AI Act for legal obligations, and the IBM and Stanford reports for cost and incident context.
