How to Evaluate an AI Agent When the Judges Disagree

Two robot judges holding a green check and a red cross while a human reviewer inspects an evidence folder under an uneven balance scale

TL;DR: When two automated judges disagree about an AI agent's run, the disagreement is information, not noise. Strong LLM judges agree with human preferences over 80% of the time in published research, which means roughly one verdict in five still needs a person. Give each judge one narrow question, check actions against backend evidence instead of the agent's own words, route disputed cases to the owner of the requirement, and record every case as Pass, Fail or Unknown before you decide on a release.

An AI agent evaluation should establish whether the agent completed the intended task within its authority. A fluent answer or a high model-generated score does not establish that the underlying action was correct.

Use automated judges for defined criteria, verify actions with execution evidence, and route disputed or incomplete cases to human review. Record what passed, failed or remains unknown.

Quick answers

Why do AI judges disagree on the same agent run? Usually because they were asked different or vague questions, saw different evidence, or carry known biases such as preferring longer answers or the first answer they read. Disagreement also happens when the requirement itself is ambiguous.

Should the majority verdict win when judges disagree? No. A vote hides the reason for the split. Treat the case as disputed, look at the evidence, and let the owner of the requirement decide.

When does a human need to review an agent evaluation? When judges disagree, when evidence is missing, when the case touches permissions or money, and whenever the requirement itself may need to change. Humans also own the final release decision.

What should an AI agent evaluation actually prove?

It should prove a business outcome, not a pleasant answer. Start from the decision the agent is allowed to make and the state it is allowed to change.

Consider this illustrative scenario: an internal service agent can answer a user's question about access and submit an access request, but only a separate approver can grant access.

The agent replies that access has been granted. A language-quality judge finds the answer clear and helpful. The tool record shows that only a request was submitted. Another judge rejects the answer because it overstates the completed action.

Both judges did their job. They answered different questions. The agent must distinguish a submitted request from an approved change, so put that criterion in the evaluation contract before deciding whether to release. For the wider framework, see our LLM evaluation framework for testing AI agents.

Why do automated judges disagree?

Model-based judges are useful and measurably imperfect. In the MT-Bench study, strong LLM judges such as GPT-4 reached over 80% agreement with human preferences, the same level of agreement humans reach with each other (Zheng et al., 2023). The same paper names position, verbosity and self-enhancement biases, plus limited reasoning ability.

Order alone can flip a verdict. Researchers showed that the quality ranking of candidate responses could be "hacked" by changing their order in the prompt: with ChatGPT as the evaluator, Vicuna-13B beat ChatGPT on 66 of 80 tested queries (Wang et al., 2023). The authors recommend balancing positions and sending hard examples to human review.

In agent testing, the most common causes of a split are practical:

  • Different questions. One judge scores tone, the other scores correctness of the claimed action.
  • Different evidence. One judge saw the tool log, the other only saw the final message.
  • Vague rubric. "Is this a good response?" invites two reasonable answers.
  • Known bias. Longer, more confident answers tend to score higher than they deserve.
  • A real defect. The agent did something wrong and only one judge was positioned to see it.

How do you define criteria that can be judged separately?

Break the workflow into observable requirements, one per criterion:

  • The agent identifies the correct user and requested resource.
  • It obtains missing information instead of inventing it.
  • It calls only tools allowed for that user's role.
  • Its action respects the approval boundary.
  • Its response accurately describes the resulting state.
  • It escalates when authority or evidence is insufficient.

Assign each criterion an evidence source: backend records for request state, permission checks for authorization, and reviewer assessment for an accurate explanation.

Keep critical requirements separate from stylistic preferences. A concise explanation cannot compensate for an unauthorized change. Weighting helps rank what matters, but decide in advance which failures block release regardless of the aggregate score.

How do you build a reference set that catches hard cases?

Include normal requests, missing information, ambiguous resource names, unavailable tools and requests that exceed the user's authority. Use synthetic or appropriately sanitized fixtures with expected behavior reviewed by a domain owner.

Add examples in which the correct behavior is to ask a question, decline an action or hand off to a person. Otherwise, the evaluation may reward completion even when completion should not occur. Our guide on how to test AI agent responses, guardrails and tool calls covers these refusal and escalation cases in more depth.

Keep a reserved set for checking changes after prompts, tools or evaluation rubrics have been adjusted. Avoid tuning every part of the system against the same visible examples. OpenAI's guidance recommends task-specific evaluations and calibrating automated scoring against human feedback (Evaluation best practices).

How should you scope an automated judge?

Give each judge a bounded job. A judge should answer a specific question against an explicit criterion. For this scenario, "Does the response distinguish requested access from granted access?" is more useful than "Is this a good response?"

Provide the judge with the relevant expected behavior and execution evidence. Request a verdict, the criterion it applied and the evidence supporting the verdict. Treat the agent's output as material to evaluate, not as instructions that can redefine the rubric.

Version the grader prompt, model and reference set. During calibration, check whether equivalent responses receive different verdicts because of answer order or unnecessary verbosity, which are documented risks of model-based grading (OpenAI guidance on model graders). Swapping the order of candidate answers and re-running is a cheap check.

Do not ask a language judge to infer a backend action when an authoritative record is available. Evaluate the action and its explanation separately. A deterministic check against a backend record is stronger evidence than any model's opinion about what probably happened.

How do you turn a disagreement into a decision?

Retain both verdicts and route the case to the owner of the disputed requirement. Never average two conflicting verdicts into a middle score.

Two reviewers at a desktop monitor checking evidence from a disputed AI agent evaluation run

The reviewer should inspect the input, tool calls, resulting state, expected behavior and applicable rubric version. Then classify the disagreement before changing anything:

Cause of disagreementWhat the reviewer checksWhat to change
Ambiguous requirementWhether two readings of the criterion are both reasonableRewrite the requirement, then rerun affected cases
Missing or wrong evidenceWhether the judge saw the tool log and backend record for this runFix evidence capture, keep the case Unknown until rerun
Grader applied the criterion wronglyGrader reasoning against the rubric textRecalibrate or narrow the grader prompt and version it
Agent violated the requirementBackend state versus the agent's claimRecord a Fail and fix the agent, not the grader
Requirement needs revisionWhether the business rule itself has changedGet owner approval, version the rubric, rerun
Classify the cause first. The fix depends on the cause, not on which judge was "right".

Record the final disposition and reason. If the rubric changes, rerun the affected evaluation cases against that version. Do not silently relabel failures to make a dashboard improve.

Why should the release record use Pass, Fail and Unknown?

Because "not failed" is not the same as "passed". Three states keep missing evidence visible:

  • Pass: evidence establishes that the criterion was satisfied for the tested agent configuration.
  • Fail: observed behavior violates the criterion, such as claiming approval when only a request exists.
  • Unknown: missing evidence, an unresolved requirement or a failed evaluation process prevents a defensible verdict.

Track false acceptance separately: cases a grader passes that human review determines violate a required criterion. This is the most dangerous number in agent evaluation, because those cases look green on a dashboard. Also record disagreement and unresolved-case counts. Report the evaluated population and its limits rather than treating an aggregate score as universal readiness.

Before expansion, require the critical permission and state-reporting cases to have explicit dispositions. Product and engineering owners should agree how unknown cases affect the proposed rollout and which failures block it. This works best when the rules are written down before the run, as described in outcome contracts as a release gate for AI agents.

What should you measure to know the process is working?

A few numbers tell you whether your judges and your review loop deserve trust:

  • Judge agreement with reviewed labels. Compare each judge's decisions with human-reviewed labels and investigate false passes as well as false failures.
  • False acceptance count. Cases a judge passed that a reviewer failed. Push this toward zero on critical criteria.
  • Disagreement rate per criterion. A criterion with constant disputes usually has a vague rubric.
  • Unknown rate. A rising Unknown count means evidence capture is breaking, not that the agent improved.
  • Time to disposition. Disputed cases that wait weeks block releases and get relabeled under pressure.

Evaluation does not stop at release. Keep sampling production runs and feed reviewed cases back into the reference set, as covered in continuous evaluation for AI agents.

How does ContextQA support judge calibration and evidence?

ContextQA's AI agent testing evaluates agents with metrics that are either an LLM Judge or a Deterministic check, each with a scope and a weight. LLM rubrics define PASS, PARTIAL and FAIL conditions, and you can inspect the reasoning behind each score (metrics and judges).

For actions, a Verifier tool lets the judge check actual backend state through a read-only REST or AWS Lambda call. Assertions run in three layers, Requested, Confirmed and Persisted, and a booking-style outcome passes only when the required layers agree. A verifier contradiction makes the related tool-grounded criterion fail, and missing evidence is not treated as a successful transaction (verify tool actions).

That maps directly to the access request scenario above: the judge reads the backend record, sees a request rather than an approval, and fails the claim instead of rewarding the fluent reply. The docs also recommend recording the selected judges and rubric revision with the run evidence, so a disputed case can be reviewed later against the exact rubric that produced it.

If you already run LLM evals and want to see where end-to-end checks fit, read Already have LLM evals? Where end-to-end agent testing fits, or compare options in our AI agent evaluation tools comparison.

Bottom line

Judge disagreement is a signal worth keeping. Narrow each judge to one question, ground action checks in backend evidence, classify every dispute before changing anything, and keep Unknown as a real state in the release record. Teams that do this can trust a green result because they know exactly what it covers.

Want to run this on a real agent workflow? Join the ContextQA pilot program or book a demo and bring one agent journey with its approval boundaries.

Sources

Share the Post:

Author

Deep Barot

CEO @ ContextQA | Agentic AI for Software Testing | Context-aware Testing

Deep Barot is the Founder and CEO of ContextQA, the only AI testing platform that understands context. He brings decades of experience across DevOps, full-stack engineering, cloud systems, and large-scale platform development.
AI Insights
Real User Intelligence Platform

Turn live sessions into test coverage. No prompts, no manual design - just pointed at your URL and generating suites within minutes.

Minutes
From URL to generated test cases
Zero
Prompts or manual test design needed
40%+
Average coverage increase after first run
100%
Based on real user behavior, not guesses

Frequently Asked Questions

It can reduce repetitive evaluation work when its judgments are calibrated against reviewed labels and monitored. Keep human ownership for ambiguous requirements, consequential disputes and the final release decision.
Keep both verdicts, mark the case as disputed and send it to the owner of the requirement. The reviewer checks the input, tool calls, backend state and rubric version, classifies the cause and records the final disposition with a reason.
If finishing within an agreed deadline is a requirement, a timeout can fail that criterion. Other criteria may stay Unknown because the evidence was never produced. Keep both distinctions in the record.
Ask each judge one narrow question, give it the expected behavior and evidence, swap the order of candidate answers and re-run, and calibrate verdicts against human-reviewed labels. Version the grader prompt and model so changes are traceable.
It establishes results for the tested configuration and scenarios. The release decision also needs deployment boundaries, monitoring, clear ownership and a response plan for behavior outside that coverage.
Related Blogs