Agent Model vs. Judge Model: How to Choose the Right LLM for Production and Evals
Learn how to choose an LLM for an agentic application and a separate, validated judge model for evaluating its outputs.
Building an agentic application often requires choosing models for two very different jobs:
- An agent model that performs production tasks such as routing requests, calling tools, retrieving information, and generating answers.
- A judge model that evaluates whether the agent performed those tasks correctly.
These models are selected using different criteria. An agent model must satisfy the application's quality, cost, latency, reliability, and tool-use requirements. A judge model must apply an evaluation rubric consistently and agree with trusted human reviewers.
In evaluation terminology, the system being tested is the candidate configuration. In an agentic application, that candidate may be one model call, a node configuration, or an entire workflow containing prompts, tools, retrieval, routing, and state.
The model that performs best as an evaluation judge is not necessarily the model you should deploy inside your agentic application, and picking one for the other's job is how evaluations end up disagreeing with what users actually experience.
Recommended Terminology
| Term | Meaning |
|---|---|
| Agent model | The production LLM wired to an agent, node, or workflow step |
| Judge model | The LLM that evaluates outputs during an evaluation |
| Candidate configuration | The complete system being evaluated: model, prompt, tools, retrieval, and workflow |
| Production reviewer | A model that checks outputs or actions inside the live application |
Avoid using "evaluation model" by itself -- it can mean either "the model being evaluated" or "the model performing the evaluation." Judge model (or evaluator model) is less ambiguous.
1. Two Model-Selection Decisions in an Agentic Application
Production decision:
Which model should perform this agent step?
Evaluation decision:
Which model can reliably determine whether that step succeeded?
Treat these as separate experiments with separate selection criteria. Confusing them leads to poor decisions: picking a production model because it scored well as a judge, or picking a judge because it's already your production model.
2. What Counts as the Agent Model
A LangGraph-style workflow (or any agent framework) may have multiple production models, not one universal "agent model":
Router node → small, fast model
Retrieval-planning node → model reliable at query generation
Answer node → model optimized for response quality
Hard-case fallback → stronger reasoning model
Model choice can happen per node. There's no requirement that a single model handle every step of the application.
3. What Is Actually Being Evaluated
Whatever runs inside a node, the unit an evaluation tests is the candidate configuration:
Candidate configuration
= model
+ prompt
+ tools
+ retrieval
+ workflow and routing logic
+ runtime settings
Depending on the evaluation framework, the candidate might be an isolated model call, a prompt-and-model combination, a RAG pipeline, an agent with tools, or an entire workflow. Clearly define the unit under test before interpreting the results -- this prevents readers (and you) from attributing every evaluation result to "the model" when the prompt, tools, or retrieval may be the actual cause.
The candidate is the "student taking the test."
Candidate flow
JSONL row:
"What is our refund policy for annual plans?"
↓
Candidate configuration
(agent model + prompt + any tools/RAG involved)
↓
Candidate answer:
"Annual plans can be refunded within..."
↓
Judge grades the answer
Whether this tests an isolated model or the complete application depends on where the evaluation harness connects to your system.
4. How to Choose a Model for the Agentic Application
Focus on:
- Task quality
- Tool selection and argument accuracy
- Structured-output reliability
- Latency
- Token and tool-call costs
- Context requirements
- Rate limits and throughput
- Retry behavior
- Safety and policy requirements
- Performance on the application's own dataset
A useful principle: choose the cheapest and fastest model that consistently clears the quality and reliability threshold for that specific agent step.
5. How to Choose a Judge Model
Focus on:
- Agreement with human labels
- Rubric adherence
- Run-to-run consistency
- False-positive and false-negative rates
- Resistance to verbosity and style bias
- Pairwise position bias
- Structured scoring reliability
- Cost at the expected evaluation volume
A useful principle: choose a judge based on demonstrated grading performance, not its price, size, or general benchmark ranking.
6. Why the Same Model May Not Be Right for Both
| Requirement | Agent model | Judge model |
|---|---|---|
| Task performance | Critical | Helpful but insufficient |
| Agreement with human graders | Usually secondary | Critical |
| Latency | Often critical | Usually less critical offline |
| Cost per call | Multiplied by production traffic | Multiplied by evaluation volume |
| Tool-use reliability | Critical when tools are used | Only needed when judging traces or actions |
| Rubric adherence | Useful | Critical |
| Run-to-run grading stability | Useful | Critical |
| Structured output | Depends on the node | Often important for score processing |
A strong production model can still be a poor judge, and a reliable judge can be too slow or expensive for the production path.
7. What a Judge Model Receives
The judge is a second model that scores the candidate's output. The judge is the "teacher grading the test."
Depending on the metric, the judge may receive the original input, candidate output, reference answer, retrieved context, tool traces, and a grading rubric. For a reference-based correctness metric, the judge might receive:
User question:
What is our refund policy for annual plans?
Expected answer:
Annual plans are refundable within 30 days.
Candidate answer:
Annual plans can be refunded within 30 days.
Judge instructions:
Mark the answer correct only if it is factually correct,
follows policy, and does not invent details.
A binary correctness score is only one option. Some evaluations benefit from richer output:
- Binary pass/fail
- Ordinal scores, such as 1-5
- Per-criterion scores
- Error categories
- Pairwise preference
- Structured failure labels
{
"pass": true,
"scores": {
"factual_correctness": 1,
"policy_compliance": 1,
"completeness": 0.8
},
"failure_types": [],
"reason": "The response states the correct refund period but omits one eligibility condition."
}
The judge prompt matters because it defines what "good" means:
- Correct: matches the expected facts
- Concise: no unnecessary padding
- Policy-compliant: follows the rules you gave it
- Safe: no unsafe or off-brand content
- Complete: answers the full question
- Grounded: sticks to the supplied context
- Valid format: e.g. valid JSON when required
- No hallucinations: doesn't invent details
8. Validate the Judge Before Trusting the Evaluation
An LLM judge is another probabilistic model, not an objective answer key. It can misunderstand the rubric, prefer verbose answers, overlook subtle errors, or produce inconsistent scores.
Before using a judge to compare agent model configurations:
- Have humans label a representative subset of examples.
- Run potential judges against that subset.
- Measure agreement with the human labels.
- Inspect false positives, false negatives, and inconsistent rationales.
- Refine the rubric and scoring format.
- Repeat difficult examples to check run-to-run stability.
- Randomize answer order in pairwise comparisons to catch position bias.
- Watch for model-family or self-evaluation bias (a judge favoring outputs from its own model family).
- Version the judge model and its prompt, and re-validate whenever either changes.
- Keep the judge and rubric fixed when comparing candidate configurations.
A judge does not necessarily need to be the largest available model. It needs to be sufficiently accurate and consistent for the distinctions your evaluation is intended to measure.
If two candidates score 88% and 89%, do not automatically conclude that the second is better. Examine sample size, repeated runs, judge variance, and the practical importance of the disagreements.
9. Use Deterministic Evaluators Where Possible
Not every criterion should be handed to an LLM judge. Use deterministic evaluators where possible:
| Criterion | Preferred evaluator |
|---|---|
| Valid JSON | JSON parser/schema validator |
| Exact field values | Programmatic comparison |
| Required citations | Structural validation |
| Tool-call arguments | Schema and rule checks |
| Forbidden operations | Policy engine or allowlist |
| Semantic correctness | Human or calibrated LLM judge |
| Tone and style | Rubric-based judge |
| Completeness | Judge, task-specific rules, or both |
Use deterministic checks whenever the requirement can be expressed as code. An LLM judge is useful for semantic qualities such as relevance, completeness, and groundedness, but it should not replace a parser, schema validator, policy engine, or authorization check. This matters especially for anything touching safety: a reviewer model should not be the sole security control before executing a destructive action like a database write.
10. Compare Production Cost With Evaluation Cost
Pricing varies by provider, hosting platform, region, batch vs. real-time API, cached vs. uncached input, model version, and contract or volume discounts -- so treat any specific number as a snapshot, not a constant. Instead, learn to calculate it, and keep the two cost surfaces separate:
Production model cost
≈ production requests
× agent model calls per request
× average cost per call
Evaluation cost
≈ dataset rows
× candidate configurations
× repetitions
× (
candidate execution cost
+ total judge cost per row
)
Production cost scales with real user traffic and compounds forever; evaluation cost is a one-off (or periodic) expense that scales with your dataset and how many configurations you compare. Underestimating this distinction is a common reason teams either overpay for evaluation or underinvest in judge quality.
Illustrative (fictional) example, not current provider prices:
| Configuration | Candidate cost/row | Judge cost/row | Rows | Approx. eval total |
|---|---|---|---|---|
| Small agent model + strong judge | $0.0002 | $0.0020 | 1,000 | $2.20 |
| Large agent model + strong judge | $0.0010 | $0.0020 | 1,000 | $3.00 |
Example: Choosing a Model for an Agent Node
Agentic application:
Customer-support agent
Node under evaluation:
Answer-generation node
Configuration A:
Small agent model + support prompt + retrieved context
Configuration B:
Larger agent model + same prompt + same retrieved context
Judge:
Fixed and validated judge model
Metrics:
Correctness, groundedness, completeness, and policy compliance
Possible results:
| Agent model configuration | Correctness | Groundedness | Cost/1,000 runs | P95 latency |
|---|---|---|---|---|
| Configuration A | 84% | 91% | Lower | Lower |
| Configuration B | 91% | 94% | Higher | Higher |
The production decision depends on whether the improvement clears the application's threshold and justifies its operational cost:
- Choose Configuration A if 84% correctness is acceptable and cost/latency matters most for this node.
- Choose Configuration B if the improvement in correctness and groundedness prevents enough mistakes to justify the higher cost and latency (use the cost formulas above to quantify that tradeoff).
- Improve the prompt, retrieved context, or output format if neither configuration clears your bar.
11. Candidates and Judges in an Agent Workflow
The candidates are the configurations you are considering for the production application, whether it uses LangGraph, another agent framework, or a custom orchestration layer.
Agent workflow
→ calls a model
→ candidate configurations under consideration
You run evals to answer: which of these configurations should my workflow use for this node or step? For example, in LangGraph or another orchestration framework:
Router/classifier node → small, cheap model
Main answer node → mid-sized model
Hard reasoning node → a stronger model
So yes: candidate configurations are potential production configurations.
Judges are usually not part of the normal user-facing workflow path. They're used in an offline test process:
Your test prompts
→ candidate generates answers
→ judge scores answers
→ you inspect scores and failures
→ choose/improve the candidate
Judge cost is incurred during evaluation, not on every customer request.
12. Production Reviewers and Escalation Models
It's not a strict rule that judges stay offline. You can use a judge-like model in your workflow if it has a production job -- that makes it a production reviewer, a distinct role from the eval judge, even if it's the same underlying model. Examples:
Draft response
→ critic/reviewer node checks for policy issues
→ final response
or:
Agent creates SQL query
→ reviewer model verifies it is read-only and safe
→ execute query
In that case, it is no longer just an eval judge -- it is a production reviewer/critic node, and you pay for it and add latency on every request. (As noted above, a reviewer model should not be the sole safety control for anything destructive or irreversible -- pair it with deterministic checks.)
Escalation: a stronger model for high-reasoning decisions
A third pattern sits between "pure eval judge" and "always-on reviewer": escalation. Instead of running a stronger, validated model on every request, the pipeline only reaches for it when the cheap candidate can't confidently resolve the step.
Cheap candidate handles the request
→ confidence low / ambiguous / high-stakes step detected
→ escalate to a stronger, validated model
→ strong model resolves the decision
→ cheap candidate resumes downstream steps
Typical escalation triggers in an agentic pipeline:
- Low confidence or conflicting tool outputs. The candidate's own score, a retrieval mismatch, or contradictory sources trigger a second opinion.
- High-stakes or irreversible actions. Refunds above a threshold, destructive database writes, or anything touching compliance/policy gets routed to a stronger reasoner before execution.
- Multi-step reasoning failures. A router/classifier node fails to converge after N retries and hands the step to a model with deeper reasoning.
- Out-of-distribution input. The request doesn't match the shapes your eval dataset covered, so you fall back to the model most likely to generalize correctly.
This is different from the always-on reviewer pattern above: the strong model is invoked conditionally, so most requests still pay the cheap agent model's cost and latency, and only the hard fraction pays for a stronger model. Validate the escalation threshold itself the same way you validate candidates: run the escalation path through evaluations, with a judge scoring whether it escalated when it should have and stayed cheap when it should have.
The Clean Distinction
| Role | Where it runs | Primary purpose | Selection criteria |
|---|---|---|---|
| Agent model | Production application | Performs an agent task or node | Task quality, cost, latency, tool use, reliability |
| Candidate configuration | Evaluation harness | Represents the system being compared | Precisely defined model, prompt, tools, retrieval, and workflow |
| Judge model | Usually offline evaluation | Scores candidate outputs | Human agreement, consistency, rubric adherence |
| Production reviewer | Live application | Checks an output or proposed action | Reliability, risk reduction, latency, cost |
| Escalation model | Conditional production path | Handles difficult or high-risk cases | Performance on escalation cases and trigger accuracy |
A reviewer and an escalation model may use the same underlying LLM as your judge, but validate each separately -- they operate under different prompts, inputs, risks, and constraints.
13. Common Evaluation Mistakes
- Picking a production model because it scored well as a judge, or picking a judge because it's already your production model -- these are separate selection problems.
- Assuming a bigger or pricier model is automatically a better judge without validating it against human-labeled examples.
- Using an LLM judge for checks that should be deterministic (valid JSON, exact field values, forbidden operations) instead of a parser, schema validator, or policy engine.
- Leaving the "candidate configuration" undefined -- not being clear whether you're testing a raw model call or the full RAG/tool/agent pipeline.
- Treating a single-run score difference as meaningful without accounting for sample size, repeated runs, and judge variance.
- Putting a judge into the production path unconditionally rather than gating it behind an explicit reviewer or escalation trigger.
- Validating a reviewer or escalation model using only judge-validation data, instead of testing it separately against its own prompts, inputs, and risk profile.
14. Decision Checklist
- Decide per node or step which agent model to run, based on task quality, cost, latency, and tool-use reliability.
- Define the candidate configuration precisely (model, prompt, tools, retrieval, or full workflow) before running an evaluation.
- Select judge models based on agreement with human labels and consistency, not size, price, or general benchmarks.
- Validate the judge against a human-labeled subset before trusting its scores, and re-validate when the judge model or prompt changes.
- Route deterministic requirements to deterministic checks, not the judge.
- Track production cost and evaluation cost as separate cost surfaces using the formulas above.
- Keep the judge and rubric fixed when comparing agent model configurations.
- Decide deliberately whether a judge-like model belongs in production as a reviewer or escalation model -- don't add it by default, and validate it separately from the eval judge.
Conclusion
Choosing a model for an agentic application and choosing a model to evaluate it are different optimization problems.
The production agent model must perform its assigned task while meeting the application's quality, cost, latency, and reliability requirements. The judge model must apply the evaluation rubric consistently and agree with trusted human reviewers.
Do not choose either model based only on size, price, or general benchmarks. Evaluate agent models on representative production tasks, calibrate judge models against human-labeled examples, and use deterministic checks whenever a requirement can be enforced in code.