Methods and Decisions

Automated Evaluation Methods

With the evaluation environment, dataset, and clear metrics system in place, the core question becomes: how to score? For tasks with clear correct answers (e.g., math problems, SQL queries), simple binary judgment (correct/incorrect) is sufficient; but for open-ended tasks (e.g., customer service dialogues, report writing), more refined evaluation methods are needed.

Code-based automatic verification only covers scenarios with standard answers; scoring open-ended tasks is the main topic of this section. Among these, the design of reward signal density (from binary rewards to process rewards to generative rewards) and training methods for reward models are left for systematic discussion in the post-training section of Chapter 7; this section answers a more fundamental question: how to use LLMs to automatically judge the output quality of open-ended tasks.

LLM-as-a-Judge: The Core of Automated Evaluation

Why is LLM-as-a-Judge needed? For open-ended tasks (e.g., generating reports, handling customer complaints, creative content), there are no standard answers for automatic comparison, and human evaluation is costly and difficult to scale. LLM-as-a-Judge balances the scalability of automation with human expert judgment by having a language model evaluate outputs against expert-defined scoring criteria (a Rubric). The method has known limitations, though: the judge model carries its own biases (most typically length bias—a tendency to score longer, more detailed responses higher even when they are no more correct), and repeated judgments of the same input can vary. Length bias in particular warrants specific countermeasures. Three common defenses are: penalize verbosity explicitly in the Rubric and cap response length per task type; in pairwise comparisons, bring the two candidates to similar lengths before judging; and regularly audit the correlation between scores and response length—if high scores almost always go to long responses, the judge has been swayed by length and the Rubric needs revision. To address these challenges systematically, Rubric design must follow the principles below:

Rubric (Scoring Criteria): The Basis for LLM Judgment.

Four Rubric Principles (Scale AI, “Rubrics as Rewards”):

(1) Based on Expert Guidance—A Rubric must reflect domain knowledge, capturing the core facts and reasoning steps. A Rubric for medical Q&A, for instance, needs diagnostic criteria and the medical errors that must be avoided; one without expert grounding can only capture surface features like fluency.

(2) Comprehensive Coverage—A Rubric should cover factual accuracy, logical coherence, completeness, and safety. It should not only define positive standards but also explicitly identify Pitfalls—i.e., high-risk common errors, such as recommending unverified therapies in medical advice.

(3) Standardized Importance Weighting—Classify criteria as Essential, Important, Optional, or Pitfall items. The scheme supports a Veto mechanism: for example, in a customer service scenario, hallucination (fabricating false information) is a typical veto dimension—regardless of how well other dimensions perform, if false information appears, it must be vetoed. This also helps prevent reward hacking through keyword stuffing.

(4) Self-Contained Evaluation—Each evaluation item is independently actionable and does not rely on the evaluator’s domain knowledge. Abstract standards like “the response demonstrates deep understanding” should be avoided, replaced by verifiable standards like “cites at least two authoritative theories and accurately explains how they support the conclusion.”

The key practice: define objectively verifiable scoring levels for each dimension, with concrete examples and edge cases to resolve ambiguous situations. Actively guard against Reward Hacking—the Agent finding a “shortcut” to high scores without actually completing the task—by explicitly penalizing hallucination, sycophancy, keyword stuffing, and dodging hard questions. A Rubric is an iterative product: trial use reveals disagreements among evaluators, and the Rubric gradually evolves through this feedback from abstract principles into a detailed casebook.

Here is a complete Rubric that follows the four principles, using a user memory Agent as the example. Test question: “Who is my daughter’s pediatrician?” (The answer requires linking information across two conversations: the first conversation mentions “my daughter’s name is Lily,” the second mentions “took Lily to see Dr. Chen”).

rubric:
  dimensions:
    - name: Factual Correctness
      weight: essential        # Essential item
      scoring:
        4_Excellent: "Correctly answers Dr. Chen, and links to daughter Lily"
         3_Good: "Correctly answers Dr. Chen but does not mention that Dr. Chen is Lily's doctor"
        2_Passable: "Gives the correct doctor but with additional uncertain information"
        1_Fail: "Gives an incorrect doctor's name, or answers 'I don't know'"

    - name: Information Completeness
      weight: important        # Important item
      scoring:
        4_Excellent: "Proactively supplements relevant information (e.g., last visit date, diagnosis)"
        3_Good: "Answers the core question without omission"
        2_Passable: "Answers the core question but omits available related information"
        1_Fail: "Key information is missing"

    - name: Reasoning Correctness
      weight: important
      scoring:
        4_Excellent: "Correctly links the two cross-session pieces of information: 'daughter=Lily' and 'Lily's doctor=Dr. Chen'"
        3_Good: "Correctly links but the reasoning path is not clear enough"
        2_Passable: "Partially correct linking"
        1_Fail: "Incorrect linking (e.g., mistaking the user's own doctor for the daughter's doctor)"

    - name: Hallucination Detection
      weight: veto             # Veto item: once triggered, total score is zero
      scoring:
        pass: "All information can be traced back to historical conversation records"
        fail: "Fabricated information not present in the conversation (e.g., fictitious visit dates, diagnoses)"

  edge_cases:
    - "If the user has multiple daughters who see different doctors, should ask which daughter"
    - "If the memory contains both 'Dr. Chen' and '陈医生' (the same name written in Chinese), should recognize them as the same person"

Good Rubric vs. Bad Rubric: Each scoring level above specifies verifiable, concrete behavior (“Correctly answers Dr. Chen”) rather than descriptions that cannot be judged objectively, like “demonstrates a deep understanding of memory.” The veto item sets the bottom line: even if every other dimension scores full marks, a single instance of hallucination results in an automatic zero.

Send this Rubric together with the Agent’s actual response to the judging model, which will score each dimension and provide reasoning. By running this across dozens of test cases, you can systematically identify the Agent’s capability gaps—for example, an average score of 2.1 on the “cross-session association” dimension clearly points to deficiencies in memory retrieval or information correlation.

The Same-Family Model Problem and Multi-Source Judging.

When the Agent and the judging model come from the same family, the Agent may learn to exploit the judging model’s preferences and blind spots.

This is precisely what Goodhart’s Law states: when a metric becomes an optimization target, it ceases to be a good metric. The more an Agent is trained or tuned on a particular scoring system, the more it tends to exploit loopholes in that system rather than genuinely improving its capabilities.

More insidiously, the Agent will gradually learn to avoid the types of errors that the judging model is not good at detecting, making the scoring system appear perfectly fine.

The mitigation is multi-source heterogeneous judging—independent judges drawn from different model families (if the Agent runs on Claude, judge with GPT-5 and Gemini). Different families’ biases are often orthogonal, so the Agent can rarely fool all the judges at once. Use the same Rubric so everyone judges the same target, and aggregate by weighted averaging or consistency checks. In deployment, a single model can handle rapid evaluation, with periodic quality audits run against the full multi-source setup.

Multi-source judging addresses the question of which models should serve as judges; the next question is which modalities should be evaluated—extending LLM-as-a-Judge from text to speech, images, and video is another axis of evaluation coverage.

Multimodal LLM-as-a-Judge.

Multimodal judging extends LLM-as-a-Judge to the domains of speech, images, and video. Four common directions are as follows.

TTS Evaluation (TTS stands for Text-to-Speech): Assesses accuracy, naturalness, voice consistency, and emotional expression. These dimensions can capture prosodic issues that traditional WER (Word Error Rate) struggles to detect.
ASR Evaluation (ASR stands for Automatic Speech Recognition): Performs semantic impact assessment—misrecognizing “today’s weather” is harmless, but misrecognizing “transfer one thousand” as “ten thousand” could have serious consequences.
UI Evaluation: Uses a Proposer-Reviewer mechanism to check for issues like text overflow, color contrast, and button placement. Here, the proposer-reviewer is used as an evaluation method, differing from its use as a generation system component in Chapter 5, but the core mechanism is the same—one model generates, another independently reviews.
Video Editing Evaluation: Verifies the correctness of clip start/end points and effect application through keyframes.

Beyond manually defining Rubrics, specialized generative reward models can be trained to automate judging—this involves training methods for reward models, which will be discussed in detail in Chapter 7.

In practical model selection, we often face the question: “Which is better, A or B?” Pairwise comparison provides an evaluation method that does not rely on absolute scores.

Pairwise Comparison and Model Ranking

Elo Rating (a ranking system originally designed for chess) quantifies the relative ability of models through a large number of pairwise matchups: the larger the rating difference, the higher the expected win rate for the stronger model. For example, if Model A has a rating of 1200 and Model B has a rating of 1000, the Elo system would predict A’s win rate to be approximately 76%. If B unexpectedly wins, B gains more points and A loses more—an upset triggers a larger correction, which is what lets rankings converge quickly on true ability. The statistical foundation is the Bradley-Terry model: each model is abstracted as a latent “strength score,” and the probability of one beating another in a matchup is determined by the difference between their scores. Elo is the engineering implementation of this model in online-update form.

Chatbot Arena uses anonymous random matchups—users blindly choose the better response without knowing the model’s identity, and rankings are derived from millions of votes. The advantage is that no “absolute standard” needs defining; all that is required is human judgment on “which is better, A or B.” The limitation: rankings depend on what users happen to ask. If a flood of users ask programming questions, models strong at programming rank higher—which may say little about their level on other tasks.

When pairwise judging is performed by an LLM rather than human voting, one must also guard against Position Bias—the judging model systematically favors the candidate appearing in a certain position (usually the first), and the judgment may remain unchanged even if the content of the two candidates is completely swapped. The standard mitigation method is to evaluate each pair twice with swapped order: once with A first, once with B first, and average the two results; a stricter approach is to only count cases where the two judgments are consistent, and treat inconsistencies as ties or send them for human review. Chatbot Arena’s approach is essentially the same—randomizing the display positions of the two responses so that position bias cancels out over a large sample.

From Evaluation to Training: Transfer of Pairwise Comparison Signals. Pairwise comparison is not only an evaluation tool but also an important source of signals for post-training. The GRPO (Group Relative Policy Optimization) algorithm, which will be introduced in Chapter 7, incorporates the “compare which is better” judging approach into model training—its core idea is to sample multiple candidate answers for the same question and estimate advantages from their relative merits (rather than absolute scores), thereby avoiding the need for the extra value network (critic, used to estimate baselines) that PPO must train. Note that GRPO drops the value network, not the reward signal: it still relies on a reward model or verifiable reward rules to judge each candidate. This is only a foreshadowing—the full derivation, the comparison with PPO/DPO, and the implementation details for Agent post-training all come in Chapter 7.

Evaluation-Driven Model Selection

Model selection is not simply about “choosing the strongest model”; it involves making evaluation-driven trade-offs across multiple dimensions based on the application scenario.

Key Dimensions for Selection

Throughput and Latency are two families of metrics that are easily confused; untangling them takes only one fact—LLM inference runs in two stages. Prefill reads the entire context at once and determines the Time To First Token (TTFT): the delay between the user pressing Enter and the first character appearing. The longer the context, the slower the prefill and the higher the TTFT. Decode then generates the response token by token, setting the generation speed (tokens/second)—which also dictates thinking time: at 50 tokens/s, a model producing 2000 thinking tokens spends 40 seconds just thinking.

Around these two stages, the main throughput and latency metrics are as follows:

Input Throughput / Output Throughput: Correspond to the speed of Prefill and Decode, respectively.
TTFT: Equals queuing time plus Prefill time; it is the user-perceived “responsiveness.”
Thinking Latency: The number of thinking tokens generated can vary severalfold across models, and thinking length is not necessarily positively correlated with task effectiveness—measure each model’s thinking token usage and the corresponding benefit on your own workload, rather than inferring from public leaderboards alone.
p95 Tail Latency: The latency that 95% of requests will not exceed. It is a better indicator of real user experience than the average, which can be pulled down by a large number of fast requests, masking severe slowdowns experienced by a minority of users.

Cost: Pricing for input/output/cache tokens. Cost should not be evaluated in isolation—a cheap model with a low success rate may actually incur higher costs due to frequent retries. The average cost per task and the cost-performance ratio need to be calculated.

Performance: The precise definitions of Pass@1, Pass^k, Pass@k, and Best@k are given earlier in the “Evaluation Metrics System.” Here, we only discuss how to choose in the context of model selection—for daily scenarios, focus on Pass@1 (single-attempt average success rate); for critical operations, prioritize Pass^k, focusing on the stability of “never making a mistake”; for exploratory tasks, prioritize Pass@k or Best@k, looking at the upper bound of capability given enough opportunities; for open-ended tasks, use multi-dimensional Rubric scoring.

Rate Limits and Reliability: RPM (Requests Per Minute) / TPM (Tokens Per Minute) limits affect concurrency capabilities, and some APIs dynamically adjust quotas during peak hours. In terms of robustness, pay attention to out-of-distribution data, adversarial inputs, and long-running stability (whether issues like mode collapse or attention drift occur).

Budget–capability curves: A single score at a fixed budget is not enough to determine whether an Agent can handle long-horizon work. In addition to success rate, report how performance changes with wall-clock time, tokens, tool calls, or compute budget. RE-Bench makes the problem concrete: with a total budget of two hours per environment, the best Agent scored about four times as high as human experts; humans, however, benefited more from additional time, narrowly surpassed the best Agent at eight hours, and scored about twice as high when multiple attempts were given 32 total hours. Short-budget leadership therefore cannot be extrapolated directly to long-running capability. Model selection should compare several budget points close to the duration of the real workload.

In practice you can mix models: lightweight models on simple requests to cut costs, powerful models on complex tasks to protect quality; or specialist models on particular sub-tasks (image understanding, code generation), collaborating through sub-agent mechanisms. Any such heterogeneous combination must itself be validated by evaluation, to confirm the overall benefit outweighs the added system complexity.

Cost Analysis of Agent Systems

Cost is the most easily underestimated dimension of model selection. If your Agent is in production or headed there, do not skip this section.

The previous section listed cost among the key selection dimensions, but Agent costs are far more complex than simple token pricing—multi-turn reasoning, tool calls, and context accumulation make costs grow non-linearly. Systematic cost analysis is an indispensable part of the evaluation system and a prerequisite for production deployment.

Components of Cost.

The cost of an Agent system can be decomposed into three levels:

Model inference cost is the most direct component, determined by the consumption of input tokens and output tokens. However, in Agent scenarios, there are two often-overlooked amplifying factors. The first is the context accumulation effect: each time an Agent calls an LLM, it sends all previous conversation history and tool outputs together (so the model can understand the context). Without effectively utilizing KV Cache (i.e., caching already processed context to avoid redundant computation), the cost grows very quickly—Round 1 sends 1000 tokens, Round 2 sends 2000 tokens, Round 3 sends 3000 tokens, totaling 1000+2000+3000=6000 instead of 3×1000=3000. The more rounds, the larger the gap. The second is thinking token cost: models that support thinking generate a large number of thinking tokens. Although these tokens are not displayed to the user, they are still billed.

Tool call cost includes external API fees (search engines charge per query, database queries consume computing resources), sandbox resources for code execution, and an easily overlooked indirect cost: the token cost incurred when tool outputs are injected into the context. The content returned from a single web search might occupy 2000-5000 tokens, and it will be repeatedly billed as input in every subsequent round of inference.

Infrastructure cost covers operational overhead for vector databases (used for RAG retrieval), message queues, relational databases, and logging and tracing storage (for observability).

A concrete example illustrates the non-linear growth of costs. Table 6-4 uses the customer service refund Agent from the beginning of this chapter as an example, with a set of illustrative token price parameters to break down the cost of three rounds of calls, demonstrating the impact of multi-round context accumulation and cache hits on expenses.

Table 6-4 Three-Round Cost Example for the Customer Service Refund Agent

Round	Operation	Input Tokens	Output Tokens	Round Cost
1	System prompt + user question → Decide to query the order	2,500 (2,000 system prompt)	150	$0.0098
2	Previous-round context + tool result → Decide whether to initiate a refund	3,200 (2,000 cache hit)	120	$0.0060
3	Previous-round context + refund result → Reply to the user	3,800 (3,200 cache hit)	200	$0.0058
Total		9,500	470	$0.022

Note: Calculated using example prices of $3/million tokens for input and $15/million tokens for output. The cache-hit portion is assumed to be billed at 10% of the input price (discounts vary by vendor; for example, Anthropic’s cache write is about 1.25 times the input price and cache read is about 0.1 times; this is simplified to only the read discount).

Three rounds come to $0.022—cheap, it seems. Without any cache, the input cost alone would be about $0.029, roughly $0.036 with output included; caching here saves nearly half the input cost, consistent with the empirical range cited later (“KV Cache can reduce input costs by 30%-60%”). But watch the amplifying factors. Enable thinking mode and each round emits an extra 500-2,000 thinking tokens, potentially tripling or quintupling the cost. Let one tool return a 5,000-token web page and every subsequent round pays for those tokens again. Let the Agent take a detour and need 10 rounds, and the context balloons past 20,000 tokens, far beyond this simple scenario. The core of cost optimization is therefore not picking a cheaper model but controlling the number of rounds and the growth of context.

Cost Optimization Strategies.

From a quantitative perspective, the most effective input-side levers are KV Cache Reuse (maintaining a stable prefix so that repeated system prompts, tool definitions, and historical rounds are billed at the cache price, reducing input token costs by 30%-60%—in the three-round example above, caching saved nearly half the input cost), Context Compression (compressing historical trajectories, truncating redundant tool outputs, directly controlling the growth rate of context, especially effective in long tasks), and Tiered Model Routing (simple requests go to lightweight models, complex reasoning goes to powerful models). The specific implementations of these three methods—prefix stability design, compression timing and strategy, and routing mechanisms—have been discussed in detail in Chapter 2 and will not be repeated here. This chapter supplements them with two methods from evaluation and operations.

Asynchronous Batch Processing accumulates non-real-time tasks for batch processing, leveraging batch pricing discounts from API providers; in self-deployment scenarios, it also improves GPU utilization during off-peak hours.

Cost Monitoring and Budget Control.

In a production environment, a real-time cost monitoring system should be established: track token consumption and API costs by task type, model, user, etc. Also, set a cost cap for each task—automatically terminate the Agent when it falls into a loop or explores too deeply, preventing a single task from incurring abnormally high costs.

Evaluation-Driven Continuous Iteration

Model selection is not a one-time decision but a continuous process, adjusted as models evolve. The chapter opened with the claim that an evaluation system lets you keep pace with model evolution; a concrete model-switching case shows how that plays out in a real decision.

Suppose your Agent system is currently built on Claude, excelling in tool calling and complex orchestration. One day, Gemini releases a new model, and public benchmarks show it surpasses Claude on several metrics at a lower price. At this point, your question is not “Is Gemini better than Claude?” but “On my specific tasks, is Gemini better than Claude? How much better? What is the switching cost?”

A team with a solid evaluation system can answer this in hours: run the new model on its own evaluation dataset and compare task success rate, tool call accuracy, latency, and cost. You might find the new model really is better and cheaper on simple tasks—but in the core scenarios involving complex multi-round tool orchestration, its success rate drops by 5%. Once you confirm the difference exceeds the estimated sampling noise (see “Statistical Significance of Evaluation Results” below), your decision becomes a differentiated strategy—migrate simple tasks to the new model to cut costs, keep the original model on complex tasks to protect quality—rather than a blind wholesale switch. Decisions this granular and data-driven are only possible with an evaluation system built in advance.

Statistical Significance of Evaluation Results

“A switching decision within hours” rests on an implicit premise: the score difference you observed is real signal, not sampling noise. With a limited evaluation set and non-deterministic model outputs, that premise does not hold automatically.

A rough estimate of this sampling noise is the standard error of a binomial proportion (which characterizes the fluctuation of the success rate due to sampling randomness; the larger the value, the less reliable the success rate). If the success rate p is measured on n test cases, the standard error is approximately √(p(1-p)/n). For a concrete example: 100 cases, success rate 70%, standard error ≈ √(0.7×0.3/100) ≈ 4.6%. An approximate 95% confidence interval is p ± 2 standard errors, meaning an interval that would contain the true rate in about 95% of repeated samples, i.e., 70% ± 9 percentage points. A three-percentage-point difference like “new model 73% vs. old model 70%” therefore sits entirely inside the noise band—treating the two success rates as independent, the standard error of their difference is about √2 times the individual standard error (here about 6.5 percentage points). One caveat: that √2 assumes the two measurements are independent, whereas in practice both configurations usually run on the same set of tasks, so the samples are not independent. The independence assumption is merely a conservative upper bound for a quick check on whether a small difference deserves attention at all. Even by that conservative yardstick, a three-percentage-point gap falls far short of the 6.5-percentage-point standard error—switching models on such evidence is little better than a coin flip.

Agent evaluation adds another layer of non-determinism: same model, same dataset, and two runs can still drift apart—temperature sampling, fluctuating tool returns, and environmental timing all inject randomness. So never base a decision on a single run’s numbers. Run multiple times and average (say, 3-5 runs per configuration), reporting both the mean and the spread. This is exactly why, in the hypothetical case later, every configuration is “run 5 times (using different random seeds).”

Hence a practical principle: when the score difference is smaller than the estimated sampling noise, do not make a switching decision. But before settling on “don’t switch,” reach for a more sensitive—and more correct—analysis. When two configurations run on the same set of tasks, the right default is paired analysis: compare win/loss task by task, look only at the cases where the two disagree (one correct, one wrong), and apply something like McNemar’s test to judge significance. Pairing subtracts out the shared noise of task difficulty, making it far more sensitive at the same sample size than differencing two independent success rates—the earlier √2 estimate is just a conservative, mental-math sieve for ruling out differences that obviously fall short. If paired analysis still leaves the difference uncertain, only then consider growing the sample—and note that the standard error scales as 1/√n, so going from 100 to 400 cases merely halves the estimated sampling noise. Expansion is expensive. Read the other way: if an improvement’s expected benefit is only 2-3 percentage points and your evaluation set has a few dozen cases, the evaluation simply cannot tell whether the improvement works—the priority is to grow the evaluation set, not to keep iterating the Agent.

One more easily overlooked pitfall: multiple comparisons. Test a batch of hypotheses in parallel and the probability that at least one conclusion is a false positive climbs fast—even at a 95% confidence level per conclusion, across 6 hypotheses the chance of hitting at least one false positive is 1 − 0.95^6 ≈ 26%. The more hypotheses you run in parallel, the harder it becomes to avoid one that merely looks significant. Countermeasures come in two kinds: tighten the significance threshold for each conclusion as the number of hypotheses grows (a Bonferroni-style correction), or re-run any positive result in an independent confirmatory pass and believe it only if it replicates. The later section “From Data to Hypotheses” will test H1–H4, four truly parallel hypotheses (H5 and H6 are conditionally triggered and not run simultaneously with the first four), which is a typical scenario for this pitfall.

Evaluation-driven decisions rely on high-quality data, which comes from the systematic recording of the Agent’s operational process—this is what observability addresses.

Datasets and Metrics — Pass@k vs Pass^k and the metric dictionary
Observability and Production Evaluation — where the high-quality run data comes from