Evaluation Environments

This chapter is book’s evaluation methodology. Zapvol has no standalone evaluation harness yet; the closest landing is the operations trace / log / cost pipeline (see Observability and Production Evaluation). The rest is kept as forward guidance.

A Concrete Evaluation Example

Before diving into the methodology, let’s build intuition through a complete example. Suppose we have built a customer service Agent and need to evaluate its ability to handle refund requests.

Test Case: The user wants to return an order from 3 days ago (Order #12345, Amount ¥299). Company policy: Full refund within 7 days.

Agent Trajectory:

User: I want to return the headphones I bought 3 days ago, order number 12345. (Today is 2026-04-10)

Agent (thinking): The user wants a refund, I need to check the order information first.
Agent → Calls tool query_order(order_id="12345")
Tool returns: {status: "delivered", amount: 299, date: "2026-04-07", item: "Bluetooth Headphones"}

Agent (thinking): Order date 2026-04-07, today is 2026-04-10, that's 3 days, within the 7-day refund period, eligible for full refund.
Agent → Calls tool process_refund(order_id="12345", amount=299)
Tool returns: {refund_id: "R-98765", status: "processing", eta: "3-5 business days"}

Agent: A refund has been initiated for you. The full refund for Order #12345 (Bluetooth Headphones, ¥299) is being processed,
      and is expected to arrive in 3-5 business days. The refund ID is R-98765, which can be used to check the progress.

Scoring with a Rubric (four dimensions, each scored 1-4). Table 6-1 provides a scoring example for this customer service refund task, illustrating how a Rubric breaks down an Agent trajectory into checkable evaluation dimensions.

Table 6-1 Rubric Scoring Example for Customer Service Refund Task

Dimension	Criteria	Score	Reason
Operational Correctness	Is the refund amount and order number correct?	4	Correctly queried and initiated a ¥299 full refund
Policy Compliance	Does it follow the 7-day refund policy?	4	Order is within the refund period, complies with policy
Information Completeness	Does it provide the amount, arrival time, and refund ID?	4	All three key pieces of information were provided
Hallucination Detection (Veto Item)	Does it fabricate non-existent information?	Pass	All information comes from tool outputs

Hallucination is listed as a veto item rather than a graded scoring dimension because it is orthogonal to quality — a fluent, detailed, and polite response containing false information is far more harmful to the user than a brief but accurate one. (For the general design of the veto mechanism, see the “Four Rubric Principles” section later.)

This test case passed. But a good evaluation doesn’t just test success scenarios; it also probes boundaries and traps — when a user wants to return an order from 15 days ago (beyond the refund period), can the Agent correctly refuse? When a user claims “a customer service representative already approved the refund,” will the Agent believe it without a system record? These boundary scenarios are what truly separate strong Agents from weak ones.

The process above — defining test cases, running the Agent, scoring with a Rubric, and analyzing results — is the basic skeleton of evaluation. The rest of this chapter fleshes out the design of each step.

Automated Evaluation Environment

Agent evaluation requires a repeatable, automated environment — one that can quickly test the effects of changes during development. Building such an environment requires answering three questions: what to evaluate (task definition and verification criteria), whom the Agent interacts with and how to simulate that counterpart, and which scoring criteria to use.

Basic Components of an Evaluation Environment

An evaluation environment consists of five elements — the following sections will focus on dataset design and scoring criteria design:

Dataset: Defines the task set, including initial state, goal description, and optional reference solutions.

Environment State: Tracks mutable state during task execution and must balance realism with controllability. For example, in a customer service evaluation, the environment state includes order records in the database and user account balances. After the Agent calls process_refund, the order status changes from "delivered" to "refunded" and the balance increases. “Realism” requires that state changes follow business logic (refund amount cannot exceed the order amount), and “controllability” requires that each test can be reset to the same initial state.

Tools: Defines the set of operations the Agent can perform — tools should not provide overly high-level abstractions (like “solve user problem”), but should provide atomic operations (like query order, modify booking, send email), forcing the Agent to combine these operations through planning and reasoning.

Rubric (Scoring Criteria): Quantifies the Agent’s performance, which can be binary (pass/fail), continuous (0 to 100 points), or multi-dimensional (scoring accuracy, efficiency, and safety separately).

Interaction Protocol: Specifies the interaction mode and termination conditions.

Tool-Calling Evaluation Environment

For tasks that primarily rely on tool usage, such as code generation and data analysis, the Verifiers framework demonstrates a typical design pattern. The Agent completes the task by calling predefined tools, and verification is based on executable criteria (whether tests pass, whether answers match), without relying on human annotation or model judgment.

Verifiers introduces a hierarchical environment design: SingleTurnEnv is suitable for single-turn tasks (e.g., simple Q&A), ToolEnv supports multi-turn autonomous loops of tool calls, and StatefulToolEnv and SandboxEnv support stateful tools and long-running sandbox environments (e.g., code execution). For example: SingleTurnEnv is suitable for posing a math question and checking the answer directly; ToolEnv fits searching several web pages and synthesizing an answer before verifying the final result; StatefulToolEnv fits modifying database records and verifying the resulting state change; SandboxEnv fits running code in a sandbox and checking the output files. Table 6-2 summarizes these environment types for readers to choose the appropriate evaluation environment based on task state, tool calls, and isolation requirements.

Table 6-2 Verifiers Environment Type Comparison

Environment Type	State Persistence	Tool Calls	Typical Use Case
SingleTurnEnv	None	None	Single-turn Q&A, math problems
ToolEnv	None	Multi-turn	Search + information synthesis
StatefulToolEnv	Yes	Multi-turn	Modifying database records
SandboxEnv	Yes + Isolation	Multi-turn	Code execution and testing

The framework supports parallel sampling and trajectory caching. The complete trajectory (observations, actions, rewards) from each evaluation is saved for subsequent analysis and replay.

The environment also needs to handle the state dependency of operations — the outcome of a tool call depends on the current state. On failure, it should provide clear error messages rather than simple failure flags, allowing the Agent to learn from errors and adjust its strategy.

Human-Computer Interaction Evaluation Environment

Many real-world tasks involve not only tool calls but also conversations with human users. A customer service Agent needs to understand vague expressions, clarify needs, query backend systems, and confirm information with the user. Evaluating such tasks faces a fundamental challenge: how to simulate real users in an automated environment?

The key design principle is Progressive Information Disclosure, which is the fundamental difference between human-computer interaction evaluation and traditional benchmarks. Most benchmarks reveal the complete requirements upfront, but real users can rarely articulate their needs from the start — they often just say “there seems to be a problem with my flight” or “the internet isn’t working.” The Agent must clarify the need by asking questions, and that process is itself a display of capability. In evaluation, therefore, the simulated user’s information must not be revealed to the Agent all at once; it should be disclosed progressively, on demand, as the conversation unfolds.

τ-bench’s solution is User Simulation: using another LLM to play the user role, conversing with the Agent according to predefined instructions. The simulated user receives task instructions (e.g., “I need to cancel tomorrow’s flight”), gradually reveals necessary information to the Agent during the conversation, responds to inquiries, and sends a termination signal when the task is complete. The prompt requires the simulated user to “not reveal all information at once, only provide what is necessary for the current step” and “not fabricate information not provided in the instructions.” The design of user simulation requires a trade-off between authenticity and controllability: behavior should be close to a real user (vague expressions, incomplete information, occasional emotional fluctuations) while following a certain script to ensure reproducibility.

The following is an example of a multi-turn conversation with progressive information disclosure (the user simulator acts according to a fixed script):

User: “There’s a problem with my flight.” Agent: “Which flight is it?” User (revealing per script): “Delta 123, tomorrow morning from San Francisco to New York.” Agent: “What’s the specific problem?” User (revealing per script): “The flight time is too long, I want to change it.” Agent: “Any preferences for the new flight?” User (revealing per script): “Any afternoon flight is fine.”

The user simulator follows a fixed script (known information + disclosure rules), ensuring evaluation reproducibility while simulating the progressive expression style of a real user.

τ-bench is a benchmark for evaluating Agent performance in structured business processes (e.g., airline customer service, retail customer service). Its checks are component-level and multi-dimensional: on one hand, it checks whether the final database state is correct (e.g., the booking record status changes to “cancelled”); on the other hand, it verifies whether the Agent provided the necessary key information during the conversation (e.g., refund amount and arrival time, verified by searching for specific strings or patterns). This dual verification simultaneously examines operational accuracy and communication effectiveness. At the task level, however, these checks ultimately collapse into a binary reward of zero or one — all checks must pass to score 1; any single failure scores 0. Binary rewards make reliability metrics like Pass^k easy to compute (see the “Evaluation Metrics System” section later), at the cost of scoring “operationally accurate but missing one non-critical field” the same as “complete failure.”

The enhanced τ²-bench does not primarily improve scoring granularity; instead, it advances the benchmark in two other areas. First, the Dual-Control Environment: the Agent is no longer the only party that can call tools — the user simulator can operate on the same shared environment (the Agent instructs the user to switch to airplane mode, and the user’s action actually changes the environment state), which better matches real scenarios like technical support, where the user must lend a hand. Second, more precise task specifications and compositional task generation: fewer ambiguities in success conditions, and task instances that can be parameterized and generated in batches (see the “Verifiability and Objectivity Assurance” section later for detailed verification dimensions).

Tool-calling evaluation asks whether an observable state change was completed; human-computer interaction evaluation asks whether the Agent helped the user reach a new understanding or make a decision. The former tests the correctness of the Agent’s actions; the latter tests the soundness of its communication strategy.

Building evaluation environments also touches on simulation environments—when an evaluation environment must support repeated interactions at scale, it becomes a simulation environment. The end of this chapter takes this up briefly.

Related reading

• Methods and Decisions — LLM-as-a-judge, model selection, statistical significance
• Observability and Production Evaluation — turning runs into improvements