Observability and Production Evaluation

Agent Observability

Evaluation-driven decisions (whether for model selection or continuous iteration) rely on high-quality operational data. Below, we first introduce how to systematically collect this data (observability), and then discuss how to translate evaluation results into system improvements.

Observability is a concept borrowed from distributed systems: you cannot open the system and watch it work; you infer what is happening from the logs, metrics, and traces it emits—the way a doctor, unable to see inside a patient, diagnoses from temperature, blood pressure, and imaging. Agent systems make this harder still: the same input can produce different outputs, multi-round reasoning and tool calls make execution paths extremely complex, and the model’s “thinking” is completely opaque from outside.

The value of observability lies first in problem diagnosis: complete traces allow developers to replay the entire process rather than guessing. Second, it is the foundation for continuous optimization—you can see which tasks require multiple rounds of iteration, which tools have the lowest success rate, and which retrieval queries always return empty results. In cost management, Agent operating costs can differ by one or two orders of magnitude between tasks, and tracing surfaces the abnormally expensive cases. Finally, accumulated trace data underpins later system optimization and model improvement.

Agent observability is built on the foundation of traces, whose data structure directly inherits the span tree model from distributed systems: one task execution corresponds to one trace, where each LLM call, each tool call, and each retrieval is a span (an execution unit recording input/output, start/end times, token consumption, and error information). The parent-child relationships between spans form an execution tree—for example, an “Agent Main Loop” span may have several “LLM Call” and “Tool Call” child spans hanging beneath it. Standardized protocols are already available for this layer: OpenTelemetry is the general-purpose distributed tracing standard, while specifications like OpenInference define LLM-specific semantic conventions on top of it (how to record prompts, model parameters, token usage, etc.). The advantage of adopting standard protocols is the decoupling of collection and analysis—the same trace data can be connected to different analysis backends, avoiding vendor lock-in.

LangSmith is one of the representative platforms in this domain (similar platforms include Langfuse, Arize Phoenix, etc.), integrating observability, evaluation, and optimization into a closed loop. Each execution creates a trace session, where model calls, tool usage, and knowledge retrieval are recorded as independent execution units, linked by causal relationships to form an execution tree. Each unit records complete input/output, timing information, cost data, and error information. The platform uses asynchronous batch data collection to ensure that tracing itself does not affect the Agent’s response latency.

The platform also supports A/B testing (routing a portion of user traffic to a new version, automatically comparing metrics, and supporting rapid rollback or gradual scaling), prompt version management (each version is associated with runtime performance data), and collaborative development (team members can share trace data and problem cases). The massive amount of real-world data from production environments is a goldmine for continuous improvement—it can uncover unforeseen scenarios and identify the features most in need of optimization.

The most valuable use of observability data is to turn it into evaluation assets. A practical loop: extract failed and suspicious cases from production traces → anonymize them (strip sensitive fields such as user data and keys) → distill them into new test cases and regression tests for the evaluation set. The evaluation set then stops being a one-time, static collection and becomes a living asset that evolves with the product and continues to reflect the real user distribution—the failure patterns exposed in production today become the regression tests guarding the baseline tomorrow. This is precisely the interface between observability and the main theme of this chapter: observability is responsible for “seeing” what happens in the real world, and evaluation is responsible for solidifying those observations into repeatable standards.

Observability faces several challenges:

• Trade-off between data volume and privacy: High-traffic systems can generate terabytes of trace data daily, while also needing to comply with data protection regulations.
• Complexity of causal attribution: Automatically identifying root causes from traces still requires more intelligent analysis algorithms; cutting-edge research is attempting causal inference and counterfactual analysis, but it is not yet mature.
• Tracing challenges in multi-Agent systems: Tracing execution flows across multiple Agents is more complex and semantically richer than tracing API calls between microservices.
• Balance between real-time guardrails and post-hoc analysis: High-risk scenarios require proactive guardrails, but these introduce additional latency and false positives.

As ML technology becomes more deeply integrated into the toolchain, future observability platforms are expected to automatically identify anomalies and pinpoint root causes.

With a comprehensive evaluation system and dataset in place, the key is to translate evaluation results into tangible system improvements.

From Benchmark Reports to System Improvements

The following is a hypothetical teaching case, using specific data to illustrate the complete decision-making process from a benchmark report to system improvements. The data is hypothetical and aims to demonstrate the methodology, not to report real experimental results.

From the perspective of Harness engineering, this section is essentially about the methodology for iterative Harness optimization—using evaluation data to identify weak points in the Harness (insufficient context? missing constraints? inadequate validation? untimely feedback?), making targeted improvements, and then re-evaluating, forming a closed loop for the Harness’s continuous evolution.

Before analyzing any benchmark report, note an easily overlooked principle: when Agent performance drops, check the evaluation system first, then the Agent. The common mistake is to start editing Agent code the moment a score falls, ignoring the possibility that the evaluation system broke first—steer by a distorted signal and the correction is wrong from the very first step. Typical evaluation-side failures include: the runtime environment running out of resources and killing processes (which shows up as random failures), bugs in the scorer that mark correct answers as failures, and test cases drifting out of sync with production scenarios. In the headline numbers, all of these look identical to model degradation; only a review of the full traces can tell them apart.

Reading a Benchmark Report: The Art of Problem Discovery

Let’s use a specific case to illustrate how to read a benchmark report. Suppose we evaluate an Agent on AndroidWorld and obtain two core report tables: a per-task performance table and a capability-tag performance matrix. The report’s value lies not in the single overall success-rate figure, but in the structural weaknesses it reveals.

The per-task table shows a clear pattern: most routine tasks have success rates close to 100%. These cover common scenarios—recording, taking photos, contact management, note creation, file operations, system settings—and require a dozen-plus steps on average, with the most complex ones requiring dozens. Successfully completing action sequences this long demonstrates the Agent’s planning and execution ability in standard scenarios.

Failures cluster tightly in a few areas: SMS replies, Wi-Fi toggling and status verification, to-do list queries, combined Wi-Fi+Bluetooth operations, and VLC playlist creation. On the surface these tasks look unrelated; the capability tag matrix reveals what they share.

The capability tag matrix is key to diagnosis—it cross-classifies all tasks by required capabilities and difficulty. The report often shows several capability dimensions with extremely low success rates: transcription (transcribing information from images/videos, exposing deficiencies in visual understanding), math_counting (the problem is not the math ability itself—modern LLMs are strong at math—but whether the Agent can recognize the need for calculation, extract numbers from the UI, and map the result to an action sequence), and complex_ui_understanding (heavily reliant on standard UI patterns, collapsing when encountering non-standard layouts).

Read the two tables together and the failures explain themselves: the to-do query failures trace to a non-standard UI the Agent cannot read and filter; the Wi-Fi failures trace to a control hierarchy in system settings that exceeds the Agent’s understanding; the VLC playlist failures trace to the Agent being unable to find the creation entry in a professional application’s complex UI.

From Data to Hypotheses: Building an Improvement Roadmap

Surface-level hypotheses (low cost, independent, can be verified in parallel): H1: Add system settings navigation hints for Wi-Fi operations (the Agent might be able to operate the toggle but cannot find the entry page), expected to resolve the concentrated failures in settings-related tasks; H2: Provide UI element identification rules for the to-do app, expected to resolve failures in to-do tasks.

Mid-level hypotheses (also independent, can be parallelized): H3: Fix the multimodal input pipeline—replaying failed traces reveals that images might be dropped or converted to text descriptions in the pipeline, rendering even the strongest multimodal models unable to transcribe; H4: Globally enable thinking to resolve counting-related failures.

Deep-level hypotheses (high verification cost, only initiated if complex_ui success rate remains below 40% after surface and mid-level improvements): H5: Replace the model with one having stronger visual understanding (GPT-5); H6: Add UI element tree information beyond screenshots (structured DOM extracted by UI Automator for cross-validation with screenshots). These two can form a 2×2 comparative experiment (Claude/GPT-5 × screenshots only/screenshots + element tree) to answer “which is more critical, model capability or information richness, and is there a synergistic effect?”

Each configuration is run 5 times on the full set of 116 tasks (using different random seeds), recording success rate, average steps, and execution time.

From Results to Decisions: Data-Driven Trade-offs

Assume the experimental data shows the following results (all data below is hypothetical): H1 raises the success rate on settings-related tasks from 0% to 75%, with an 8% increase in input tokens; H3 improves transcription from 0% to 80%, with a 15% increase in vision tokens and a 1-second increase in latency per step; H4 improves counting from 0% to 70%, but latency per step increases from 4 seconds to 12 seconds, and cost triples; H6 improves complex_ui from 17% to 52%, with a 30% increase in tokens and a 2-second increase in latency per step; H5 (GPT-5) improves complex_ui from 17% to 35%, but latency per step increases from 4 seconds to 15 seconds.

The decision is not simply to adopt all effective improvements:

Deploy H1 and H3 immediately: H1 is low-cost and high-benefit, with no side effects. H3 adds 15% to vision-token costs and one second of latency, but it turns transcription from a non-functioning capability into a working one, and it fixes an architectural defect—the input pipeline dropping multimodal information—which may lift other visual understanding tasks along the way.

Globally enabling thinking for H4 is unacceptable: overall success does rise from 88% to 91%, but the capability tag distribution shows only about 8% of tasks involve counting—forcing every task to incur three times the latency and cost for the sake of a minority is a classic case of using a sledgehammer to crack a nut. H4 does prove, however, that thinking works for counting tasks, laying the ground for conditional activation in the next round.

H6 beats H5: with H5 (GPT-5), latency per step rockets from 4 seconds to 15 while complex_ui only reaches 35%—the bottleneck is not the model’s reasoning but whether the input carries enough information. H6 (adding the element tree) buys a 35-percentage-point improvement for 30% more tokens and 2 seconds of latency—a far better bargain. The H5+H6 combination scores highest (68%), but its task duration is unacceptable at scale; it suits only selective activation on critical asynchronous tasks (bank transfers, medical appointments), while H6 suffices for everyday scenarios.

H2 doesn’t scale: writing bespoke rules for every non-standard application is unsustainable. It can only be a stopgap; the long-term solution is to improve the Agent’s ability to generalize.

Continuous Iteration: From First Improvement to System Evolution

After implementing the three improvements H1, H3, and H6 (H4 not deployed), the Agent’s success rate on AndroidWorld rises from 88% to 94%. Rerunning the full benchmark, the new report reveals a different failure pattern: transcription, settings, and complex UI tasks have all improved significantly. The remaining failure rate, about 6%, is concentrated in unresolved counting tasks, unstable Wi-Fi status verification (up from 0% to 60% but still unstable), and a handful of new failures, possibly caused by longer prompts or too much element-tree information distracting the model.

Based on the new report and insights from the H4 experiment, new hypotheses can be formed. H7: Conditional activation of thinking—use a quick LLM call (about 1-2 seconds) before a task starts to analyze the task description, enabling thinking mode only for tasks involving counting or complex reasoning, thus confining the latency increase to tasks that truly need it. H8: Expand the action space to support complex gestures (pinch-to-zoom, long-press drag, multi-touch)—replaying the remaining failed traces reveals that some tasks require operations like map zooming, image cropping, and long-press menus on lists.

This kind of iteration based on benchmark feedback steadily improves the Agent’s capabilities. A benchmark is not a one-time exam but a continuous health check. A regular evaluation cadence (say, the full test suite weekly) lets you watch the capability curve, catch regressions early (a new feature introducing bugs), confirm improvements (the optimization really worked), and accumulate knowledge (which kinds of improvements usually pay off, which tend to backfire). This methodology—data-driven, hypothesis-tested, continuously iterated—is the key path from experience-driven Agent engineering to scientific engineering.

From External Evaluation to Internal Evaluation: Evaluation Infrastructure for Production-Grade Agents

So far this chapter has evaluated Agent systems from the outside—building an evaluation environment, designing datasets, analyzing benchmark reports. But the best Agent products do more than undergo external evaluation; they build continuous self-evaluation infrastructure into the product. Below, using the open-source general-purpose Agent OpenClaw introduced in Chapter 5 as an example and drawing on public technical analyses of leading Coding Agent products and practitioner insights, we present an internal evaluation system worth emulating: one that systematically embeds the experimental methodology of ML research into product engineering.

Ablation Infrastructure: Understanding the True Contribution of Each Feature

ML researchers have long used ablation studies to learn which components of a model actually matter—ablation means “removing” one component at a time and observing how much overall performance drops. OpenClaw brings this methodology into product engineering: a built-in master switch can disable several major features at once (thinking mode, context compression, automatic memory, background tasks, and more), creating a “bare model” baseline. That lets the team answer a key question: does a feature truly improve the user experience, or does it just feel useful?

Making ablation a routine engineering practice, rather than a one-time research activity, has several practical implications. First, the ablation switch must be injected very early in the startup path—before any module-level constant captures configuration values—meaning the ablation infrastructure must be designed into the system architecture from the start, not retrofitted later. Second, running ablation experiments regularly (e.g., before each major release) can uncover “feature debt”—features that were once effective but are no longer necessary as models evolve. For any team building a production Agent, the recommended practice is: Every major feature should be independently disableable, and the team should regularly verify the actual contribution of each feature.

A/B Testing Methodology: Distinguishing Mechanism from Goal

Mature Agent products conduct rigorous A/B testing on their own behavior (i.e., randomly dividing users into two groups, one using the old version and one using the new version, and comparing actual data from both groups to determine if a change is effective). A well-designed Agent A/B test case illustrates several key methodological principles:

Multiple variants, not just a binary comparison. Instead of just comparing “with” and “without,” design multiple progressive variants (e.g., when testing different strengths of prompt constraints, set up a control group and three experimental groups with progressively stricter constraints). This design can reveal dose-response relationships and help find the optimal point.

Distinguishing mechanism metrics from target metrics. This is the easiest mistake to make—treating what you are changing as the optimization target. For example, if you are testing “shortening the Agent’s plan file length,” plan length is a mechanism metric (something you directly change), but it is not the target. The real target might be “reducing session-level cost.” Shortening the plan file may lower costs, but it could also lead to more edit-check-edit loops due to insufficiently detailed plans, increasing total output. Always ask yourself: Is what I am changing (the mechanism) the same as what I truly care about (the target)? If not, prioritize the target.

Setting guardrail metrics. Even if the target metric improves, the experiment should be stopped if user satisfaction declines, the number of operations increases, or the error rate rises. Guardrail metrics are non-negotiable thresholds that must not regress.

Recording baseline statistics. Include sample size, distribution percentiles, and correlation analysis (e.g., “rejection rate increases monotonically with plan size”) to provide the necessary context for interpreting experimental results. Without a baseline, you cannot determine whether the experimental results are statistically significant.

Two-Layer Feature Flag System

Agent products need a Feature Flag infrastructure designed from day one—a feature flag is a remotely controllable switch that determines whether a function is enabled or disabled for users, without requiring code redeployment. It serves three purposes simultaneously: experimentation, gradual rollout, and emergency circuit breaking.

Compile-time flags physically remove the relevant code from the build artifact during the build phase. Internal-only features simply do not exist in external builds—even reverse engineering cannot discover the removed functionality. This also provides a clean ablation mechanism: disabling a feature does not skip logic at runtime; the corresponding code is physically absent.

Runtime flags have their configuration delivered by the server and cached locally on disk. The design prioritizes reading slightly stale cached configuration over blocking the Agent’s startup while waiting for a network request. Specific grouping decisions are made through an experimentation platform (e.g., GrowthBook) for assigning A/B test groups. A key design detail is that each feature’s exposure event is logged at most once per session to avoid duplicate records polluting the experimental data.

The lesson for Agent developers: feature flags are not debugging tools; they are first-class architectural components.

Prompt Sensitivity Assessment

The system prompt is the core “code” of Agent behavior, yet it often lacks the version control and regression testing afforded to regular code. OpenClaw’s approach is to provide a dedicated tool that can extract the fully rendered system prompt at a specified Git revision or commit—including the final text after all dynamic conditions are expanded. This allows the team to precisely answer: Which commit changed the prompt? What was the impact on the evaluation set?

For any Agent team, the recommended practices are: (1) The system prompt should be deterministically renderable (given the same configuration input, it always produces the same output); (2) Establish a versioned snapshot mechanism for prompts; (3) Every prompt change should run regression tests on the evaluation set—just as code changes require CI.

Privacy-Aware Analytics as an Evaluation Foundation

Evaluation relies on good data, but Agent products often handle sensitive user content. OpenClaw resolves this contradiction through a type system: the analytics interface only accepts values wrapped in special types, where the type name itself serves as an audit trail—it explicitly declares “I have verified this is not code or a file path.” This design transforms privacy constraints from documented specifications into compile-time enforced type checks.

The core principle is: Design privacy constraints into the system from the start; do not bolt them on afterward. If your analytics system cannot safely collect data, you cannot evaluate effectively. Privacy and evaluation are not opposing forces—privacy-aware design forces you to think carefully about what truly needs to be measured, which in turn fosters more precise evaluation metrics.

From External to Internal: A Shift in Evaluation Thinking

The core message of this section is: The previous sections taught you how to evaluate an Agent externally; this section reveals how the best Agent products evaluate themselves internally. External evaluation tells you “how good the Agent is”; internal evaluation infrastructure tells you “which change made it better.” Ablation experiments discover which features truly matter, A/B testing quantifies the impact of each change, feature flags provide the infrastructure for experimentation and rollback, prompt sensitivity assessment integrates the system prompt into the CI system, and privacy-aware analytics ensures compliance in data collection. These five components together constitute evaluation-driven product engineering—not evaluating occasionally, but embedding evaluation into every product decision.

Simulation Environments: The Bridge from Evaluation to Post-Training

The endpoint of evaluation is not scoring, but improvement. This chapter has already demonstrated two paths for improvement: adjusting the Harness (from Benchmark reports to system improvements) and embedding evaluation into product engineering (internal evaluation infrastructure). The strongest form of improvement is training—when the goal expands from “evaluating existing capabilities” to “cultivating new capabilities,” especially through the post-training techniques discussed in Chapter 7, the evaluation environment needs to evolve into a simulation environment: a virtual playground where the Agent can repeatedly practice and be automatically scored. The core differences between simulation environments and evaluation environments are: much higher interaction frequency (millions vs. thousands), the need for randomization (to prevent memorizing specific configurations), and the requirement for immediate feedback. From an application perspective, simulation environments are divided into two categories: digital environments (information processing tasks) and embodied environments (physical world perception and manipulation).

Here is how the two ends of the bridge meet. Assets accumulated on the evaluation side convert almost seamlessly into training signals: a well-defined Rubric or validator is essentially a reward function for Reinforcement Learning with Verifiable Rewards (RLVR)—the scoring script becomes the reward script; whether a test passes or a state meets the standard serves both as an evaluation criterion and as a reinforcement learning reward. But training brings demands evaluation never had to worry about. The first is reliable reset semantics: training runs millions of episodes (an episode is one complete interaction round from an initial state to task completion), and each episode must be able to reset the environment to a deterministic, clean initial state; otherwise, the gradient signal will be contaminated by residual states from the previous episode. The second is throughput far exceeding evaluation: a few thousand evaluations are enough to draw conclusions, but training requires feeding the model millions of interactions within an acceptable wall-clock time; the degree of environment parallelism and per-instance overhead directly determine whether training is feasible. These two points—validators turned into reward functions, and training-grade reset and throughput—will be elaborated in Chapter 7.

On the digital environment side, the AWorld framework builds a controllable MCP server sandbox for GAIA tasks, providing 26 MCP servers covering 126 tool functions, avoiding the bans and uncontrollable side effects of directly accessing real APIs. All tool calls are replayable and auditable. AWorld’s distributed architecture reduces the traditional serial execution time from 7695 seconds to 525 seconds (a 14.6x speedup), and the environment’s stateless design makes each instance completely independent, supporting efficient parallelism.

On the embodied environment side, RoboTwin2 builds dual-arm manipulation tasks based on a physics engine, randomizing object positions, orientations, and appearances to improve generalization. The observation space includes multi-camera visuals and joint states, achieving real-time control through Action Chunking—where the model plans multiple consecutive actions at once (detailed in Chapter 9). OSWorld provides reset capability through virtual machine snapshots, and AndroidWorld focuses on mobile application automation. Whether digital or embodied, simulation environments also require the isolated execution environments and virtual identity mechanisms discussed in Chapter 4 (VM/container isolation, residential proxies, Human-in-the-Loop authentication, shared file systems), which will not be repeated here.

Fidelity Trade-offs and Domain Randomization

High-fidelity environments support better transfer to the real world but have high computational costs. Another dimension of fidelity is the degree of randomization: moderate randomization improves generalization, while excessive randomization can make tasks too difficult. Domain Randomization is a key technique for narrowing the sim-to-real gap: introducing a wide range of random variations in physical parameters, visual appearance, sensor noise, etc.—just like practicing grasping under various lighting and angles, so you won’t fail in the real world just because the light changes. In digital environments, sim-to-real manifests as differences in interface rendering, response times, etc., which can be mitigated by introducing randomization in latency and failures.

With that, the evaluation environment completes its final evolution: from an exam hall that measures ability into a training ground that builds it. Chapter 7 will show how AWorld-train turns such simulation environments into trainable arenas, and the engineering challenges involved—the evaluation system and simulation environments established in this chapter are the two cornerstones of post-training.

Engineering Practice

Zapvol’s observability lands in operations: the trace / log / cost pipeline records every run’s LLM calls, tool calls, tokens, and cost as a replayable execution tree, and runtime-health dashboards watch success rate, latency, and cost anomalies. This trace infrastructure is the first half of book’s “turn runs into improvements” — the capture side is in place; what is missing is the second half, an evaluation harness (replaying production failures into regression cases, running ablations and model-swap experiments), which is Zapvol’s current direction. In short, we can already “see”; we still need to systematically turn “seeing” into “a standard that can be re-checked.”

Related reading

• Methods and Decisions — how run data is scored into decisions