The Evolution Loop

The four update methods become continual evolution rather than one-off optimization only when incorporated into the same autonomous loop. Figure 8-5 shows a more robust dual-loop architecture for production systems: the online execution loop only completes tasks and records evidence, without directly rewriting the production Agent; the offline evolution loop aggregates trajectories, diagnoses root causes, generates candidate modifications, and releases new versions only after they pass validation gates. The two loops are connected through versioned experience repositories and evaluation sets.

Voyager demonstrates a relatively complete continual-evolution loop. In Minecraft, it selects new goals based on its current capabilities, iteratively refines programs using environmental feedback, stores successfully validated code in a skill library, and then combines existing skills to solve harder tasks. An automatic curriculum, executable skills, and environmental validation are all indispensable: with a skill library but no curriculum, the Agent does not know what to learn next; with self-reflection but no environmental validation, the skill library accumulates errors; with exploration but no persistence, every task must still begin from scratch. Although the knowledge, Prompt, tools, and parameters of real-world Agents are more complex, the basic learning process is similar.

More specifically, Voyager consists of three interlocking mechanisms. The automatic curriculum generator proposes a suitably challenging next objective from the current inventory, environment, and acquired skills, so exploration does not become random wandering. The skill library stores successful programs as retrievable, composable code; an advanced gathering skill, for example, can invoke basic movement and crafting skills. The iterative prompting mechanism feeds environmental observations, execution errors, and self-verification results back into the next round of code generation until the task actually passes. Compared with the baselines used in the paper, Voyager obtained 3.3× as many unique items, traveled 2.3× as far, unlocked key technology-tree milestones up to 15.3× faster, and transferred its skill library to new Minecraft worlds. These metrics measure how capability grows with experience rather than how a frozen Agent performs on a single examination.

From Problem Diagnosis to Experience Consolidation

The same surface-level problem may require different forms of modification. When a customer-service Agent hallucinates fabricated facts, the cause may be missing information in the knowledge base, or the Prompt may fail to require citations. When an Agent falsely promises “it has been completed” before completing a task, the problem can be corrected through instructions or by having the Harness enforce consistency between the response and tool state. The evolution module should first identify the root cause and then select the smallest modification target that is easiest to validate and roll back. Sporadic failures with insufficient evidence should not immediately trigger learning; the system should continue collecting examples instead.

This choice may also change as experience accumulates. A newly discovered strategy can initially be stored as an experience document for retrieval; after repeated validation across multiple cases, it can be promoted to knowledge. Knowledge can be expressed in three ways: rules that can be clearly described in natural language can be consolidated into a Skill; stable procedures that require no natural-language understanding can be compiled into tool code; and capabilities that actually reflect broad, implicit decision-making can enter post-training.

Validation, Release, and Rollback

Every modification should first produce a candidate capability or candidate Agent rather than directly overwrite the production version. Knowledge documents must be tested to determine whether retrieval improves performance on new tasks; Prompts and Skills must be checked against edge cases and for regressions on previous tasks; programs must be tested in sandboxes and reset environments; and parameter updates must be evaluated for forgetting, safety, and out-of-distribution performance. Even after validation, a new version should be released gradually and monitored on real traffic; if key metrics deteriorate, the system should automatically roll back to a known safe version.

Validation must also separate two capabilities that are often conflated. Harness updating is the ability to produce valuable persistent changes from trajectories; Harness benefit is the task Agent’s ability to find, activate, and correctly use those changes later. A Skill may be correct in itself, yet a weaker task model may fail to load it in the right situation or fail to follow it over a long trajectory. Either failure makes the final score look as if no evolution occurred. End-to-end performance alone therefore cannot diagnose the updater. Model-swapping experiments by Lin et al. indicate that the two abilities relate differently to base-model capability. The exact relationship requires validation on more tasks, but evaluating them separately is broadly useful.

Table 8-3 Layered evaluation metrics for continual evolution

Metric	Question answered	Primary evidence
Candidate-change validity	Does the updater propose useful changes?	Acceptance rate and gain in independent validation
Artifact activation rate	Does the task Agent load the new Skill, memory, or tool in the right situation?	Retrieval, routing, and tool-call traces
Successful adherence rate	After activation, does the Agent follow the new rule or process?	Action sequences and process verifiers
Held-out task gain	Does the whole system improve on tasks not used during evolution?	Held-out success, quality, and cost

For diagnosis, hold a candidate Harness fixed and swap only the task model. If a strong model benefits while a weak model never activates the new artifact, retrieval or routing is the bottleneck. If both activate it but only the strong model executes it correctly, instruction following or long-horizon planning is the bottleneck. If every model regresses, the change itself is more suspect. Conversely, hold the task model fixed and swap the model that proposes changes to compare updater quality directly. This two-way model swap locates where capability budget should be spent more effectively than a single post-evolution score.

Evaluation is not an examination performed after learning ends, but an indispensable part of self-evolution. Long-term evaluation should observe at least five types of outcomes simultaneously:

Regression, namely whether new experience conflicts with other existing experience and whether previously successful cases begin to fail;
Generalization, namely the improvements produced by new experience in scenarios not yet covered by the test set;
Token efficiency, namely the token cost of completing tasks;
Safety, namely whether rules, privacy protections, and refusal boundaries drift during evolution;
Long-term engineering quality, namely whether maintenance complexity, architectural consistency, ownership boundaries, backward compatibility, and future migration and debugging costs deteriorate.

Fixing only the current failed case while degrading performance on other existing cases or in new domains does not constitute successful continual learning.

The Boundary of a Verifiable Loop: When “Done” Does Not Mean “Progress”

The preceding loop works most naturally for Coding, tool use, and business-state changes, where tests, environment state, or deterministic rules can provide rapid feedback. Open-ended research, strategic planning, and complex product design are different: feedback is delayed, there may be no unique correct answer, and the objectives that matter most—research taste, long-term value, and maintainability—are difficult to turn into an immediate score. A Harness can then execute the process flawlessly while merely producing things that look like results rather than advancing the real objective.

Autonomous research is a useful stress test. Trehan and Chopra documented four end-to-end attempts to turn research ideas into papers. Three failed during implementation or evaluation, and only one completed the full pipeline. The failures fall into three groups. First, implementation drift: once the proposed method becomes difficult, the Agent retreats toward a familiar implementation from its training distribution that no longer tests the original hypothesis. Second, epistemic over-optimism: while the signal may still be noise, the system begins explaining it, patching the method, and announcing a finding, while failures and negative results are more easily ignored. Third, missing tacit judgment: an Agent may be able to run experiments without knowing which baseline matters, which anomaly deserves investigation, or when a hypothesis should be abandoned.

These tasks require changes to the evidence and supervision structure, not merely a model that writes better papers:

Separate claims from evidence: Record provenance separately for citations, numbers, methods, and conclusions; the final document is only one rendering of the evidence graph. ScientistOne’s Chain-of-Evidence design links each class of claim to auditable sources. This improves traceability but does not by itself make the research question valuable.
Retain negative results: Write failed experiments, rejected candidates, and stopping reasons to an immutable log with the same retrieval status as successes. Otherwise the evolution module sees only survivors, revisits disproved paths, and learns to interpret ambiguous results as success.
Preserve search diversity: Open-ended search should not retain only the currently highest-scoring chain. The candidate pool should also preserve some lower-scoring but meaningfully different branches by mechanism, code novelty, or hypothesis type, so that every solution does not converge on the same easy-to-score template.
Move human involvement upward: Human input is not limited to approving dangerous tool calls. It also includes defining problems, reviewing evaluation criteria, interpreting anomalous results, and deciding when to stop. With ambiguous feedback, these high-level judgments are harder to automate—and more valuable—than taking over individual execution steps.

The same limitation appears in ordinary software engineering. Passing every unit test proves only that currently observable behavior satisfies the tests; it does not prove that the codebase will remain maintainable months later. That is why the previous section treats long-term engineering quality as an independent metric rather than expecting present task success to cover delayed externalities. The ceiling of continual evolution is ultimately set by whether the system can evaluate what it actually cares about, not merely the easiest proxy to measure.

Safety Boundaries for Continual Evolution

An Agent’s self-evolution capability can turn a single error into a long-term risk. If Prompt injection in web pages, email, or tool output is summarized as experience, it may take effect repeatedly across sessions. If a malicious package found through automated search is wrapped as a tool, its impact can spread from one sandbox run to every subsequent task. A defective verifier may also continue approving candidates that appear to improve but actually regress. An Agent self-evolution system must therefore ask not only whether a candidate is stronger, but also who may modify what and what evidence justifies the change.

The first boundary is separating evidence from instructions. Raw web pages and tool output are untrusted evidence and must not be written directly into a Skill or similar capability; an LLM must first summarize them. Writes should be version-controlled and submitted as pull requests, which are merged only after review by a reviewer LLM from a different source.

The second boundary is separating candidate capabilities from production capabilities. New knowledge, Prompts, Skills, programs, and parameters first enter a candidate area that cannot serve real traffic. Newly generated code and external dependencies must also pass security checks such as sandbox execution, permission review, supply-chain scanning, and behavioral testing. Only after security checks and regression tests pass may a candidate serve real traffic as a production capability.

The third boundary is that safety mechanisms must not be self-modifiable. A business Agent may modify Prompts, Skills, the knowledge base, and tools, but it must not modify the validators, test cases, release thresholds, audit logs, or stable-version backups that approve its own updates. Otherwise, an Agent can disguise regression as progress simply by lowering a test threshold or deleting failing cases.

Sleep Learning: Consolidation, Forgetting, and Capability Maintenance

“Sleep learning” is a cognitive analogy for offline consolidation; it does not require the process to run literally at night. The online Agent’s primary responsibility is to complete the current task and append immutable evidence. A background learning process reads a batch of new experience during idle periods or when gating conditions are met, compares old and new conclusions, merges duplicates, resolves conflicts, proposes candidate updates, and runs regressions. Separating collection from organization prevents an accidental success, network failure, or malicious input from immediately rewriting long-term capabilities, and it allows consolidation to use larger batches and cheaper models.

A typical sleep-learning cycle has five steps:

Trigger: Reach a threshold for elapsed time, number of new trajectories, storage use, or error frequency, while confirming that no high-priority online task is running.
Orient: Read the production knowledge, Prompt, and Skill directories and their versions to understand existing capabilities and immutable boundaries.
Collect and consolidate: Find new signals in recently evaluated trajectories, merge duplicates, mark conflicts and applicability conditions, and prefer local patches.
Validate and approve: Evaluate candidates on transfer, retention, and safety sets; high-risk writes wait for human approval.
Prune and index: Update retrieval indexes and mark capabilities that are long unused or contradicted by new evidence as expired, archived, or deleted, while retaining provenance and rollback versions.

User memory is the most intuitive example, but it must be distinguished from action experience. Claude Code’s auto memory maintains a MEMORY.md index and topic-specific detail files for each project. At session startup it loads only a bounded prefix of the index and reads the remaining content on demand; when the index approaches its limit, the Agent is instructed to merge or move details elsewhere. This shows that even plain-text memory requires capacity limits, layered loading, and active organization. The currently documented mechanism primarily writes memory during sessions and should not simply be equated with a fixed nightly background task.

Hermes provides a more complete example of background evolution. It separates long-term information into bounded MEMORY.md and USER.md files, SQLite/FTS5 search over prior sessions, on-demand Skills, and optional external memory providers such as Honcho. Session search returns original messages rather than first summarizing them with an LLM, keeping retrieval distinct from generation and auditable. When a task contains many tool calls, recovers from an error or dead end, receives a user correction, or discovers a non-obvious workflow, a background review can create or locally revise a Skill; memory and Skill writes can also pass through an approval gate. A separate Curator tracks Skill usage, staleness, and archival status, performs deterministic pruning while idle, and may optionally invoke an LLM to merge content. It snapshots changes first so that incorrect consolidation can be rolled back. This turns “record–consolidate–validate–prune” from a metaphor into an operational capability lifecycle.

Continual evolution does not mean allowing knowledge, Prompts, and tools to grow without limit. The context corruption discussed in Chapter 2 reappears over longer timescales: experience documents conflict with one another, Prompts become overwhelmed by boundary rules, Skill libraries accumulate duplicate capabilities, and repeated fine-tuning causes catastrophic forgetting. The system therefore requires periodic offline consolidation:

Merge duplicate experience while retaining provenance and version information;
Move local rules from the global Prompt into domain-specific Skills to keep the global Prompt clean;
Keep Prompts and Skills clearly structured, like a handbook for new employees, and avoid enumerations resembling “99 ironclad rules.”
Revalidate tools that have not been used for a long time;
Delete knowledge invalidated by new evidence;
Retrain LoRA from the original base model.

Learning Signals — the first two links of the loop
Observability and Production Evaluation — the trajectory capture the offline loop depends on