API-Level Context Structure

Everything the model sees arrives as one flat message list of four roles — system, user, assistant, tool. The agent's core loop is mechanical — build the request, dispatch the tool calls the model returns, append results, call again, until it stops asking for tools.

This section uses OpenAI’s Chat Completions API as a concrete example. Anthropic, Google, and other providers differ in details, but their Agent-facing APIs follow a similar pattern: each model call is constructed from a structured conversation history plus a set of available tool definitions. Understanding this structure is the foundation for the context engineering techniques discussed later in this chapter.

The Four Message Roles

In Chat Completions-style APIs, the core input is a message list, usually named messages. Each message has a role field that tells the model how to interpret the message and where it came from:

  • system: Developer-written instructions that define the Agent’s identity, behavior, constraints, and workflow. The model treats this as a high-priority instruction. In most conversations, the system message appears once at the beginning of the message list.
  • user: Input from the end user, representing the request the Agent needs to handle.
  • assistant: Previous model outputs, including natural-language replies and tool call requests. In multi-turn interactions, these messages are included in later requests so the next stateless model call has access to the prior trajectory.
  • tool: Results returned after the Agent framework executes a tool. Each tool result is linked to the corresponding tool call through tool_call_id, allowing the model to associate each result with the request that produced it.

Tool definitions are not messages. They are provided in a separate tools field, which declares the tools available to the model and specifies the parameters each tool accepts.

Single-Turn Request: The Simplest API Call

Start with the simplest case: a single request without tool calls. The user asks, “Hello, who are you?” The example uses a locally deployed Qwen3-0.6B model, connecting it to the local LLM deployment experiment later in this section. The timestamps in the example are for demonstration only and are unrelated to the book’s timeline.

// ═══ Request constructed by the Agent framework ═══
{
  "model": "Qwen3-0.6B",
  "messages": [
    {
      "role": "system",                           // ← Written by developer
      "content": "You are a helpful coding assistant. Follow user instructions."
    },
    {
      "role": "user",                              // ← User input
      "content": "Hello, who are you?"
    }
  ]
}
// ═══ Response returned by the API ═══
{
  "choices": [{
    "message": {
      "role": "assistant",                         // ← Generated by model
      "content": "Hi! I'm a coding assistant. I can help you write code, debug issues, and explain technical concepts. How can I help?"
    }
  }]
}

This request contains only two messages: one system message containing rules written by the developer and one user message containing the user’s input. The model returns an assistant message as the reply. This is the most basic LLM API interaction pattern: each call is stateless, so the request’s message list must contain all the information the model needs.

Multi-Turn Interaction with Tool Calls: The Core Loop of an Agent

Real Agent workflows are usually more complex than a single-turn Q&A. When a user asks, “What’s the current time and weather in Vancouver?”, the model needs access to dynamic external information: the current time and the latest weather. The following example walks through each interaction between the Agent framework and the model.

First API call — Agent framework sends the initial request:

// ═══ Request constructed by the Agent framework (1st call) ═══
{
  "model": "Qwen3-0.6B",
  "messages": [
    {
      "role": "system",                           // ← Written by developer
      "content": "You are a helpful assistant. Use the provided tools to get real-time information when needed."
    },
    {
      "role": "user",                              // ← User input
      "content": "What's the current time and weather in Vancouver?"
    }
  ],
  "tools": [                                       // ← Tools defined by developer
    {
      "type": "function",
      "function": {
        "name": "get_current_time",
        "description": "Get the current date and time in a specific timezone",
        "parameters": {
          "type": "object",
          "properties": {
            "timezone": { "type": "string", "description": "Timezone name, e.g. America/Vancouver" }
          }
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a specific city",
        "parameters": {
          "type": "object",
          "properties": {
            "city": { "type": "string", "description": "City name" },
            "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
          }
        }
      }
    }
  ]
}

Model returns a tool call request (not a final reply):

// ═══ Response returned by the API (model decides to call tools) ═══
{
  "choices": [{
    "message": {
      "role": "assistant",                         // ← Generated by model
      "content": null,                             // No text response
      "tool_calls": [                              // Model requests two tool calls
        {
          "id": "call_abc123",
          "type": "function",
          "function": {
            "name": "get_current_time",
            "arguments": "{\"timezone\": \"America/Vancouver\"}"
          }
        },
        {
          "id": "call_def456",
          "type": "function",
          "function": {
            "name": "get_weather",
            "arguments": "{\"city\": \"Vancouver\", \"unit\": \"celsius\"}"
          }
        }
      ]
    }
  }]
}

The model does not answer the user’s question yet. Instead, it returns two tool call requests: one for the current time and one for the weather. Because these requests are independent, the Agent framework can execute them in parallel. The model issues the call requests; the Agent framework performs the actual execution. This division of responsibility is central to Agent architecture: the model decides which tool to call and what arguments to pass, while the framework calls APIs, runs code, and returns the results.

The Agent framework executes the tools and then initiates a second API call:

After receiving the model’s tool call requests, the Agent framework executes the two tools (for example, by calling a time API and a weather API), then sends the complete conversation history along with the tool execution results back to the model:

// ═══ Request constructed by the Agent framework (2nd call) ═══
{
  "model": "Qwen3-0.6B",
  "messages": [
    {
      "role": "system",                           // ← Same as 1st call
      "content": "You are a helpful assistant. Use the provided tools to get real-time information when needed."
    },
    {
      "role": "user",                              // ← Same as 1st call
      "content": "What's the current time and weather in Vancouver?"
    },
    {
      "role": "assistant",                         // ← Model output from 1st call, included verbatim
      "content": null,
      "tool_calls": [
        { "id": "call_abc123", "function": { "name": "get_current_time", "arguments": "{\"timezone\": \"America/Vancouver\"}" } },
        { "id": "call_def456", "function": { "name": "get_weather", "arguments": "{\"city\": \"Vancouver\", \"unit\": \"celsius\"}" } }
      ]
    },
    {
      "role": "tool",                              // ← Generated by Agent framework (tool execution result)
      "tool_call_id": "call_abc123",
      "content": "{\"timezone\": \"America/Vancouver\", \"datetime\": \"2025-09-13T05:18:47\", \"day_of_week\": \"Saturday\"}"
    },
    {
      "role": "tool",                              // ← Generated by Agent framework (tool execution result)
      "tool_call_id": "call_def456",
      "content": "{\"city\": \"Vancouver\", \"temperature\": 13.2, \"unit\": \"celsius\", \"conditions\": \"clear\", \"humidity\": 93}"
    }
  ],
  "tools": [ ... ]                                 // ← Same tool definitions as above, omitted
}

There are three key details here:

  1. The second request includes the full conversation history from the first request — the system message, the user message, the assistant message containing tool calls, and the newly added tool results. This illustrates the stateless nature of the API: the Agent framework must include the relevant history in every request.
  2. The first assistant message is inserted back into the message list verbatim — this gives the next model call access to the tool-call decisions made in the previous call.
  3. Tool messages are linked to their corresponding tool calls via tool_call_id — this tells the model which result belongs to which requested call.

The model generates the final response based on the tool results:

// ═══ Response returned by the API (final reply) ═══
{
  "choices": [{
    "message": {
      "role": "assistant",                         // ← Generated by model
      "content": "It's currently 5:18 AM on Saturday, September 13, 2025 in Vancouver.\n\nWeather: 13.2°C with clear skies and 93% humidity. It's quite cool this morning - you might want to grab a jacket."
    }
  }]
}

This time, the model does not return tool_calls; it returns a text response because the tool results provide enough information to answer the user’s question. If more information is needed (for example, if the user asks “What about Tokyo?”), the model can return tool_calls again, and the Agent framework repeats the same cycle: execute the tools, send back the results, and call the model again. This “request → tool call → execution → return results → next request” cycle is the API-level implementation of the ReAct loop introduced in Chapter 1.

How Context Is Composed at the API Level

The example above shows the complete composition of context each time the Agent calls the model:

The upper part (System Prompt + Tool Definitions) remains unchanged throughout the conversation, while the lower part (conversation history, i.e., the trajectory defined in Chapter 1) grows with each interaction. This is how the five context components from Chapter 1 appear at the API level: the system prompt and tool definitions form a static prefix, while user messages, model replies, and tool execution results form a dynamically growing message history. This “static prefix + trajectory” structure is the foundation for later discussions of KV Cache optimization, context compression, and related techniques: the prefix should remain stable, while later trajectory segments can be summarized or replaced when the trade-off is worthwhile.

The rest of this chapter examines each layer of this structure: how to use a stable static prefix to accelerate inference (KV Cache), how to design an effective System Prompt (prompt engineering), how to prevent external content from hijacking the context (prompt injection defense), how to load specialized knowledge on demand (Agent Skills), how to inject dynamic state at the end of the conversation (Agent Status Bar), and how to compress conversation history when it grows too large (compression strategies).

Engineering Practice

Zapvol’s “framework executes, model decides” loop is runAgentLoop(): each round it builds the request, dispatches the tool_calls the model returns, appends the tool results to the message list, and starts the next round — until the model stops requesting tools. The “stateless” property lands as crash recoverability: the loop holds no state that needs persisting, so after a process restart, replaying the message record (TaskRepository.getMessages()) returns it to where it stopped. This is also why the static prefix is never rewritten mid-stream and dynamic content is only appended at the end — both a KV Cache requirement and a precondition for replay.

Was this page helpful?
Edit on GitHub