A deep-dive on how to assemble context for an LLM agent

On each model call, a coding agent’s harness chooses what context to send: the full conversation, selected messages, summaries, tool results, and instructions. These choices affect what the model can act on, how much the call costs, and whether earlier work remains available.

We compare four context-assembly strategies on a small bug-fix fixture, using saved runs to inspect their behavior. The experiment illustrates trade-offs rather than establishing a general ranking.

The harness uses pi, whose transformContext hook lets us change the message view before each model call. One strategy keeps recent tool results intact and shortens older ones.

Why context is the agent’s central control problem

A coding agent combines a model, a control loop, and a harness that manages tools and context. The harness determines which information is available on each call, so context handling can change behavior even when the model stays fixed.

The agent’s loop, stripped to its essentials, is:

1. Send the current context to the model.
2. Receive text, tool calls, or both.
3. Execute tool calls and append their results, then repeat.
4. If the response contains no tool calls, finish this agent loop.

Tool output can become a large part of a coding session’s context. Its size depends on the files, commands, and number of calls; thirty turns do not imply a fixed token count. Characters and tokens are also different units.

We therefore focus on handling tool results: retaining them, truncating them, or summarizing older history. These choices interact with message selection and prompt caching.

If we simply leave the history to grow unchecked, we run into four problems:

• Cost: longer prompts can increase input cost; cache discounts and output usage also matter.
• Latency: transmitting and processing a large prompt can delay the response. Cached prefill can reduce some of that work.
• Capacity: the request must fit the selected model’s context limit, including space for output.
• Quality: relevant details can become harder to use in long contexts. The effect depends on the model, task, and placement of information.

A context-assembly strategy isn’t a single decision — it’s a stack of interacting choices:

• What gets dropped or rewritten?
• When — every turn, or only at thresholds?
• Where in the stack — tool layer, strategy layer, or both?
• How aggressively — in tokens, characters, messages?
• What do you summarize — older history, whole conversation, certain tool types?
• How do you summarize — freeform, structured template, multi-round chained?
• What about cache — does your transformation respect the prefix or invalidate it every turn?

We ran three trials per strategy on one small fixture. Reported means and sample standard deviations describe those runs; three trials cannot establish a reliable success rate or the frequency of rare expensive failures.

Read task success alongside cost and prompt size. A cheaper run that fails the task is not necessarily a useful improvement.

A taxonomy of strategies

With the metrics established, let’s look at the design space they’ll be applied to. The strategies that show up in production CLIs fall into a handful of families along two axes: where the compression happens and what gets compressed.

The simplest strategy sends the full conversation on each call. We call it baseline. An unchanged prefix allows cache reuse, subject to the provider’s rules. History still grows toward the context limit, and long-context performance can depend on where relevant information appears.

Every other strategy is a way to implement compression. Compression can happen in three places:

• Per-turn. Apply a transformation to the message list on every LLM call, shaping each turn’s prompt as the conversation grows.
• At thresholds only. Leave the conversation alone until it crosses some size limit, then fire a one-time operation (typically summarization) on the older portion and freeze the result for the rest of the run.
• At the tool level. Cap or rewrite a tool’s return value before it ever enters the conversation history. Lives one layer below the other two and composes with them — we cover it in detail further down.

Per-turn transformations and threshold-triggered compaction can be combined. Deterministic truncation avoids a summarizer call, but may hide needed text. Summarization can shorten much more history, but adds cost and may omit details. Neither approach automatically guarantees good cache reuse.

A sliding window keeps recent messages and drops older ones. It can remove useful history and reduce prefix reuse, but may suit bounded tasks with externally stored state. We do not evaluate it here.

There are several choices about what to remove or condense:

• Drop old turns, optionally retaining the original task.
• Replace old tool outputs with shorter results or retrievable references.
• Cap tool-result text while retaining a truncation marker.
• Summarize older history.
• Retrieve relevant information from a full log as needed. Preserve valid tool-call/result relationships when building the request.

These patterns aren’t mutually exclusive — most production strategies stack two or three, e.g. truncate every tool result, then summarize older turns, then drop pre-summary content.

Cache stability

Prefix caching reuses computation for an unchanged beginning of the model input. Matching rules, minimum sizes, retention, and billing differ by provider. See the OpenAI, Anthropic, and Gemini documentation. The relevant match is the rendered model context, not necessarily the raw HTTP request bytes.

A transformation supports prefix reuse when already-emitted content remains unchanged. Rewriting an older message breaks the match from that point onward for the existing cache entry; a later request may reuse the newly formed prefix.

A simple cache-stable example is a cap on tool results: every turn the strategy walks the conversation with full tool-call output and truncates each tool result to 500 chars. Because the rule is deterministic and the underlying tool result text doesn’t change, the truncated version is bit-identical at the same position on every subsequent call.

A sliding window usually changes the conversation prefix after the fixed instructions. Age-aware truncation also changes a prefix whenever a result first ages out. Determinism makes the shortened text stable afterward, but does not prevent that transition from invalidating later cached content.

Compaction can replace an old summary or append another summary block. Replacing it loses some prefix reuse but keeps the prompt smaller; appending preserves more history but lets it grow. Cache keys do not make changed content reusable: OpenAI’s prompt_cache_key is not an Anthropic-style cache_control breakpoint.

Two meanings of “caching”: prefix vs. semantic

Everything above is about prefix caching — the provider-side mechanism that makes resending a long, stable conversation cheap. It’s worth separating that from a different thing that also gets called “caching” in agent systems: semantic caching, which sits in front of the model and tries to skip the call entirely.

The two are easy to conflate because both promise “cheaper LLM calls”, but they operate at different layers and have different failure modes:

• Prefix caching reuses computation for matching input prefixes. The model still generates a fresh response; caching does not promise identical completions.
• Semantic caching returns a stored response to a sufficiently similar request. This can avoid inference, but the application must handle incorrect matches and stale answers.

A semantic-cache key for a coding agent would need to account for repository state, instructions, and tool results. Similar wording alone is not enough to reuse an old answer safely. The rest of this article concerns prefix caching.

Lossy storage vs. lossy view

Standing separately is the question of where compression happens — and therefore what gets stored in the conversation. Two options, same observable effect on the model:

• Tool layer: the tool limits output before it enters the conversation. Omitted text is absent from that result, but may still exist in a file or separate output log.
• Strategy layer: the harness stores full results and builds a shortened view for each model call. This is how our four strategies work.

The difference doesn’t show up in the LLM’s prompt — both designs produce the same text. It shows up in what stays on disk:

A strategy can restore text retained in its log without rerunning a tool. A tool-layer cap may require another read, and that read can return a newer file version. Neither approach guarantees that old information remains available indefinitely.

This split is reflected in how agent frameworks expose hooks. Tool-layer compression doesn’t need a framework hook — tools are just functions you write, so capping at the tool layer means putting the cap in the tool’s implementation. The strategy layer is different: it runs on every turn against a moving target (the growing conversation), so the framework has to expose an entry point for it.

Each trial records its full tool output. Trials are independent agent runs, not replays of one fixed conversation: their actions and trajectories can differ.

Tool output design: the other half of the picture

Everything so far has operated at the context-assembly layer — transformContext runs on a message list that’s already in hand. But there’s a parallel design space one layer down: what the tools themselves choose to return. A tool that dumps raw output makes your strategy do all the work. A tool that bounds its own output makes your strategy’s job smaller — sometimes vanishingly so.

Two tool features matter here: a per-call output cap and pagination. For example, the pinned opencode read implementation limits output bytes and line length and supports reading a selected range. Pagination lets the agent request omitted portions.

Tool caps bound individual results, not the total conversation. Pagination is selective retrieval rather than compression, and repeated reads can still accumulate a large history.

How real CLIs handle context

Before we pin down which strategies we’ll measure, it’s worth seeing how production CLIs actually solve the problem. Each one picks a specific blend of the patterns from the taxonomy, sometimes informed by what their target provider’s API exposes. Here’s a quick survey of what’s visible in source.

Claude Code

Claude Code’s documentation describes automatic compaction when context fills up. The layouts and per-tool policies below are illustrative designs, not a verified reconstruction of its internal implementation.

A cache-friendly layout places stable instructions before changing history. The following pseudocode shows a seed message before compaction and a summary afterward. In a real Anthropic request, message-level breakpoints belong on content blocks.

// Early in a session, before history has crossed the compaction threshold.
await client.messages.create({
  model: "claude-...",
  system: [
    // ─── STATIC: bit-identical across turns ─────────────────
    { type: "text", text: SYSTEM_PROMPT },                    // ~5KB, never changes
    { type: "text", text: TOOL_DESCRIPTIONS },                // ~8KB, never changes
    { type: "text", text: workspaceSummary },                 // computed once at session start
    { type: "text", text: CLAUDE_MD_CONTENTS,
      cache_control: { type: "ephemeral" } },                 // 👈 cache breakpoint #1
                                                              // (everything above this point is cached)
  ],
  messages: [
    // ─── frozen prefix: just the seed user message ─────
    { role: "user", content: [{ type: "text", text: SEED_USER_MESSAGE,
      cache_control: { type: "ephemeral" } }] },                 // 👈 cache breakpoint #2 (on the seed)
    // ─── tail: every turn so far, appended ─────────────
    ...allTurnsSoFar,
  ],
});

// After compaction has fired at least once: the older portion of the
// conversation has been replaced by a frozen summary, and breakpoint #2
// has shifted forward to land on it.
await client.messages.create({
  model: "claude-...",
  system: [
    // ─── STATIC: bit-identical across turns ─────────────────
    { type: "text", text: SYSTEM_PROMPT },                    // ~5KB, never changes
    { type: "text", text: TOOL_DESCRIPTIONS },                // ~8KB, never changes
    { type: "text", text: workspaceSummary },                 // computed once at session start
    { type: "text", text: CLAUDE_MD_CONTENTS,
      cache_control: { type: "ephemeral" } },                 // 👈 cache breakpoint #1
  ],
  messages: [
    // ─── frozen prefix: bit-stable from compaction onward ─────
    { role: "user", content: SEED_USER_MESSAGE },             // first user message in the run
    { role: "assistant", content: [{ type: "text", text: FROZEN_SUMMARY,
      cache_control: { type: "ephemeral" } }] },                 // 👈 cache breakpoint #2 (last frozen item)
    // ─── tail: appended each turn since compaction ───────────
    ...recentKMessages,
  ],
});

These markers identify nested prefixes, not independent cached fragments. Changing an earlier block invalidates the match for later content. Replacing the summary therefore requires a new prefix match; the stable instruction prefix may still be reusable.

Per-tool aging policies

A per-tool policy can shorten old file reads while retaining write confirmations and test results. The following table is an example policy for our four tools. We do not benchmark this policy separately.

A small clarification on what “older than the last 3 reads” actually means: it counts calls of the same tool, not turns. To make that concrete, suppose the agent’s call history so far is:

turn 1:  read_file(api.ts)         ← 1st read
turn 2:  list_files(./src)
turn 3:  read_file(api.ts)         ← 2nd read
turn 4:  run_tests()
turn 5:  read_file(storage.ts)     ← 3rd read
turn 6:  read_file(api.ts)         ← 4th read
turn 7:  read_file(serializer.ts)  ← 5th read

After turn 7, looking only at read_file calls (the ones at turns 1, 3, 5, 6, 7), the 3 most recent are turns 5, 6, and 7. So:

• Reads at turns 1 and 3 → stubbed.
• Reads at turns 5, 6, 7 → kept verbatim.

If turn 8 is run_tests() (not a read), nothing changes. The moment turn 9 is another read_file, the read at turn 5 ages out — it becomes the 4th-most-recent read — and gets stubbed. Each tool is independently ranked by call recency, and the top-K of each rank stay verbatim.

Keeping a path lets the agent read the file again, but does not preserve the old contents. Test results can also become stale after edits. The policy must reflect which historical facts the task needs.

The structured-summary template

Our structured compaction experiment uses the following five-section summary template. It asks the summarizer to preserve the task, current state, discoveries, next steps, and exact details needed to continue.

You produce continuation summaries for coding agents that have run out of context.

Output the summary wrapped in <summary></summary> tags, with the following five sections
as level-2 markdown headings, in order:

## Task Overview — what the user asked for, in one or two sentences.
## Current State — files created, modified, or analyzed, listed with their full paths;
                   state of the test suite; open work.
## Important Discoveries — key facts the agent learned, including approaches that did
                           NOT work and why.
## Next Steps — the immediate action the continuing agent should take.
## Context to Preserve — user preferences, promises made, constraints that must not
                         be violated.

Be specific. Cite exact filenames. No filler. No conversational framing.

The structure gives the summarizer a concrete handover format. It can still omit or distort details, so the template’s usefulness must be evaluated on downstream tasks.

Codex

Codex exposes separate controls for automatic compaction and stored tool output. Its configuration reference documents model_auto_compact_token_limit and tool_output_token_limit. These address different sources of context growth.

opencode

opencode combines bounded tool output with conversation compaction. These operate at different layers; limiting each result does not eliminate the need to manage the history.

pi-coding-agent

Our experiment uses the bare Agent with custom tools and explicit context strategies. This should not be confused with the higher-level pi coding agent, which adds its own tools and context management.

The strategies we implement

We compare four strategies against baseline, which sends the unmodified history.

Each strategy gets three independent trials on the same fixture, model, and initial prompt. We report means and sample standard deviations. Three trials can expose behavior in this fixture, but cannot establish reliable failure rates or a general ranking.

We use one small fixture and one parameter setting for each strategy. The names below identify the implementation and its parameters.

In age-truncate-500-keep-3, 500 is the retained prefix length per shortened text block and 3 is the number of complete recent tool results. The truncation marker adds extra characters. compact-at-12000-structured performs one structured compaction after the 12,000-character threshold.

Sliding windows, retrieval, per-tool stubs, and repeated compaction are outside this experiment’s scope. Their cache behavior depends on implementation and cannot be inferred from these four conditions.

The four-strategy table below is the same set, grouped by mode — the axis the article’s cost analysis turns on:

Per-turn (transformation applied on every LLM call).

At thresholds (fires once when the conversation crosses a size limit, then freezes).

The reference table summarizes the four implementations and their limitations.

How pi wires them

The strategies above are agent-framework-agnostic — they describe what to do with the message list. To actually run them in our experiment we need a place to plug them in. We use pi because, unlike most agentic CLIs, it exposes context assembly as a first-class extension point — a function you write — which makes strategies trivially swappable for comparison.

Pi is structured so that, on every iteration of the agentic loop — right before sending the message history to the LLM, after the latest tool results have been appended to that history — the agent calls two user-overridable hooks between “the current transcript” and “what the LLM actually sees”:

new Agent({
  initialState: { systemPrompt, model, tools, thinkingLevel: "off" },
  // Structural layer: prune, summarize, or inject messages.
  transformContext: async (messages) => { /* ...your logic... */ },
  // Mapping layer: filter or translate custom message types.
  convertToLlm: (messages) => messages.filter(/* ... */),
});

transformContext is the hook that decides what conversation the agent should have. It takes the full conversation as AgentMessage[] — pi’s name for the unified message type that covers user, assistant, and tool-result messages — and returns a (possibly modified) AgentMessage[]. Same type in, same type out. This is where every strategy in the taxonomy above lives: cap each tool result to N chars (truncate-500), cap only older ones (age-truncate-500-keep-3), summarize at threshold (compact-at-12000-structured), and so on.

convertToLlm converts internal AgentMessage[] into the message format used by pi’s model interface. Provider adapters perform the provider-specific request conversion afterward.

For this article we leave convertToLlm at its default (the identity filter for standard message roles) and focus entirely on transformContext. In pi’s architecture, a context-assembly strategy is just a function:

type Strategy = (messages: AgentMessage[]) => Promise<AgentMessage[]>;

That signature is all the extension surface there is. Because it’s code (not a config blob), a strategy can do arbitrary work: call another LLM to summarize old turns, embed past messages and retrieve by similarity, read files from disk, maintain state across turns via a closure. The strategies we walked through above span from three lines (baseline) to a few dozen (compact-at-N-structured), but they all share this same shape — and they’re all swappable by passing a different function to the same hook.

To make this concrete, here’s what pi does on a single turn — picking up mid-session, after the conversation history already holds the user’s seed prompt, several rounds of LLM responses, and a stack of tool results from earlier turns:

1. transformContext runs over the full conversation history — including any 50KB tool results sitting in there untouched — and produces the message list to send. The strategy decides what to do with each piece: pass through (baseline), truncate uniformly to 500 chars (truncate-500), truncate only older results (age-truncate-500-keep-3), fold older history into a summary (compact-at-12000-structured), and so on. Pi sends the resulting list to the LLM. The LLM sees the strategy’s view, not the original.
2. The LLM responds — text, tool-call intents, or both.
3. If the LLM emitted tool calls, pi executes each one. Each tool returns its full output (e.g., 50KB of file content). Pi appends both the LLM’s response and each tool result to the conversation history.
4. Loop back to step 1.
5. Repeat until the LLM emits a response with no tool calls — that’s the agent’s signal to stop.

In this experiment, the logger retains full tool results while strategies shorten only the model’s view. Restoring an old result from that log does not require repeating the tool call.

One detail worth being explicit about: in pi, the bare Agent class ships no default strategy. If you instantiate new Agent({...}) without supplying transformContext, you get the baseline identity behavior — the entire conversation is sent every turn. The higher-level pi-coding-agent package built on top of Agent does ship a default (multi-round summarization on overflow, similar shape to opencode). We use the bare Agent for these experiments so that every strategy in the comparison is something we wrote explicitly, with nothing built-in to control for.

Experiment setup

The fixture we’re going to use to test different strategies on is the service-and-storage layer of a TODO web app. Two classes do the work: TaskStore keeps an in-memory list of tasks, and Api is a thin dispatch layer that takes request objects and routes them to the store:

export type ApiRequest =
  | { action: "add"; payload: { title: string } }
  | { action: "complete"; payload: { id: unknown } }
  | { action: "get"; payload: { id: unknown } }
  | { action: "list" };

export class Api {
  constructor(private store: TaskStore) {}

  handle(request: ApiRequest): ApiResponse {
    switch (request.action) {
      case "complete": {
        // 👇 The bug. `payload.id` is typed `unknown` and arrives as a string
        //    when the request comes from JSON. The cast silences TypeScript
        //    but does no runtime coercion — so `markComplete("1")` reaches
        //    `t.id === id` where t.id is a number, and the lookup misses.
        const found = this.store.markComplete(request.payload.id as number);
        return { ok: true, data: { completed: found } };
      }
      // …other cases…
    }
  }
}

export class TaskStore {
  private tasks: Task[] = [];

  markComplete(id: number): boolean {
    // 👇 Strict equality. If `id` arrives as a string ("1"), this returns
    //    undefined even when a Task with id 1 exists. Combined with the
    //    missing coercion in api.ts, this is what breaks the test.
    const task = this.tasks.find((t) => t.id === id);
    if (!task) return false;
    task.completed = true;
    return true;
  }
  // …add, get, list, clear…
}

The bug spans src/api.ts (the dispatcher) and src/storage.ts (the typed store) — the cast in one file plus the strict equality in the other is what produces the failing test.

The fix converts the request’s ID with Number() at the API boundary. The affected line occurs after character 500 of api.ts, beyond the uniform truncation limit.

This fixture is much smaller than repository-level evaluations such as SWE-bench. Its advantage here is that each transcript is short enough to inspect. We hold the model fixed and vary context handling; larger tasks would require a separate evaluation.

The tests in test/tasklist.test.ts exercise the full surface end-to-end: add tasks, list them, mark complete via a JSON-encoded request, fetch by id. 3 source files, 1 failing test, single-file fix.

// the failing test, abridged
import { test } from "node:test";
import assert from "node:assert/strict";
import { Api, parseRequest } from "../src/api.ts";
import { TaskStore } from "../src/storage.ts";

test("complete via API with a JSON payload marks the task completed", () => {
  const store = new TaskStore();
  const api = new Api(store);
  api.handle({ action: "add", payload: { title: "buy milk" } });

  // Clients serialize ids as strings (JSON over HTTP, URL path params, etc.).
  const raw = JSON.stringify({ action: "complete", payload: { id: "1" } });
  const request = parseRequest(raw);
  const res = api.handle(request);

  assert.equal(res.ok, true);
  if (!res.ok) return;
  assert.deepEqual(res.data, { completed: true });
});

// …also: "add creates a task with an id", "list returns all tasks"

The tests serialize requests with a string-valued id and pass them through parseRequest(). JSON preserves that string; it does not turn numeric IDs into strings. The agent gets four tools and the task of making the tests pass.

Our four custom tools are thin wrappers registered with pi’s AgentTool interface. File reads are deliberately uncapped and have no pagination, so tool-layer limits do not obscure the strategy comparison. Independent runs can still follow different trajectories.

Below you can see the four tool bodies, stripped to their execute paths (schemas, labels, and workdir resolution elided for clarity):

// no per-call cap, no pagination, no per-line limit
async (_id, args) => {
  const abs = resolveInWorkdir(workdir, args.path);
  const contents = fs.readFileSync(abs, "utf8");
  return textResult(contents);
}

// plain overwrite — no diff, no validation, no edit-tolerance policy
async (_id, args) => {
  const abs = resolveInWorkdir(workdir, args.path);
  fs.mkdirSync(path.dirname(abs), { recursive: true });
  fs.writeFileSync(abs, args.content, "utf8");
  return textResult(`wrote ${args.content.length} bytes to ${args.path}`);
}

// recursive walk; returns every file path joined by newlines
async (_id, args) => {
  const rel = args.path ?? ".";
  const abs = resolveInWorkdir(workdir, rel);
  const entries: string[] = [];
  const walk = (dir: string) => {
    for (const name of fs.readdirSync(dir)) {
      const full = path.join(dir, name);
      if (fs.statSync(full).isDirectory()) {
        if (name === "node_modules" || name === ".git") continue;
        walk(full);
      } else {
        entries.push(path.relative(workdir.root, full));
      }
    }
  };
  walk(abs);
  entries.sort();
  return textResult(entries.join("\n") || "(empty)");
}

// shells out to `node --test`; returns full stdout + stderr + exit code
async () => {
  const testFiles = fs.readdirSync(path.join(workdir.root, "test"))
    .filter((n) => n.endsWith(".test.ts"))
    .map((n) => path.join("test", n));
  const result = spawnSync(
    "node",
    ["--experimental-strip-types", "--test", ...testFiles],
    { cwd: workdir.root, encoding: "utf8", timeout: 30_000 },
  );
  return textResult(
    `exit_code: ${result.status ?? -1}\n` +
    `--- stdout ---\n${result.stdout}\n` +
    `--- stderr ---\n${result.stderr}`,
  );
}

The chart shows one representative run for each strategy. Select prompt size, cumulative logged cost, or cache hit ratio.

• Prompt size: new plus cached input tokens, using pi’s normalized usage fields.
• Cumulative logged cost: the sum of agent-call costs. The separate summarizer call is not logged here.
• Cache hit ratio: cached input divided by total input for that call.

You can also step through any of the four runs turn-by-turn below. A few terms first.

A turn is one LLM call. The cycle around each turn:

1. transformContext runs over the agent’s accumulated conversation history.
2. The LLM is invoked with the result.
3. The LLM emits a response — text and/or tool-call intents.
4. The agent dispatches the tool calls, runs each tool, and appends the results to the history.

Each subsequent model call counts as another turn. The number of messages depends on how many tool calls each assistant response contains.

Click any Turn N in the left navigator to focus on it. The widget shows the moment just before that turn’s LLM call: the Before strategy side is everything accumulated through turn N-1’s tool results (turn N’s response hasn’t happened yet at this point); the After strategy side is what transformContext produced from that input — the prompt the LLM actually saw. The default Diff view colors what changed: red lines were dropped by the strategy, green lines were added or replaced, gray lines align across both sides. Switch to Cards for a structured per-message view. For baseline, the two sides are byte-identical — that’s the control case. For the other three strategies, the diff is the article’s central question made literal.

=== USER ===
A test in test/tasklist.test.ts is failing. Find the bug in the source code under src/, fix it, and make the whole test suite pass.

=== USER ===
A test in test/tasklist.test.ts is failing. Find the bug in the source code under src/, fix it, and make the whole test suite pass.

For each run we copy the fixture to an isolated scratch directory, give the agent the four tools, and let it work until it stops. verify() runs node --test one more time and records whether the test suite is green.

Model: Gemini 2.5 Flash, with temperature = 0 set through pi’s onPayload hook. We observed different trajectories even at this setting, so each condition has three trials. The experiment does not identify the source of that variation.

What we measure per run:

• pass — did the test suite go green at the end?
• turns — number of assistant turns (i.e., LLM calls).
• cost — logged agent-call cost, excluding the separate summarizer call.
• peak prompt — the largest prompt size (new + cached tokens) the agent ever sent.
• new input tokens — uncached prompt tokens, summed. The ones you pay full price for.
• cached input tokens — tokens served from Gemini’s implicit prefix cache.

Cached-token counts reflect both prefix reuse and how many calls the run made. They are not a standalone measure of a strategy’s cache stability.

Results per strategy

The numbers below are means across the 3 trials (k=3) per strategy; ± values are the sample standard deviation (so 13±3 means a mean of 13 turns with σ ≈ 3 across the runs). The new and cached columns are also per-run means — what a typical single trial paid in uncached vs. cached input tokens.

Uniform truncation passed 0/3 trials, with logged costs roughly six times baseline. The relevant code lies beyond character 500. The truncation marker reveals that text is missing, but these tools offer no paginated read to recover it within the capped view.

Age-aware truncation passed 3/3 trials at a mean logged cost of $0.017. Fresh results remain complete, but older information can disappear from view. Its 11K cached tokens are fewer than baseline’s 18K; aging a result changes the prefix, even when the transformation is deterministic.

Compaction passed 3/3 trials, with $0.016 mean logged agent cost. The logger omits the summarizer call’s usage, so this is not an all-in cost and does not show that savings paid for summarization. The summarizer also receives a pre-truncated transcript: tool output is limited to 1,500 characters and tool-call arguments to 200 characters.

The experiment demonstrates one failure of an aggressive cap and three successful alternatives on this fixture. It does not establish which alternative is best across coding tasks.

A proposed custom algorithm: age-aware tool-result truncation

The age-aware strategy keeps the last three tool results intact and shortens older results:

export function makeAgeAwareTruncate({ keepRecent, maxChars }) {
  return async function(messages: AgentMessage[]) {
    const resultIndices = messages
      .map((m, i) => (m.role === "toolResult" ? i : -1))
      .filter((i) => i !== -1);
    const keepFromIndex =
      resultIndices.length > keepRecent
        ? resultIndices[resultIndices.length - keepRecent]
        : -1;

    return messages.map((msg, i) => {
      if (msg.role !== "toolResult") return msg;
      if (i >= keepFromIndex) return msg;  // recent — leave verbatim
      return truncateTextBlocks(msg, maxChars);
    });
  };
}

• Recent results remain complete, regardless of size.
• Older results can lose details that are still relevant.
• Aging a result changes the prefix from that point onward.
• The transformation needs no extra model call, but history still grows.

The choice of three protected results is a parameter of this fixture, not a general recommendation. Longer tasks and large fresh outputs need separate tests and size controls.

How this differs from opencode’s two-layer approach

A tool cap limits what enters the conversation; an age-based view limits what is sent later. The file may remain on disk in either case, while only a saved full result preserves the exact historical read. These mechanisms can be combined, but pagination and access to omitted content matter.