Prompt caching for long agent runs

A coding agent does not remember anything between turns. Every time it calls the model, it sends the system prompt, the tool definitions, every file it has read, and every turn so far, then adds one new tool result at the end. By turn 80, a single request can carry a couple of hundred thousand tokens, almost all of which the model saw a few seconds ago.

Prompt caching is what makes that affordable. XField caches automatically: there is no flag to set and no cache_control marker to place. But the cache only helps if your requests are shaped so it can match. This post covers how it works, what breaks it, and how to check it from the usage fields.

Why agents resend everything

The model APIs are stateless. POST /api/v1/chat/completions and POST /api/v1/messages take a full message list and return one reply. The agent loop owns the state: it appends the assistant reply and the next tool result to its list, then sends the whole list again.

So each request is the previous request plus a small tail. That shape is exactly what a prefix cache is built for.

How prefix caching works

To produce a reply, the model first processes every input token and builds a key-value (KV) cache: the attention state for each position. That prefill step is most of the compute in a long request.

The KV state for a token depends only on the tokens before it. If two requests start with the same 150,000 tokens, the KV state for those 150,000 positions is identical. A prefix cache stores it, keyed by the token sequence. When a new request arrives, the server finds the longest stored prefix that matches the start of the request token for token, reuses that state, and only computes the rest.

Three consequences follow.

• Matching is exact and left-to-right. One changed token invalidates everything after it, even if the rest is byte-identical.
• Caches are stored in blocks. The reported cached count is usually rounded down to a block boundary, so it will rarely equal your full prefix.
• Entries are evicted when idle. A session that pauses for a long time may start its next turn cold. The lifetime is not guaranteed, so do not design around a specific number.

What breaks the cache

Most cache misses are self-inflicted, and they all have the same cause: something near the start of the request changes between turns.

Timestamps and random IDs in the system prompt. Current time: 14:02:17 or Session: 7f3a… at the top of the system prompt changes every turn, and every token after it misses. If the model needs the date, put the date (not the time) in the system prompt, or pass the time in the newest user message.

Reordering tools. Tool definitions are serialized into the prompt near the start. Building the tool list from a set, from a dict with unstable ordering, or from MCP servers that connect in a different order each run gives you a new prefix. Sort tools by name, and serialize JSON schemas with stable key order.

Editing early history. Rewriting an old tool result, trimming a file you read on turn 3, or redacting an earlier message invalidates every turn after the edit. Append instead.

Changing models mid-session. Each model has its own cache. Switching from deepseek-v4.1-flash-1017-uncensored to glm-5.3-uncensored for one turn and back again means two cold turns. If you want a second model for a subtask, give it its own short context.

How to structure requests

Order the request from most stable to least stable:

1. System prompt, fixed for the session.
2. Tool definitions, in a fixed order.
3. Project context that does not change: CLAUDE.md, a repo map.
4. Conversation history, append-only.
5. The newest message or tool result.

import os
from openai import OpenAI
 
client = OpenAI(base_url="https://xfield.cloud.brutal.ai/api/v1", api_key=os.environ["XFIELD_API_KEY"])
 
SYSTEM = open("system_prompt.md").read()          # no timestamps, no IDs
TOOLS = sorted(load_tools(), key=lambda t: t["function"]["name"])
 
messages = [{"role": "system", "content": SYSTEM}]
 
def turn(new_content):
    messages.append({"role": "user", "content": new_content})
    resp = client.chat.completions.create(
        model="deepseek-v4.1-flash-1017-uncensored",
        messages=messages,
        tools=TOOLS,
    )
    messages.append(resp.choices[0].message)   # append, never rewrite
    return resp

Compaction without losing the prefix

Long runs eventually need compaction: replacing old turns with a summary. That is an edit to early history, so it costs one cache miss. Make it a deliberate, rare one.

• Compact at a threshold (say, when the context passes 60% of the window), not a little every turn.
• Keep the system prompt and tools untouched, and put the summary after them. Those tokens stay cached through the compaction.
• After compacting, go back to append-only. The new, shorter history becomes the prefix for every turn that follows.

One compaction trades a single partial miss for many cheaper turns afterwards. Trimming a little every turn trades nothing for a miss on every turn.

Reading the usage fields

Chat Completions reports cached input inside prompt_tokens. On a stream, usage arrives in the last chunk.

{
  "usage": {
    "prompt_tokens": 161500,
    "completion_tokens": 512,
    "prompt_tokens_details": { "cached_tokens": 158976 }
  }
}

Fresh input is prompt_tokens - cached_tokens. The Responses API (POST /api/v1/responses) reports the same thing as usage.input_tokens_details.cached_tokens.

The Messages API uses Anthropic's fields, and here input_tokens is only the uncached part:

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic({
  baseURL: "https://xfield.cloud.brutal.ai/api",
  apiKey: process.env.XFIELD_API_KEY,
});
 
const msg = await client.messages.create({
  model: "deepseek-v4.1-flash-1017-uncensored",
  max_tokens: 4096,
  system: SYSTEM,
  tools: TOOLS,
  messages,
});
 
const u = msg.usage;
const cached = u.cache_read_input_tokens ?? 0;
const total = u.input_tokens + cached + (u.cache_creation_input_tokens ?? 0);
console.log(`cached ${cached} of ${total} input tokens (${((100 * cached) / total).toFixed(1)}%)`);

Log the cached share on every turn. From turn two onward it should sit in the high nineties. A turn that drops to near zero tells you exactly where the prefix changed.

A worked example: 100 turns

Take a session with plausible numbers:

• System prompt plus tool definitions: 12,000 tokens.
• Each turn adds 2,000 tokens of new input (a tool result or a message).
• Each reply is 500 tokens, which becomes history for the next turn.

Turn 1 sends 14,000 tokens, all fresh. Turn k sends 14,000 + 2,500 × (k − 1) tokens, of which only the last reply and the new input, 2,500 tokens, are fresh. The rest matched the previous request.

Notice that fresh input equals the final context size. Every token is fresh exactly once, on the turn it is added, and cached on every turn after that. That is the whole economy of an agent run: you pay full price for each token once and the cached rate for every reread.

Now put a timestamp at the top of the system prompt. Every turn misses, and all 13,775,000 input tokens are fresh: about 53 times the fresh input of the stable version, for the same work.

Picking a model for it

Since cached input is most of what an agent sends, the cached-input rate matters more than the headline input rate. That is why deepseek-v4.1-flash-1017-uncensored, the newest model we serve, is the usual pick for agents: 1M context, strong tool calling, and cheap cached input. Pricing is shared with your invitation.

The coding agents guide covers model choice in more detail, and the Claude Code setup shows how to point an agent at XField. No key yet? Request access.