Prefix and state reuse

Most inference traffic repeats itself. An agent sends the same system prompt and tool definitions on every step and appends a turn to a growing history. A chat application re-sends the whole conversation each time. A document assistant asks many questions about the same file. Recomputing all of that on every request wastes time and money.

Tenzro Network 1 reuses the state a model has already computed for a prompt prefix, for every architecture class the network serves: dense, sliding-window, hybrid recurrent and state-space, mixture-of-experts, multimodal and media models. When a request starts with content a provider has already processed, the provider skips that part and computes only what is new.

Why it matters

A request has two phases. Prefill reads the whole prompt and builds the model's internal state. Decode then generates the reply one token at a time. For long prompts, prefill dominates the time before the first token appears, and it is billed as input.

With reuse:

• Latency falls. Time to first token depends on the new part of the prompt, not the whole history.
• Price falls. Reused prompt tokens are billed as cached-read tokens, at a rate each provider sets separately from fresh input.
• Capacity rises. Providers spend less compute per request, so the same hardware serves more callers.

Reuse for each architecture

Different model families keep different kinds of state, so the network reuses each one in the way that fits it.

Dense attention

Standard transformers keep a key-value (KV) cache: one entry per token per layer. The provider matches the new prompt against the cached tokens, keeps the longest common prefix and prefills from the first token that differs. A conversation that appends a turn reuses the whole earlier history. A prompt that shares only its system prompt with an earlier one reuses just that part.

Sliding-window attention

Models with sliding-window layers attend only to a recent window of tokens in those layers, so the state that matters is the window, not the full history. The provider keeps that windowed state alongside the full-attention layers and resumes from the matching position, rather than rebuilding the window from the beginning of the prompt.

Hybrid recurrent and state-space models

Hybrid models mix attention layers with recurrent or state-space layers. A recurrent layer compresses everything it has read into a fixed-size state, and that state cannot be wound back to an earlier token. Plain KV-cache matching therefore works only when the new prompt extends the old one exactly.

Tenzro providers snapshot the recurrent state at checkpoints as a prompt is processed, placed at the points where conversations actually diverge, such as turn boundaries in the chat template. When a new prompt departs from the cached one, the provider restores the deepest checkpoint at or before the point of divergence and prefills only from there. Agent traffic, where a re-sent history often differs slightly from what was generated, keeps most of its reuse.

Mixture-of-experts

In a mixture-of-experts model, each token is routed to a few expert networks, but the cached state is still the attention and recurrent state of the model's shared layers. Reuse works exactly as it does for the underlying attention type. When a model is sharded across the network by expert (see Distributed MoE), the node running the shared layers holds the reusable state, and expert holders receive only the new tokens.

Multimodal models

Vision-language and audio-language models encode each attachment into embeddings before prefill. Those encodings are keyed by the content hash of the attachment, so an image or clip that appears again in a later turn is not re-encoded, and the interleaved text-and-media prefix is reused like any other prefix.

Media models

Image and video generation repeats work too: the same prompt rendered at several seeds, or the same reference image edited many times. Media workers reuse the encoded prompt conditioning and the encodings of reference images, which are addressed by content hash in network storage, across jobs that share them. The denoising itself still runs for each job, because each render is new work.

Routing to warm state

Reuse only helps if the request reaches a provider that holds the prefix. Providers advertise compact summaries of the prefixes they hold warm, never the underlying state or prompt text. When the router scores providers for a request, it adds a bias toward those that already hold the request's prefix.

The bias is bounded. It chooses between comparable offers; it never overrides your price ceiling, jurisdiction pin, trust requirements or confidential-hardware requirement. See Inference.

Privacy

Reused state belongs to the caller whose request produced it. One caller's cached prompt state is not used to serve another caller, so response timing does not reveal what someone else asked. Prefix advertisements are summaries that cannot be read back into prompt content. Prompts and completions are not written to disk.

What you pay

Providers price cached prompt tokens separately:

tenzro provider pricing set \
  --input-price-wei <fresh-input> \
  --output-price-wei <output> \
  --price-per-cached-read-token-wei <reused-prompt-token> \
  --price-per-cached-write-token-wei <prompt-token-written-to-cache>

Your bill shows the split. GET /v1/generation?id=<completion_id> (or tenzro_getGeneration) reports cached_read_tokens and cached_write_tokens alongside input and output tokens whenever a call used the cache. input_tokens still counts the whole prompt, as OpenAI-compatible clients expect; the cached share is priced at the cached rate.

Getting the most from reuse

• Put stable content first. System prompt, tool definitions and reference documents at the start; the changing part at the end.
• Append, do not rewrite. Send earlier turns exactly as before. Reordering or editing history moves the point of divergence earlier.
• Keep the model fixed within a conversation. State is model-specific, so switching models starts from scratch.
• Use one identity for one workload. Reuse is scoped to the caller, so an agent that keeps the same account and API key across steps benefits across steps.

Next

• OpenAI-compatible API
• Model serving: setting your cached-token prices as a provider.
• SLA attestation and metering