Inference Optimisation From Consumer GPUs to the Data Centre

Inference cost is set by two resources that behave differently. Prefill, reading the prompt, is compute-bound and parallel. Decode, generating one token at a time, is bound by memory bandwidth: every step reads the weights and the growing KV cache to produce a single token per sequence. Nearly every optimisation of the last few years either makes those bytes smaller, gets more tokens out of each read, or keeps the two phases from getting in each other's way.

This post surveys five techniques, with the 2025-2026 work that moved them, and then asks a question most serving guides skip: what changes when the GPUs belong to many independent operators connected by the public internet rather than one fabric in one building.

1. Quantisation: fewer bytes per weight

Decode reads every active weight per step, so halving the bytes per weight roughly halves the bandwidth each token needs. The newest step is hardware-native 4-bit floating point. NVIDIA's NVFP4 on Blackwell stores 4-bit values in 16-value micro-blocks, each with an FP8 (E4M3) scale, plus a per-tensor FP32 scale. NVIDIA reports about 3.5 times less memory than FP16 and about 1.8 times less than FP8, and on DeepSeek-R1-0528 an accuracy change of 1% or less on key tasks after post-training quantisation from FP8.

The catch for a heterogeneous fleet is that formats are runtime- and hardware-specific. An NVFP4 checkpoint needs a runtime and a GPU generation that support it; a GGUF k-quant runs almost anywhere. Tenzro's serving path is built on llama.cpp's ggml runtime, which has backends for CUDA, ROCm, Metal, Vulkan, SYCL, OpenVINO, OpenCL, CANN and others, so one catalog entry serves on very different hardware. NVFP4 builds are not loadable by that runtime today, so the catalog does not list them: an entry that fails at download time on an operator's machine is worse than no entry.

2. Speculative decoding: more tokens per read

A small drafter proposes several tokens; the target model verifies them in one forward pass and keeps the longest prefix it agrees with. The output is identical to plain decoding, and each expensive read of the target's weights can yield several tokens.

The drafters have improved quickly. EAGLE-3 reported speedups of up to 6.5 times over standard decoding. DeepSeek-V3 trained multi-token prediction into the model, and that head can serve as a built-in drafter. DFlash replaced the autoregressive drafter with a lightweight block diffusion model that drafts a block in parallel, reporting more than 6 times lossless acceleration and up to 2.5 times the speedup of EAGLE-3.

The engineering lesson is that speculation competes with batching for the same spare compute. In our own measurements of a 27B-parameter model on a single GB10-class machine, a block drafter proposing seven tokens raised throughput from 13.2 to 32.8 tokens per second at one stream and from 92.4 to 102.5 at sixteen. The model's own multi-token-prediction head helped at one stream (23.1) but cost throughput at four and sixteen streams. So Tenzro's batch engine uses the MTP head only while at most two sequences are generating, and a block drafter at every width. Speculation also composes with verifiability: a request that asks for top-k logit commitments gets one record per verified token, speculative steps included.

3. Continuous batching: keep the GPU full

Static batching waits for the slowest sequence in a batch. Orca introduced iteration-level scheduling: requests join and leave the batch at every decode step, so the GPU never idles on finished sequences. It is now standard.

Tenzro's batch engine runs one context with a fixed pool of sequence slots and decodes every running request in one step. Two details matter in production. A client that reads its stream slowly is buffered per client and ended if it falls more than 1 MiB behind, so one slow reader cannot stall the step for everyone. And the queue is bounded: once a slot's worth of requests is waiting, new submissions are refused rather than queued without limit.

4. KV cache: page it, then reuse it

The KV cache grows with every token and fragments memory if allocated naively. PagedAttention applied virtual-memory paging to the cache and reported 2 to 4 times higher throughput at the same latency, largely by fitting more sequences.

For agents, the bigger win is reuse. An agent's next turn repeats the whole conversation so far, so a cache that survives between requests turns a prefill of tens of thousands of tokens into a few hundred. Reuse is harder than it looks on newer architectures. Attention caches can be trimmed back to any position. Sliding-window and recurrent layers (Mamba, RWKV, and hybrids of recurrent layers with attention) cannot be rewound that way.

Tenzro's batch engine decides what can be reused from what the loaded model's memory can do, read at load time, never from its name. For recurrent and windowed models it takes partial checkpoints of the state at chat-turn boundaries, found by rendering a probe conversation through the model's own chat template, and budgets them against measured free host memory. Prefix affinity is learned without leaking it: providers announce only a count of warm prefixes, and which provider holds a caller's prefix is found by probing with digests salted per principal.

5. Prefill/decode disaggregation: separate the phases

Because prefill and decode stress different resources, running them on separate GPU pools lets each use its own parallelism and stops long prompts from stalling ongoing decodes. DistServe made the case in terms of goodput under latency targets. NVIDIA Dynamo implements it with direct GPU-to-GPU KV transfer, and its documentation is candid that it is not free: aggregated serving is often better at low concurrency, or when decode KV capacity is the bottleneck.

What changes on a distributed network

Inside one data centre, KV transfer rides a fast fabric. Across operators on the internet, moving a KV cache per request is usually a bad trade. That reshapes which techniques travel:

• Quantisation and speculation are local. They need no coordination and help every provider, which makes them the most valuable optimisations for an open network. The constraint is format portability.
• Continuous batching is local too, but routing decides how full each provider's batch is. A router that spreads load evenly can leave every provider half-empty.
• Cache reuse depends on routing. Sending a conversation's next turn back to the provider that holds its prefix is worth more than a slightly lower price elsewhere.
• Disaggregation belongs inside an operator's site, not across the network. Tenzro does not split prefill and decode across operators. It moves KV state over the network only at turn boundaries, for session continuity: a session's cache deltas are sealed to a successor provider in the same jurisdiction, which takes over without re-prefilling if the primary fails.
• Activations can cross the network when the model demands it. For Mixture-of-Experts models too large for one provider, Tenzro dispatches per-token expert batches to the holders of each expert, compressing hidden states to 8-bit blocks (about four times smaller than FP32, with roughly 0.4% error) and folding responses in as they arrive.

What this means for GPU operators

On consumer and workstation GPUs, the biggest gains come from portable quantisation, a good drafter and a cache that survives between turns. On data-centre GPUs, add native low-precision formats and, within a site, disaggregation where load is sustained. On a shared network, the operator who keeps prefixes warm and batches full earns the next request.

Where Tenzro fits

Built: a ggml-based serving path with backends across major accelerator vendors; a continuous batch engine with bounded queues; speculative decoding with MTP heads and block drafters, chosen by batch width; architecture-aware prefix reuse; session continuity with sealed KV checkpoints; expert-sharded MoE dispatch; and verifiable top-k commitments that survive speculation. Not supported today: NVFP4 checkpoints and cross-operator prefill/decode disaggregation. The network is live in preview; governance decides the steps to testnet and mainnet.