Multi-region AI: latency, jurisdiction and failover

Most AI products now serve users on more than one continent. That brings three problems together that are usually handled by different teams. Network engineers worry about latency, lawyers about jurisdiction and site reliability engineers about failover. In multi-region inference the three interact. The fastest failover target may be in the wrong jurisdiction, and the lawful one may be too far away to meet the latency budget. This article puts numbers on the trade-offs and describes a design that handles all three at once.

What distance costs, in measured numbers

Microsoft publishes median round-trip times between its regions, measured continuously by backbone probes. Figures from the 30-day period ending 30 July 2026 are a useful reference for any provider:

These are medians on a well-engineered private backbone. Public internet paths and tail percentiles will be worse.

Set them against what inference actually costs. A short chat response from a mid-sized model takes hundreds of milliseconds to seconds to generate, so an 85 ms transatlantic round trip is noticeable but tolerable. For agents, though, network latency multiplies. An agent that makes ten sequential model and tool calls pays the round trip ten times. Served from the wrong continent, that adds more than 800 ms of pure network time across the Atlantic and more than two seconds between Europe and Japan. Voice and real-time vision have tighter budgets still.

The first design rule follows: keep the interactive loop in-region, and cross regions only for steps that are not on the critical path.

Jurisdiction draws different borders

Network topology and law do not draw the same map. Frankfurt and Amsterdam are 11 ms apart and both inside the EU, but Zurich is close by and outside it. Inside the US, latency varies but the legal regime is uniform. In APAC, a short hop can cross several legal regimes at once.

The rules also differ in kind. In the EU, personal data may go to certified US organisations under the Data Privacy Framework, but many buyers still require EEA-only processing as policy. India's DPDP Act permits transfers by default unless the government restricts specific countries, with its cross-border provisions operative from 13 May 2027 according to Hogan Lovells. Sector rules in finance, health and the public sector often add stricter localisation on top.

The design consequence is that jurisdiction must be a per-request property, set by whoever knows the data, and not a deployment-wide setting. A single product may serve one tenant pinned to the EU, another pinned to a single country and a third with no constraint at all.

Failover is where the two collide

Every failover is a placement decision made under pressure. If the primary provider in Frankfurt fails, where does the request go? The fastest answer may be Zurich or London. The lawful answer for an EU-pinned tenant is another EU site. The resilient answer is a site run by a different operator, so one company's outage cannot take out both.

This leads to three rules.

Choose the successor in advance, under the same constraint. At the start of a long session, assign a successor that serves the same model, satisfies the same jurisdiction pin and is run by another operator or at another site. Ship it the session's state as you go, sealed so that only it can read the state.

Fail closed, not open. If no lawful successor exists, refuse the request rather than routing it out of jurisdiction. Then make sure a lawful successor exists, by having more than one provider in every jurisdiction you promise to serve.

Keep retries bounded. One hedge and one failover per request is usually enough. Unbounded retries turn a partial outage into a self-inflicted overload.

The tail is the real service level

Users experience the slowest step, not the median. Dean and Barroso showed that in fan-out systems, the tail of each component sets the experience of the whole. Their remedy, the hedged request, applies directly. If a provider has not answered by its usual 95th-percentile latency, send the same request to a second provider and take whichever answers first. Because hedges fire only in the tail, the extra load is small.

Within a region, hedging between independent providers is cheap and effective. Across regions, it costs the full round trip and may break jurisdiction rules, so keep hedge targets inside the same pin.

State and cache across regions

Inference has state that classic web services do not: the KV cache. A multi-turn conversation that moves to a new provider must either re-process its whole history, which is expensive for long contexts, or bring its cache along. Checkpointing the cache to a pre-assigned successor at turn boundaries makes failover close to free for the user. Sealing it to the successor's key keeps it confidential in transit.

Prefix caching raises a quieter issue. Routing a user to the provider that already holds their prompt prefix saves work, but advertising which prefixes are warm can leak what other users asked. The safe pattern is for providers to announce only a count of warm prefixes. A router then learns where a caller's own prefix is warm through probes salted per caller, which only that caller's requests can produce.

What this means for infrastructure operators and CTOs

Map your regions twice, once by latency and once by law, and design around where the two maps differ. Keep the interactive loop inside the latency budget. Make jurisdiction a request parameter. Pre-assign successors under the same constraint and with a different operator. Hedge only within a pin. And measure the tail, not the median, because that is what your users feel.

Where Tenzro fits

Tenzro is an open network where machines sell AI compute and inference. It is live in preview, and governance decides the steps to testnet and mainnet. What is built:

• Region and jurisdiction. Providers declare a coarse region and a jurisdiction (country plus optional blocs). Requests can carry a pin that fails closed, and pinned responses can carry a signed jurisdiction receipt.
• Session successors. Sessions can request standby or hot_successor continuity. The successor serves the same model, satisfies the same pin and is run by another operator or at another site. It receives KV-cache deltas sealed to its key and refuses checkpoints built with different weights.
• Hedging and failure detection. The router hedges at the primary's observed p95 latency, clamped between 40 and 500 ms, with at most one hedge per request. Providers whose signed heartbeats stop for six seconds are skipped.
• Streaming failover. If a provider drops mid-stream, the gateway resumes on another provider serving the same model with the same seed and the text emitted so far, as a single bounded retry.
• Private prefix affinity. Providers announce only a count of warm prefixes. Affinity is learned through probes salted per caller, and probe results break ties without overriding price.