An AI factory is a capital asset whose main cost runs whether or not it is working. Accelerators depreciate on a calendar, power contracts carry commitments, and staff and cooling are largely fixed. What changes with demand is how much useful work comes out. For anyone building, financing or running one, the decisive operating number is served work per installed accelerator-hour, or simply utilisation.
This article looks at where utilisation is lost in practice, why a single operator struggles to recover it alone, and what changes when demand is matched across many operators.
Where utilisation is lost
Utilisation losses fall into four groups, and each needs a different remedy.
Demand shape. Inference follows people, and people sleep. The Aryl study of a production inference cluster of about 4,000 GPUs measured a daily peak-to-trough ratio of about 2.2 and average utilisation of about 65%. The cluster was sized for a four-hour evening peak. This is not mismanagement. A cluster that must meet its latency targets at peak will be partly idle off-peak.
Reliability. Large systems fail often. During a 54-day snapshot of Llama 3 405B pre-training on up to 16,000 H100 GPUs, Meta recorded 466 job interruptions, 419 of them unexpected, with about 78% attributed to confirmed or suspected hardware issues, according to the Llama 3 paper. Meta still achieved more than 90% effective training time, but only through substantial engineering. Every interruption, failover and restart costs capacity, and smaller operators feel it more.
Fragmentation. Capacity comes in awkward shapes. A server may have enough memory for a mid-sized model but not a large one, or spare memory sitting behind a busy compute unit. Large mixture-of-experts models may need more memory than any single machine an operator owns. Capacity that cannot be combined goes unused.
Tail latency. Strict latency targets force operators to keep headroom. As Dean and Barroso set out, the slowest responses in a large system dominate user experience, and the usual defence is spare capacity. Headroom held for the tail is utilisation given up.
Why one operator cannot fix this alone
An operator can improve its own scheduling. It can move batch work into troughs, as Aryl proposes by loaning inference capacity to training jobs, or reserve capacity for internal teams. These help, but they share a limit: they draw only on the operator's own demand, which has the same shape as its own peak.
Demand aggregated across many operators and many buyers looks different. Time zones offset each other. Batch embedding jobs, evaluation runs, agent workloads and interactive chat each have their own rhythm. One operator's quiet hour is another region's busy hour. The statistical argument is simple: the sum of many imperfectly correlated demand curves is flatter than any one of them. A flatter curve lets each operator run closer to its capacity without breaking its latency targets.
So the useful question for an AI factory is not only "how do I fill my own capacity?" but also "how can my spare capacity be reached by demand I do not own?"
What demand matching requires
Matching demand across operators is not a load balancer with more endpoints. It needs four things that a single-operator platform takes for granted.
Live, signed load signals. A router must know which providers have a model warm, how many requests they are carrying and how long a cold start would take. These signals must be signed by the machine that sends them, or a provider can attract traffic it cannot serve.
Prices that respond to load. If prices never move, quiet capacity stays quiet. A per-period rate that rises when utilisation runs above target and falls below it, with bounded steps and smoothing so one burst does not whipsaw it, gives buyers a reason to shift flexible work into troughs.
Observed, not declared, performance. Providers announce their own prices and latencies, and announcements are the one source that cannot be trusted. A price and latency index built from measurements by independent routing nodes is much harder to game.
Pooling fragmented memory. Mixture-of-experts models activate only a few experts per token. That makes it possible to spread experts across machines that individually cannot hold the full model, so memory that would otherwise sit unused can serve frontier-scale models.
Reliability is a utilisation lever too
The other side of utilisation is how much capacity you hold in reserve. Fast failure detection, tail-latency hedging and pre-assigned successors for long sessions all reduce the headroom each operator must keep. If a peer in the same jurisdiction can take over a session within seconds, each operator needs less spare capacity of its own. Reliability then becomes something the network shares, not something each factory pays for alone.
What this means for AI factory investors and operators
For those evaluating AI factories, utilisation assumptions deserve the closest scrutiny in any plan. Ask what the demand curve looks like hour by hour, not just in total, and what happens to the capacity in its troughs. Ask how failures, cold starts and tail-latency headroom are accounted for. And ask whether the facility can sell spare capacity to demand beyond its anchor tenants, with verifiable service levels, without new contracts for each buyer.
For operators, the practical step is to treat open demand matching as one channel alongside contracted capacity, not instead of it. Contracts cover the base. An open network can absorb the troughs and handle the overflow.
Where Tenzro fits
Tenzro is an open network where machines sell AI work. It is live in preview, and governance decides the steps to testnet and mainnet. The demand-matching mechanisms described here are built:
- Signed load signals. Providers send a signed heartbeat every two seconds. Discovery reports per-model state (cold, warming, warm), requests in flight, admission ceiling and warm-up queue depth. Providers whose heartbeats stop for six seconds are routed around.
- Load-responsive rates. Compute rental can use a network-dynamic rate that tracks smoothed utilisation against a 50% target, with steps bounded at 12.5% per period and clamped between a floor and a ceiling. Providers can also set a fixed rate.
- Observed price feed. After each routed call, the routing node signs the price, latency, token counts and outcome it measured. Only observations from active validators count, and an entry needs at least two distinct validators before it enters the index. The index is a median of per-provider medians of price per compute-second.
- Routing. Strategies include lowest price, lowest latency, highest standing and a weighted score. A hedge request goes to a second provider at the primary's observed p95 latency, with at most one hedge per request.
- Distributed MoE serving. Independent nodes each hold a subset of a model's experts, and a router node gates tokens and dispatches expert batches to the holders. Cross-node serving has been exercised on the live network.
- Session continuity. Sessions can name a standby or hot successor in the same jurisdiction, which receives sealed KV-cache checkpoints and takes over when the primary fails.
Work settles on chain in TNZO, the network's unit for pricing and settling work. Fees are set in USD terms, and buyers are not required to hold it.
