Distributed Training Across Clusters: What Works Over the WAN Today

Large-model training has assumed one building. Data-parallel training synchronises gradients at every step, which needs the kind of bandwidth only a single fabric provides. That assumption drives where AI factories are built, how big a single site must be, and who can take part.

Over the last three years a line of research has loosened it. Training can now be split across sites that talk rarely and compress what they say. This post summarises what has been shown, what still limits it, and the parts of the problem an open network of machines is suited to.

The core idea: synchronise rarely

DiLoCo reframed distributed training as two loops. Each worker, which can be a whole cluster, runs many local optimiser steps on its own data. Only then do workers exchange a pseudo-gradient, the difference between where they started and where they ended, and an outer optimiser combines them. On C4, eight workers matched fully synchronous training while communicating 500 times less.

The idea has since been pushed in three directions.

Overlap the communication. Streaming DiLoCo synchronises subsets of parameters on a schedule and overlaps that communication with computation, reporting a two-orders-of-magnitude reduction in required bandwidth at similar quality for billion-parameter models.

Compress what is sent. SparseLoCo combines top-k sparsification with 2-bit quantisation of the pseudo-gradients, reaching 97 to 99% sparsity while ending with lower loss than dense DiLoCo, across dense models from 178M to 2B parameters and a small mixture-of-experts model.

Run it in the open. INTELLECT-1 trained a 10B-parameter model on 1 trillion tokens across up to 14 concurrent nodes on three continents, with 30 independent compute providers joining and leaving, reporting 83 to 96% compute utilisation and a 400 times reduction in communication bandwidth against data-parallel training. Covenant-72B pre-trained a 72B-parameter model on about 1.1 trillion tokens over the internet with SparseLoCo, among peers free to join and leave.

What still binds

The progress is real, and the constraints have moved rather than vanished.

• Each worker still needs a fast interior. Low-communication methods remove the need for a fabric between sites, not within them. A worker large enough to hold a model replica still needs tight interconnect inside.
• Model FLOPS utilisation stays modest. INTELLECT-1 reported 36 to 41% MFU. Wide-area training trades some efficiency for reach.
• Heterogeneity and churn are engineering, not theory. Workers with different speeds, peers that drop out mid-round, and stragglers all need handling that a single-site job never sees.
• Trust is the open problem. When peers are independent, the outer loop must decide whose contribution to accept. A worker can submit garbage, a stale update or someone else's update. Open runs have handled this with allow-lists or with dedicated validation schemes, each with trade-offs.

Where an open network helps

The last point is where a network of identified, accountable machines has something to offer. Wide-area training needs four things beyond the optimiser:

1. Identities that cannot be multiplied, so one operator cannot appear as many workers.
2. Payment that tracks delivered work, released per round rather than trusted up front.
3. Evidence that a contribution was computed as claimed.
4. Transport for large artefacts (checkpoints, adapters, compressed updates) without a central bucket.

Tenzro supplies these as general primitives rather than a training-specific protocol.

How training works on Tenzro today

Training is not a protocol feature. A training job runs in a harness as a task whose rounds are milestones, each paid from an escrow when its round is delivered. The chain holds the money, checks the evidence the harness asks it to check, and serves the trained model once it is published to the catalog like any other model.

On the node side, training is compute an operator sells, like serving. An operator loads a catalog language model for training, and the node trains low-rank adapters on its CUDA device. A machine without CUDA refuses the load; training never silently moves to the CPU.

The part that matters for trust is determinism. Each training round runs in deterministic mode: every operation of every step must run on the CUDA device and belong to that device's audited deterministic kernel set, or the round is refused. A round answers with a commitment covering the base weights, the tokenised records, the recipe, the seed, the kernel class, the step count, the starting and ending state roots and the step log, plus each step's loss and the adapter's hash. These fold into a single result digest, which the machine attests with its hardware identity key.

Another machine of the same kernel class, meaning the same GPU model, that replays the round with the same inputs reaches the same digest bit for bit. A harness pays a round's milestone when two machines' attestations over the same digest both verify. That is verification by replication, not by proof, and it is cheap to reason about: two independent, hardware-identified machines agreeing exactly is strong evidence that neither cheated.

Large artefacts move as content-addressed blobs over the nodes' peer-to-peer endpoints, verified chunk by chunk against their hashes, so a checkpoint or adapter can come from whichever peer holds it.

What this means for AI factory planners and policy makers

Low-communication training changes the siting question from "one site big enough for the whole run" towards "several sites, each big enough for a replica, connected by ordinary links". That favours regional capacity, energy-constrained locations and sovereign compute that can still cooperate on a shared model. It does not remove the need for strong single sites; it changes how many are needed and how close together they must be. For policy makers, it means training capacity can be pooled across borders while data and weights stay where rules require.

Where Tenzro fits

Built: hardware-rooted machine identities, escrowed per-round milestones, deterministic LoRA training rounds with hardware-attested result digests that a second machine of the same GPU model can reproduce exactly, evidence checking on chain, and content-addressed transfer of artefacts. Not built as part of the protocol: a DiLoCo-style outer loop for pre-training across independent peers. Such a loop could run as a harness on these primitives; we have not shipped one. The network is live in preview; governance decides the steps to testnet and mainnet.