Distributed training on Tenzro

Abstract

Distributed training on Tenzro is built on the network's public interfaces: the public RPC, the SDKs and escrow. A sponsor posts a training run and funds each round in escrow. Trainers on independent hardware run an inner training loop locally and exchange compressed outer gradients once per round. A committee of syncers aggregates the gradients under a rule suited to the run's trust tier, publishes a signed round record and releases the round's escrow. Every round is replayable from its seed, so anyone can recompute what the round should have produced and challenge a contribution that does not match. Alongside supervised and reinforcement-learning objectives, a chat fine-tuning objective trains models on agent trajectories, with the loss on the assistant's turns and tool calls. Contributions settle in TNZO through ordinary escrows.

1. Introduction

Training a model has usually meant renting a large, tightly connected cluster from a single provider. That limits who can train, concentrates cost and leaves no public record of what was trained on what.

Low-communication training methods change the constraint. If each participant trains for many local steps between synchronisations and sends compressed updates, training can run over ordinary internet links between machines that do not trust each other. Distributed training on Tenzro turns that into an application on a public ledger: runs, enrolment, rounds, aggregation and verification in the training software, and payment in escrows on chain. It uses only the network's public interfaces, so anyone can run it, fork it or build a different trainer beside it.

It is aimed at the Machine Economy. Its first job is to make models better at acting as agents, by training them on the trajectories agents actually produce.

2. Roles

Sponsor, trainer and syncer are roles in the training software, not on the chain. On chain they are ordinary accounts: the chain sees escrows being created and released. Every participant acts with a hardware-rooted DID, and a trainer enrols as a did:tenzro:machine identity.

3. Runs

A run spec states everything needed to run and verify training:

• the base model, by manifest root, and the modality (language, vision or time series);
• the objective (section 6);
• the dataset reference and how it is sharded;
• the trust tier and aggregation rule;
• the round schedule: inner steps per round, outer optimiser and learning rate;
• the communication settings: gradient quantisation, sparsification and streaming;
• the budget, the per-round payment terms and the syncer panel that releases each round.

A run is a task with one milestone per round. The spec is a document signed by the sponsor's account. Each round is funded by an escrow from the sponsor that carries the round's reference, a hash of the run and the round number, so anyone can find a run's payments again from public escrow reads.

4. Rounds

Each round runs the same cycle.

1. Seed. The round's seed is derived from the run, the round number and entropy from finalised chain state. Nobody can choose it in advance.
2. Assignment. From the seed, each trainer's data shards, sampling order and any randomised choices in training are fixed. The syncer committee for the round is selected from registered syncers with the same seed.
3. Inner loop. Each trainer starts from the round's committed model state and trains locally for the configured number of steps.
4. Outer gradient. Each trainer computes the difference between its local result and the round's starting state, compresses it, and submits it with a commitment. The commitment includes the payload hash and signed probes of the training trajectory, such as loss values and activation samples at checkpoints the seed selects.
5. Aggregation. The syncer committee aggregates the submitted gradients with the run's aggregation rule and applies the outer optimiser step.
6. Finalisation. The committee finalises the round by signing a round record that commits the new state root. Finalisation is idempotent: redundant submissions of the same result are accepted, and conflicting results are rejected.
7. Settlement. A threshold of the committee releases the round's escrow to the accepted contributors, and the release is recorded on the ledger.

If the committee cannot form a quorum within the round's grace window, it issues a no-endorsement certificate and the run advances, carrying the previous state forward. A missing syncer delays a round; it does not stall the run.

5. Replayable rounds

Every round is replayable from its seed. Everything that determines a round's outcome is either in the run's signed records or derived from the seed:

• the starting state root;
• the shard assignment and sampling order;
• the randomised choices in training and the positions of trajectory probes;
• the run's configuration;
• the gradients submitted, by hash;
• the aggregation rule and the resulting state root.

Replayability serves three purposes.

• Verification. Anyone can recompute a trainer's contribution for a round and compare the recomputed trajectory probes with the committed ones. A contribution that does not match can be challenged. A successful challenge excludes the contribution, and the committee does not pay it.
• Provenance. A model's training history is a chain of signed round records, each tied to an escrow release on the ledger. A relying party can check which data, objective and contributions produced a given state.
• Recovery. A new syncer or a restarted trainer can reconstruct the run's state from the round records, the ledger and the stored payloads.

Payloads are content-addressed. Gradients and model states are stored and fetched by hash, and each chunk is verified as it arrives.

6. Objectives

6.1 Chat fine-tuning for agent trajectories

The chat fine-tuning objective trains a language model on conversations in the OpenAI chat shape: a list of messages and the tools available. Each assistant turn, including its tool calls, is one training example, rendered with the model's own chat template against the conversation so far. The context is masked; the loss falls only on what the assistant said and did.

This is the objective for teaching a model to act as an agent: take corrected runs of an agent, including its tool calls, and fine-tune the model so it behaves well with a minimal harness. The objective supports LoRA and QLoRA adapters and alternating-freeze aggregation for adapter runs.

A dataset shard is a JSONL file of conversations:

{"messages": [
  {"role": "system", "content": "You are a procurement agent."},
  {"role": "user", "content": "Rent one GPU for two hours at the index price."},
  {"role": "assistant", "tool_calls": [{"type": "function", "function": {"name": "get_price_index", "arguments": "{\"class\": \"workstation\"}"}}]},
  {"role": "tool", "content": "{\"price\": \"...\"}"},
  {"role": "assistant", "content": "Booked at the index price."}
], "tools": [{"type": "function", "function": {"name": "get_price_index"}}]}

6.2 Reinforcement learning

The reinforcement-learning objective samples a group of completions per prompt, scores them with the reward function the sponsor references, computes group-relative advantages, and takes an optimiser step on a clipped objective with a KL penalty. It needs no value model and leaves the outer-gradient exchange unchanged.

7. Trust tiers and aggregation

Robust aggregation limits what a minority of dishonest trainers can do to a round. Replay-based challenges catch contributions that do not follow from the committed data, and a contribution that is caught is not paid.

7.1 Confidential runs

For a confidential run, the sponsor seals each data shard to a trainer enclave's key using HPKE, and publishes a sealed dataset manifest that names the expected enclave measurement and key. A trainer enrols with an attestation report. The report is verified against pinned vendor roots, simulated reports are refused, and the report must commit to the enclave key and measurement from the sponsor's manifest, not values the trainer supplies. The shards are decrypted only inside the attested enclave; the host never sees the plaintext.

8. Communication efficiency

Each mechanism is declared in the run spec and enforced by the syncers.

• Quantisation. Outer gradients are quantised blockwise to Int8 or Int4. A submission whose quantisation differs from the run's is rejected.
• Sparsification. Top-k sparsification with error feedback sends only the largest components each round and carries the remainder forward.
• Streaming synchronisation. The model is partitioned into fragments; each round synchronises a subset, overlapping communication with computation.
• Delayed application. The aggregate from one round is applied at the next, so the inner loop never waits for synchronisation.
• Adaptive outer learning rate. The outer step scales with the agreement between submitted gradients.
• Pipeline groups. Trainers can enrol as stages of a pipeline, so a group jointly holds one replica and no single trainer needs to fit the whole model.
• Inner optimiser. The reference trainer supports Muon for matrix parameters, with AdamW for the rest, which reduces the number of outer synchronisations needed.

9. Economics

• The sponsor's budget is held in escrow, one escrow per round, and released per round against accepted contributions.
• Trainers are paid for accepted gradients; syncers for finalised rounds, on the terms in the run spec.
• What a round never releases returns to the sponsor when its escrow expires.
• Payments settle in TNZO. See TNZO and Settlement for the Machine Economy.

10. Interfaces

Training needs nothing from a node beyond its public interfaces:

Specs, enrolments, gradients and round records are signed documents the training software keeps and serves; payloads are content-addressed. Every state-changing call on chain is signed by the acting account. The same pattern serves any task market: see Task marketplace.

11. Design choices

• Built on public interfaces. Runs, rounds, aggregation and commitments live in the training software; the chain holds the money and the identities. Anyone can run, fork or replace the trainer without a change to the protocol.
• Low communication by default. Rounds exchange compressed outer gradients rather than per-step gradients, so training runs over ordinary links.
• Evidence over trust. Seeds, commitments and replay make a contribution checkable after the fact, and a contribution that fails replay is not paid.
• One identity and one ledger. Trainers, syncers and sponsors use the same hardware-rooted identities and settle in the same escrows as every other service.

12. Conclusion

Distributed training on Tenzro lets anyone with hardware contribute, lets anyone check how a model was trained, and gives agents a direct path to better models through the chat fine-tuning objective. For hands-on detail, see Task marketplace and Agents.