Tenzro Train.

Two layers. The Rust crate owns the protocol: OuterGradient, Fragment, SyncRound, aggregation rules, the outer optimizer, the syncer state machine, on-chain commitments, fraud proofs, RPC, CLI. The Python trainer owns the inner loop: PyTorch FSDP2 + Hivemind + safetensors with per-modality adapters.

Mean, LoraAlternating, TrimmedMean, CoordinateMedian, Krum. Tier-gated: Open admits Mean and LoraAlternating (the alternating-freeze rule for LoRA/QLoRA adapter runs); Verified and Confidential add the Byzantine-robust rules. Nesterov SGD outer optimizer.

k-of-N witness committee with idempotent on-chain finalize. No-endorsement certificate carries the run forward when a quorum cannot be assembled inside grace_window_ms.

Open-tier gradients carry a TOPLOC-class ActivationCommitment — per-step loss trajectory plus top-k delta probes, hash-bound into the gradient signature. The syncer validates the structure fail-closed on submission; a challenger can re-execute the inner loop and compare within tolerance bands sized for cross-GPU nondeterminism. A failed challenge evicts and slashes.

Trainers run on heterogeneous, variably-reachable hardware, so a round cannot wait for every enrolled trainer. Each round is bound by a wall-clock grace window: the syncer finalizes once it holds enough gradients, rather than blocking on the slowest trainer. decide_round returns the current decision — wait (with remaining grace-window ms), finalize (enough gradients arrived, this round closes), or no_quorum (the window elapsed without a quorum; the run advances carrying the prior state root forward under a no-endorsement certificate).

tenzro train decide-round --task-id <run_id>
# Decision: wait | finalize | no_quorum

timeseries   TimesFM-class 200M
language     Qwen 3 0.6B default, swappable via metadata
vision       timm ViT-B/16 default, swappable via metadata

Dataset shards travel over the network's own content-addressed blob store: publish once, reference by hash. tenzro://blob/<hash> URIs are fetched by the trainer through the local node (iroh-blobs, BLAKE3-verified on transfer) and cached locally, so repeated rounds hit the network once. ipfs:// and ar:// are supported alternatives resolved through HTTP gateways, alongside plain http(s):// and file:// paths. Confidential-tier shards bypass all of this — they arrive sealed and are unwrapped inside the trainer's enclave.

tenzro iroh publish --file shard-3.parquet
# tenzro://blob/<blake3-hash>

tenzro-trainer run --task-id <run_id> --trainer-did <did> \
  --shard-uri tenzro://blob/<blake3-hash>

Five mechanisms, each declared on the training task and enforced by the syncer, cut the per-round transfer for larger models: blockwise gradient quantization (Int8, 4× smaller than f32; Int4, ~8×), streaming synchronization (fragments partition into shards; each round syncs one shard, overlapping outer sync with inner compute), delayed application (the aggregate from round r applies at round r+1), adaptive outer learning rate (the Nesterov step scales with the pairwise cosine agreement of submitted gradients), and pipeline-parallel trainer groups (trainers enroll as (group, stage) pairs so no single trainer needs to fit the model). The Python trainer supports Muon as the inner optimizer, which converges with fewer outer synchronizations than AdamW at equal quality.

The reference trainer scales from one GPU to a multi-GPU host with no configuration. Under torchrun, the language adapter shards the model with FSDP2 (per-parameter DTensor sharding, bf16 compute with fp32 gradient reduction); every rank runs the training loop, only rank 0 speaks JSON-RPC to the node, and each rank samples distinct batches. Attention dispatches to FlashAttention-2 when flash_attn and a CUDA device are present (SDPA otherwise), and architecture.metadata.fp8 opts eligible linear layers into torchao FP8 training on Ada/Hopper-class GPUs (compute capability ≥ 8.9). Muon runs over the sharded model: gradients gather for Newton-Schulz, momentum stays sharded, the orthogonalized update distributes back.

torchrun --nproc-per-node 8 -m tenzro_trainer.cli run \
  --task-id <run_id> --trainer-did <did> --shard-uri file:///data/shard

tenzro train post-task
tenzro train list-runs
tenzro train get-run --id <run_id>
tenzro train enroll-trainer
tenzro train submit-gradient
tenzro train finalize-round
tenzro train decide-round --task-id <run_id>