Tenzro Train

Production decentralized training runs in 2026 use Python + PyTorch for the inner training engine. Rust ML frameworks (Candle, Burn, tch-rs) exist but no production decentralized training project picks them. PyTorch's FSDP2 / DTensor / torch.compile / Hivemind ecosystem and per-architecture implementations (TimesFM, Chronos, Qwen, ViT, etc.) are irreplaceable for training.

Rust shines at the protocol / orchestration layer — the same split adopted by open decentralized-training protocols. So Tenzro Train mirrors that: Rust owns OuterGradient, Fragment, SyncRound, aggregation rules, OuterOptimizer, syncer state machine, on-chain commitments, fraud-proof verification, RPC, CLI. Python owns the inner training loop.

Open — anyone can join; aggregation restricted to Mean and LoraAlternating (the alternating-freeze rule for LoRA/QLoRA adapter runs). Verified — stake + KYC tier; admits the Byzantine-robust aggregators as well (TrimmedMean, CoordinateMedian, Krum). Confidential — Verified plus sealed-shard ingestion via HPKE RFC 9180 base-mode unwrap and AES-256-GCM shard decryption inside the trainer's TEE enclave.

validate_aggregation_rule_for_tier() is the admission gate. validate_confidential_enrollment() enforces attestation ↔ enclave_pubkey ↔ enclave_measurements_hex parity at enroll time.

Phase 2c training-side multi-syncer coordination uses the k-of-N witness committee pattern established by production decentralized-training protocols. committee::select_witness_committee deterministically selects a quorum from registered syncers using finalized block hash as chain entropy.

SyncerState::finalize_round is idempotent: redundant submissions from concurrent witnesses for the same (round, state_root) return Ok; conflicting state_roots return ConflictingFinalize for fork detection.

When the committee cannot assemble a quorum within grace_window_ms, SyncerState::build_nec_sync_round emits a no-endorsement certificate carried in SyncRound.no_quorum_witnesses; the run advances to round+1 carrying forward the prior state_root.

SealedDatasetManifest carries SealedShardEnvelope { trainer_did, shard_index, shard_ciphertext_hash, shard_ciphertext_bytes, wrapped_data_key, wrap_alg = "hpke-x25519-hkdf-sha256-aes-256-gcm", enclave_pubkey, enclave_measurements_hex, created_at }.

Manifest hash binds into TrainingTaskSpec::dataset_ref as tee://<64-hex>. The Python trainer unwraps the data key via pyhpke inside the enclave, decrypts shards with cryptography (AES-256-GCM), and trains on the plaintext — without exposing the shards to the host.

Timeseries — TimesFM-class 200M models (Phase 1 priority — cheapest complete run).

Language — Qwen 3 0.6B default via transformers.AutoModelForCausalLM. Catalog-member LM families swappable via architecture.metadata.hf_repo: Qwen 2/3/3.5/3.6, Gemma 3/4, Mistral, Phi 3, DeepSeek V3, Granite, Granite-H. Language tasks additionally support an RL post-training objective: the task spec declares RlPostTraining and the trainer runs a GRPO inner loop — per step it samples a rollout group from one shard prompt, scores completions with the sponsor-referenced reward callable, computes group-relative advantages, and takes one optimizer step on the clipped surrogate with a KL penalty against the sampling-time policy. No value model, no frozen reference copy, and the outer-gradient contract is unchanged.

Vision — timm ViT-B/16 default matching the inference-side DINOv3/SigLIP2/CLIP-B/16 family. Swappable via architecture.metadata.timm_model. ImageFolder shard layout with Pillow decode and ImageNet normalization.

Five mechanisms, each declared on the training task and enforced by the syncer, cut the per-round transfer for larger models:

Gradient quantization— blockwise symmetric Int8 (4× smaller than f32) or Int4 (~8×) on the outer-gradient payload; the syncer rejects submissions whose declared quantization differs from the task's. Streaming synchronization — fragments partition into shards and each round synchronizes one shard, overlapping outer sync with inner compute. Delayed application — the aggregate from round r is applied at round r+1 instead of stalling the inner loop. Adaptive outer learning rate — the outer Nesterov step scales with the pairwise cosine agreement of submitted gradients. Pipeline-parallel trainer groups — trainers enroll as (group, stage) pairs so a group jointly holds one replica and quorum counts groups per fragment; no single trainer needs to fit the model.

The Python reference trainer supports Muon as the inner optimizer (matrix parameters take the Newton-Schulz-orthogonalized update, the rest fall back to AdamW), which converges with fewer outer synchronizations than AdamW at equal quality.

Node RPC namespace tenzro_training_*: postTask / listRuns / getRun / getReceipt / enrollTrainer / submitOuterGradient / finalizeRound / installSealedManifest / getSealedManifest. CLI: tenzro train {post-task, list-runs, get-run, get-receipt, enroll-trainer, submit-gradient, finalize-round, install-sealed-manifest, get-sealed-manifest}.

Gossipsub topics tenzro/training (training-event broadcast) and tenzro/training/syncer (cross-syncer round-state sync) registered with GossipTransport. Cross-region multi-syncer fan-out remains Phase 2c.

Phase 1 baseline complete plus tier × aggregation policy, gossipsub cross-syncer wiring, real per-modality adapters, Confidential-tier sealed-shard ingestion, training-side multi-syncer coordination (k-of-N witness committee with idempotent on-chain finalize), and the communication mechanisms above: quantized gradients, streaming synchronization, delayed application, adaptive outer learning rate, pipeline-parallel trainer groups, and the Muon inner optimizer.