Tenzro Train.

Decentralized training splits cleanly into a Rust protocol layer (aggregation, sync rounds, on-chain commitments) and a Python reference trainer (PyTorch FSDP2 + Hivemind + safetensors). Same architectural split as Prime Intellect's INTELLECT-2 and Nous's Psyche.

STATUS: Phase 1+ shipped
PROTOCOL CRATE: tenzro-training
PYTHON PKG: integrations/trainer
TIER: Open / Verified / Confidential

Two-layer architecture

tenzro-training (Rust) owns the protocol: OuterGradient, Fragment, SyncRound, aggregation rules (Mean / TrimmedMean / CoordinateMedian / Krum), the OuterOptimizer (Nesterov SGD), the syncer state machine, TrainingTaskSpec, TrainingReceipt, gossip topic handling, on-chain commitments, fraud-proof verification, RPC, and CLI. No tensor library lives in the Rust workspace.

integrations/trainer (Python) wraps PyTorch FSDP2 + Hivemind + safetensors. Per-modality inner training loops use the SOTA Python library: transformers for language, gluonts for timeseries, timm for vision. The trainer talks to the Rust syncer over JSON-RPC and the gossip topics.

Why Python for the inner loop

Every production decentralized training run in 2026 (Prime Intellect's INTELLECT-1/2/3, Nous Research's Hermes 4.3 on Psyche/DisTrO, OpenDiLoCo) uses Python + PyTorch for the inner training engine. Rust ML frameworks exist (Candle, Burn, tch-rs) but no production decentralized training project picks them — PyTorch's FSDP2/DTensor/torch.compile/Hivemind ecosystem and the long tail of per-architecture implementations (TimesFM, Chronos, Qwen, Gemma, Mistral, ViT, DINOv3, …) are irreplaceable for training. Rust shines at the protocol/orchestration layer.

Witness committee + idempotent finalize

Phase 2c ships training-side multi-syncer coordination as a k-of-N witness committee with idempotent on-chain finalize and a no-quorum-cert carry-forward. This mirrors the Psyche (Solana, 2025) and INTELLECT-2 (May 2025) production pattern.

SyncerState::finalize_round is idempotent: redundant submissions from concurrent witnesses for the same (round, state_root) return Ok. Conflicting state_roots return TrainingError::ConflictingFinalize for fork detection. When the committee cannot assemble a quorum within grace_window_ms, SyncerState::build_nec_sync_round produces a no-endorsement-cert sync round and the run advances to round+1 carrying forward the prior state_root.

Tier-gated aggregation

validate_aggregation_rule_for_tier() admits aggregation rules per trust tier:

Open — Mean only
Verified — Mean, TrimmedMean, CoordinateMedian, Krum
Confidential — same set, with TEE-sealed shards

Confidential sealed shards

SealedDatasetManifest carries SealedShardEnvelope rows: shard_ciphertext_hash, shard_ciphertext_bytes, wrapped_data_key, wrap_alg = "hpke-x25519-hkdf-sha256-aes-256-gcm", enclave_pubkey, enclave_measurements_hex, created_at. validate_confidential_enrollment() enforces attestation ↔ enclave_pubkey ↔ enclave_measurements_hex parity at enroll time.

RPC + CLI surface

# JSON-RPC namespace: tenzro_training_*
postTask | listRuns | getRun | getReceipt
enrollTrainer | submitOuterGradient | finalizeRound
installSealedManifest | getSealedManifest

# CLI
tenzro train post-task ...
tenzro train enroll-trainer ...
tenzro train submit-gradient ...
tenzro train finalize-round ...
tenzro train install-sealed-manifest ...

Reference adapters

The Python reference trainer ships real adapters: timeseries (TimesFM-class 200M), language (Qwen 3 0.6B default — any catalog-member LM family is swappable via architecture.metadata.hf_repo: Qwen 2/3/3.5/3.6, Gemma 3/4, Mistral, Phi 3, DeepSeek V3, Granite, Granite-H), vision (timm ViT-B/16 default matching the inference-side DINOv3/SigLIP2/CLIP-B/16 family, swappable via architecture.metadata.timm_model).

← All docs