Tenzro Train

Every production decentralized training run in 2026 — Prime Intellect's INTELLECT-1/2/3, Nous Research's Hermes 4.3 on Psyche/DisTrO, OpenDiLoCo — uses Python + PyTorch for the inner training engine. Rust ML frameworks (Candle, Burn, tch-rs) exist but no production decentralized training project picks them. PyTorch's FSDP2 / DTensor / torch.compile / Hivemind ecosystem and per-architecture implementations (TimesFM, Chronos, Llama, ViT, etc.) are irreplaceable for training.

Rust shines at the protocol / orchestration layer — Prime Intellect's protocol repo follows the same split. So Tenzro Train mirrors that: Rust owns OuterGradient, Fragment, SyncRound, aggregation rules, OuterOptimizer, syncer state machine, on-chain commitments, fraud-proof verification, RPC, CLI. Python owns the inner training loop.

Open — anyone can join; aggregation restricted to Mean. Verified — stake + KYC tier; admits all four aggregators (Mean, TrimmedMean, CoordinateMedian, Krum). Confidential — Verified plus sealed-shard ingestion via HPKE RFC 9180 base-mode unwrap and AES-256-GCM shard decryption inside the trainer's TEE enclave.

validate_aggregation_rule_for_tier() is the admission gate. validate_confidential_enrollment() enforces attestation ↔ enclave_pubkey ↔ enclave_measurements_hex parity at enroll time.

Phase 2c training-side multi-syncer coordination ships the k-of-N witness committee pattern mirroring Nous Psyche (Solana, 2025) and Prime Intellect INTELLECT-2 (May 2025). committee::select_witness_committee deterministically selects a quorum from registered syncers using finalized block hash as chain entropy.

SyncerState::finalize_round is idempotent: redundant submissions from concurrent witnesses for the same (round, state_root) return Ok; conflicting state_roots return ConflictingFinalize for fork detection.

When the committee cannot assemble a quorum within grace_window_ms, SyncerState::build_nec_sync_round emits a no-endorsement certificate carried in SyncRound.no_quorum_witnesses; the run advances to round+1 carrying forward the prior state_root.

SealedDatasetManifest carries SealedShardEnvelope { trainer_did, shard_index, shard_ciphertext_hash, shard_ciphertext_bytes, wrapped_data_key, wrap_alg = "hpke-x25519-hkdf-sha256-aes-256-gcm", enclave_pubkey, enclave_measurements_hex, created_at }.

Manifest hash binds into TrainingTaskSpec::dataset_ref as tee://<64-hex>. The Python trainer unwraps the data key via pyhpke inside the enclave, decrypts shards with cryptography (AES-256-GCM), and trains on the plaintext — without exposing the shards to the host.

Timeseries — TimesFM-class 200M models (Phase 1 priority — cheapest end-to-end).

Language — Qwen 3 0.6B default via transformers.AutoModelForCausalLM. Catalog-member LM families swappable via architecture.metadata.hf_repo: Qwen 2/3/3.5/3.6, Gemma 3/4, Mistral, Phi 3, DeepSeek V3, Granite, Granite-H.

Vision — timm ViT-B/16 default matching the inference-side DINOv3/SigLIP2/CLIP-B/16 family. Swappable via architecture.metadata.timm_model. ImageFolder shard layout with Pillow decode and ImageNet normalization.

Node RPC namespace tenzro_training_*: postTask / listRuns / getRun / getReceipt / enrollTrainer / submitOuterGradient / finalizeRound / installSealedManifest / getSealedManifest. CLI: tenzro train {post-task, list-runs, get-run, get-receipt, enroll-trainer, submit-gradient, finalize-round, install-sealed-manifest, get-sealed-manifest}.

Gossipsub topics tenzro/training (training-event broadcast) and tenzro/training/syncer (cross-syncer round-state sync) registered with GossipTransport. Cross-region multi-syncer fan-out remains Phase 2c.

Phase 1 baseline complete plus Phase 2a (tier × aggregation policy), Phase 2b (gossipsub cross-syncer wiring), Phase 3 (real per-modality adapters), Phase 4 (Confidential-tier sealed-shard ingestion), Phase 2c (training-side multi-syncer coordination). Rust protocol layer 54/54 tests. Python reference trainer 14/14 tests.

Phase D (TEE-seal single-key bridge signer) shipped via EvmTransactionSigner::with_tee_sealed (SNP + TDX paths). Phase D CGGMP24 t-of-n remains blocked on upstream LFDT-Lockness/fast-paillier#23.