Train and finalize

tenzro train post-task \
  --modality timeseries \
  --architecture timesfm-2.5 \
  --tier open \
  --aggregation mean \
  --target-rounds 64

tenzro train enroll-trainer --task <task-id>

The Python trainer wraps PyTorch FSDP2 and Hivemind. It talks to the Rust syncer over JSON-RPC and the gossip topics.

pip install "tenzro-trainer"
tenzro-trainer run --task <task-id> --shard <shard-uri>

The trainer computes an outer gradient, serializes it to safetensors, hashes it (SHA-256), and publishes via IrohGradientStore. The on-wire submission is small — just the SHA-256 + signature.

tenzro train submit-gradient --task <task-id> --round <r>

SyncerState::finalize_round is idempotent — k-of-N witnesses can submit the same (round, state_root) concurrently with no double-finalize hazard. Conflicting state_roots return TrainingError::ConflictingFinalize for fork detection. If the committee cannot assemble a quorum within grace_window_ms, build_nec_sync_round emits a no-endorsement-cert sync round and the run advances carrying forward the prior state_root.

Trainers run on heterogeneous, variably-reachable hardware, so a round does not wait for every enrolled trainer — it is bound by a wall-clock grace window. Ask the syncer what it will do with the current round instead of guessing: decide-round returns wait (with remaining grace-window ms), finalize (enough gradients arrived), or no_quorum (the window elapsed; the run advances carrying the prior state root forward under a no-endorsement certificate).

tenzro train decide-round --task-id <task-id>
# Decision: wait | finalize | no_quorum
# wait     -> Grace window remaining (ms): <n>
# finalize -> Round: <r>
# no_quorum -> Round: <r>

tenzro train get-receipt --task <task-id> --round <r>