Serve a distributed MoE model

Start from the catalog-side shape — how many experts the model has, how many are active per token, and the layer layout — so you know which experts you can hold.

tenzro moe catalog-shape --model qwen3.5-397b-a17b

Slice a layer's experts out of the catalog checkpoint and block-quantize each projection, then publish the blobs holders fetch. A preset (q4_k_m, q8_0, q4_k, q6_k) applies one format to all three FFN projections; --quant-json sets each projection independently. Omit both to publish dense f32. The block layout is GGUF-compatible, so a holder decodes a prepared expert with the same dequant path it uses for a local GGUF.

tenzro moe prepare-experts \
  --model-id qwen3.5-397b-a17b --layer 0 \
  --experts 1,2,5 \
  --quant q4_k_m
# → { job_id, ... }; poll it:
tenzro moe prepare-status --job-id <job_id>

Load per-expert FFN weights (gate/up/down projections) into the node's expert runtime, keyed by (model, layer, expert). Shards can come from a local safetensors file or a content-addressed tenzro://blob/<hash> URI printed by prepare-status and fetched over iroh-blobs. Load the gating network for each layer you participate in.

tenzro moe load-expert \
  --model qwen3.5-397b-a17b --layer 0 --expert 1 \
  --file ./l0-e1.safetensors

tenzro moe load-gate \
  --model qwen3.5-397b-a17b --layer 0 \
  --file ./l0-router.safetensors

Show the node's resident experts and gates. Each reports its residency tier — Warm in memory or Cold on disk — its byte footprint, the memory budget, and whether a GPU compute backend is active on this node. A holder can advertise more experts than fit in memory; the runtime keeps them in a byte-bounded LRU over a disk tier and promotes the experts a forward is about to hit back to warm before it dispatches.

tenzro moe status

The shard map aggregates every holder's declaration into a network-wide view: expert coverage, replication per expert, which experts are hot (committed TPS over the threshold), and which are under-replicated. The default policy asks every active expert to be held by at least two distinct providers.

tenzro moe shard-map --model qwen3.5-397b-a17b

Before running a request, you can inspect how it would fan out. Feed a JSON file of top-k routing decisions — [{token_index, experts: [{layer, expert}]}] — and the planner returns the per-holder batches it would dispatch.

tenzro moe plan-dispatch \
  --model qwen3.5-397b-a17b \
  --routings ./routings.json

A distributed forward gates locally, hands the routing decisions to the planner, sends each per-holder batch — executed locally when this node holds the expert, over the holder's iroh QUIC endpoint when advertised, over HTTP otherwise — and recombines per-token outputs weighted by the gate probabilities.

tenzro moe forward \
  --model qwen3.5-397b-a17b \
  --hidden ./hidden.f32 \
  --d-model 5120 \
  --out ./combined.f32