Distributed MoE.

MoE declarations install on the local provider record at expert/gate load and unload, and ride every provider heartbeat — the network's shard view converges without a separate announcement path. The gossip consumer admits pure expert holders that serve no full model of their own.

Each holding carries the content-addressed `tenzro://blob/<hash>` the holder loaded those weights from. That URI is what lets another provider fetch the identical bytes when replication needs raising. Experts admitted from an inline payload declare no `blob_uri` and cannot be replicated from the declaration alone.

ProviderCapacity {
  // ... existing fields
  moe_holdings: [
    { model_id: "qwen3.5-397b-a17b", layer: 0, expert: 1,
      residency: Warm, committed_tps: 800,
      blob_uri: "tenzro://blob/<hash>" },
    // ...
  ],
  moe_roles: [ExpertHolder, Router],
  iroh_endpoint_id: "ep-…",
}

tenzro_moeShardMap { model_id }  →

{
  covered_experts: 256,
  distinct_providers: 12,
  expert_holders_role_count: 9,
  router_role_count: 2,
  policy: { min_replication: 2, max_replication: 8, hot_threshold_tps: 1000 },
  under_replicated_experts: [{ layer: 7, expert: 142 }],
  hot_experts: [{ layer: 3, expert: 5 }],
  holders: [
    { layer: 0, expert: 1, replication: 3, holders: [
      { provider: <hex>, residency: "Warm", committed_tps: 800,
        iroh_endpoint_id, http_endpoint,
        blob_uri: "tenzro://blob/<hash>" },
    ] }
  ],
}

tenzro_moePlanDispatch {
  model_id,
  routings: [{ token_index, experts: [{ layer, expert }, ...] }, ...],
  allow_cold: false,
}  →

{
  batches: [
    { layer: 0, expert: 1, provider: <hex>,
      iroh_endpoint_id, http_endpoint,
      token_indices: [0, 1, 4, 9] },
    ...
  ],
  token_assignments: [
    { token_index: 0, slots: [{ layer, expert, provider }, ...] }
  ],
}

Replica       — full model on one provider (default; smallest models)
Router        — runs the gating step + fans out batches
ExpertHolder  — holds one or more experts (declared in moe_holdings)
PrefillDecode — co-located prefill + decode
Prefill       — prefill phase only; hands off KV cache over iroh
Decode        — decode phase only; consumes KV cache over iroh

The default policy requires every active expert to be held by at least 2 distinct providers; up to 8 holders may advertise a hot expert (committed TPS ≥ 1000). Governance tunes these via the same proposal path that drives `adaptive-burn` and `pkr_scheduler`.

Raising replication is pull-based. Every node runs a repair pass every 120 seconds over the models it already holds experts for: it computes one candidate holder per missing replica by rendezvous hash over the providers already declaring holdings for that model, discards every row that names someone else, and fetches the ones naming itself from the holder's `blob_uri`. No node can make a peer allocate memory, and two nodes never race to cover the same replica slot. Each pass admits at most two experts, so one membership hiccup cannot pull a whole model onto a single node; membership-view skew produces mild over- or under-replication that heals as views converge.

Every node embeds an expert-host runtime. Holders load expert FFN weights (gate/up/down projections, SwiGLU) and gating networks from safetensors payloads, keyed by (model_id, layer, expert). A distributed layer forward gates locally, feeds the routing decisions to the dispatch planner, sends each per-holder batch as base64-encoded f32 rows — executed locally when this node holds the expert, over the holder's iroh QUIC endpoint (tenzro/moe ALPN, moe/execute + moe/status) when advertised, or over HTTP otherwise — and recombines per-token outputs weighted by the gate probabilities.

Both router families in the catalog run through the same gating network. Qwen-layout checkpoints route by softmax top-k. DeepSeek-layout checkpoints (DeepSeek V3/V4, Kimi K2/K3) route by sigmoid scoring with a per-expert selection bias and a routed scaling factor, and carry a fused shared-expert FFN that rides along as one extra weight-1.0 slot per token. The gate blob is self-describing — bias tensor and scaling metadata ship inside it — so holders load either family with the same call, and the shared expert is announced, held, dispatched, and settled like any routed expert.

A holder can advertise more experts than fit in memory. The runtime keeps experts in a byte-bounded memory-tier LRU — the budget auto-sizes to 60% of the host's available memory, or 4 GiB where that reading is unavailable — over a disk tier that spills raw safetensors and decodes them back on demand. Before a forward dispatches, readahead promotes the disk-tier experts the routing decision selected back into memory, so the experts a batch is about to hit are warm when it arrives. Residency (Warm memory / Cold disk) is read from this live tier state.

tenzro_moeExpertLoad    — load one expert FFN (safetensors)
tenzro_moeGateLoad      — load a layer's gating network
tenzro_moeExpertUnload  / tenzro_moeGateUnload
tenzro_moeExpertStatus  — resident experts/gates: per-expert tier
                          (memory/disk), memory_bytes, budget, counts
tenzro_moeRoute         — gate a batch of hidden states (top-k)
tenzro_moeExecute       — run a batch through one resident expert
tenzro_moeForward       — gate → plan → dispatch → combine,
                          bounded in-flight failover; allow_partial
                          degrades to a renormalized partial combine
tenzro_moeListReceipts  — sampled execution receipts this router
                          has persisted (model, expert, holder,
                          commitment hash)
tenzro_moeDisputeReceipt — re-execute a stored receipt against a
                           local copy of the expert; an upheld
                           dispute penalizes the signer
tenzro_moePrepareExperts — slice + block-quantize a layer's experts
                           for holders (ranged tensor fetches)
tenzro_moePrepareStatus  — prepare-job progress + tenzro://blob/ URIs

`tenzro_moePrepareExperts` slices a layer's experts out of a catalog checkpoint and block-quantizes each projection before publishing the blobs holders fetch. The `quant` argument is either a preset string or a per-projection object naming the format for each of the three FFN projections (`gate`, `up`, `down`).

Q8_0   ~8.5 bpw   32-element blocks, one f16 scale
Q4_K   ~4.5 bpw   K-quant super-blocks, gate/up default
Q6_K   ~6.6 bpw   K-quant super-blocks, down default

Preset q4_k_m  → { gate: Q4_K, up: Q4_K, down: Q6_K }
Preset q8_0    → all three Q8_0
Per-projection → { gate: "q4_k", up: "q4_k", down: "q6_k" }

The block layout is GGUF-compatible, so a holder decodes a prepared expert with the same dequant path it uses for a local GGUF. Quantizing the FFN weights is what lets a holder advertise more experts than its raw-f32 footprint would allow.

The expert matmul (`Y = X·Wᵀ`) runs behind a compute seam. The CPU backend is always present — a dense f32 path plus a runtime-detected AVX-512-VNNI Q8_0 dot product for quantized experts — so a default build carries no GPU dependency. Two GPU backends are feature-gated: a CUDA (cuBLAS) path for NVIDIA and a cross-vendor WGSL path.

A holder that built with a GPU backend advertises a `moe_gpu` flag on its provider record; the dispatch planner biases batches for hot experts toward GPU holders and falls back to CPU holders when none cover the shard. `tenzro_moeExpertStatus` reports whether a GPU backend is active on this node.

When a layer forward fans out to remote holders, the router compresses activation rows to Q8_0 blocks on the wire, keeps a warm-first backup holder ready to redispatch a batch if the primary stalls, and streams per-holder results into a pipelined, gate-weighted combine rather than blocking on the full set. A dense (non-MoE) layer that exceeds one holder is split across holders on the same pipeline.

Holder failures are absorbed inside the forward. A batch walks its planned holder set in warm-first order; when every known holder for an expert fails, the affected (expert, token) pairs are replanned against a rebuilt shard view that excludes every provider that already failed, while results already gathered stay in the combine. Replanning is bounded per forward. Each failure records a reputation penalty against that holder, and the winning holder's measured latency feeds its serving metrics. By default a forward that still cannot cover every token fails closed; `allow_partial` instead renormalizes the gate weights over the experts that responded and reports the unserved (expert, token) pairs under `missing` — the distributed analogue of serving with a reduced top-k under partial outages.

Every holder meters its aggregate expert-forward throughput over a rolling minute and stamps the measurement onto each advertised holding as `committed_tps`, so the dispatch planner ranks holders by observed serving capacity rather than self-declared numbers.

Every remote expert execution comes back signed. The holder commits to its output rows — per token, the top-k features by absolute value — and signs the commitment together with a hash of the exact input carrier it received, under the same key that signs its provider announcements. The router verifies each receipt inline before accepting a batch: recompute the commitment from the returned outputs, check the signature, the provider binding, and the token set. A bad or missing receipt is treated as a holder failure — the standby/replan path takes over and the holder takes the reputation penalty. A holder without a signing key serves receiptless and remote routers reject it.

After the forward completes, the router settles the remote expert work per holder in the background: each holder is paid at its own advertised per-input-token price, the settlement split is applied, and the holder's reputation is credited with the net amount — the only path that raises a provider's score, so reputation tracks paid, receipt-verified work rather than liveness. The settlement record binds to the concatenated commitment hashes of that forward's receipts.

Roughly one in 64 verified batches is persisted in full — request carrier, commitment rows, signed receipt — sampled by the commitment hash itself, so a holder cannot predict which of its batches are retained. Any node holding the same expert can re-execute the stored request and compare its rows against the committed sketch; an upheld dispute is a fraud proof against the signature and applies the quarantine-grade penalty.

Qwen 3 30B-A3B (128/8)        Qwen 3.5 35B/122B/397B-A* (128/8)
Qwen 3.6 35B-A3B (128/8)      Qwen 3.5 0.8B–397B MTP variants
Gemma 4 26B-A4B (128/4 + 1)   DiffusionGemma 26B-A4B
Kimi K2 / K2.5 / K2.6 / K2.7 Code (384/8 + 1)
Kimi K3 (896/16 + 2, 2.8T total / 104B active)
MiniMax M1 (32/2) / M3 (32/2)
DeepSeek V3 0324 (256/8 + 1, native MTP)
DeepSeek V4 Pro 1.6T / Flash 284B (1M context, MTP)
GLM 5 / 5.1 / 5.2 (5.2 has improved MTP)
Qwen 3 Coder 30B-A3B (128/8)
Nemotron Nano 30B-A3B (16/4)
gpt-oss 120B (128/4)

Per-expert extraction sources (safetensors):
Qwen 3 30B-A3B · DeepSeek V3 0324 · DeepSeek V4 Flash/Pro
Kimi K2 Instruct · Kimi K2.6 · Kimi K3