Inference

Distributed MoE.

Tenzro batches per holder: each provider declares the subset of experts it holds for a model, and the planner aggregates tokens whose top-k routing landed on the same (expert, holder) tuple into one fan-out call. The dispatch path runs over the provider's iroh QUIC endpoint when available, and falls back to the OpenAI-compatible HTTP endpoint otherwise.

STATUS: Testnet
CRATES: tenzro-model::{moe_shard, moe_router, moe_exec}
TYPE: Inference primitive

Provider declaration

ProviderCapacity {
  // ... existing fields
  moe_holdings: [
    { model_id: "qwen3.5-397b-a17b", layer: 0, expert: 1,
      residency: Warm, committed_tps: 800 },
    // ...
  ],
  moe_roles: [ExpertHolder, Router],
  iroh_endpoint_id: "ep-…",
}

Shard view

tenzro_moeShardMap { model_id }  →

{
  covered_experts: 256,
  distinct_providers: 12,
  expert_holders_role_count: 9,
  router_role_count: 2,
  policy: { min_replication: 2, max_replication: 8, hot_threshold_tps: 1000 },
  under_replicated_experts: [{ layer: 7, expert: 142 }],
  hot_experts: [{ layer: 3, expert: 5 }],
  holders: [
    { layer: 0, expert: 1, replication: 3, holders: [...] }
  ],
}

Dispatch plan

tenzro_moePlanDispatch {
  model_id,
  routings: [{ token_index, experts: [{ layer, expert }, ...] }, ...],
  allow_cold: false,
}  →

{
  batches: [
    { layer: 0, expert: 1, provider: <hex>,
      iroh_endpoint_id, http_endpoint,
      token_indices: [0, 1, 4, 9] },
    ...
  ],
  token_assignments: [
    { token_index: 0, slots: [{ layer, expert, provider }, ...] }
  ],
}

Roles

Replica       — full model on one provider (default; smallest models)
Router        — runs the gating step + fans out batches
ExpertHolder  — holds one or more experts (declared in moe_holdings)
PrefillDecode — co-located prefill + decode
Prefill       — prefill phase only; hands off KV cache over iroh
Decode        — decode phase only; consumes KV cache over iroh

Replication policy

The default policy requires every active expert to be held by at least 2 distinct providers; up to 8 holders may advertise a hot expert (committed TPS ≥ 1000). Governance tunes these via the same proposal path that drives `adaptive-burn` and `pkr_scheduler`.

Expert-host execution

Every node embeds an expert-host runtime. Holders load expert FFN weights (gate/up/down projections, SwiGLU) and gating networks from safetensors payloads, keyed by (model_id, layer, expert). A distributed layer forward gates locally, feeds the routing decisions to the dispatch planner, sends each per-holder batch as base64-encoded f32 rows — executed locally when this node holds the expert, over the holder's iroh QUIC endpoint (tenzro/moe ALPN, moe/execute + moe/status) when advertised, or over HTTP otherwise — and recombines per-token outputs weighted by the gate probabilities.

tenzro_moeExpertLoad    — load one expert FFN (safetensors)
tenzro_moeGateLoad      — load a layer's gating network
tenzro_moeExpertUnload  / tenzro_moeGateUnload
tenzro_moeExpertStatus  — resident experts/gates + memory footprint
tenzro_moeRoute         — gate a batch of hidden states (top-k)
tenzro_moeExecute       — run a batch through one resident expert
tenzro_moeForward       — gate → plan → dispatch → combine

MoE catalog coverage

Qwen 3 30B-A3B (128/8)        Qwen 3.5 35B/122B/397B-A* (128/8)
Qwen 3.6 35B-A3B (128/8)      Qwen 3.5 0.8B–397B MTP variants
Gemma 4 26B-A4B (128/4 + 1)   DiffusionGemma 26B-A4B
Kimi K2 / K2.5 / K2.6 / K2.7 Code (384/8 + 1)
MiniMax M1 (32/2) / M3 (32/2)
DeepSeek V3 0324 (256/8 + 1, native MTP)
DeepSeek V4 Pro 1.6T / Flash 284B (1M context, MTP)
GLM 5 / 5.1 / 5.2 (5.2 has improved MTP)
Qwen 3 Coder 30B-A3B (128/8)
Nemotron Nano 30B-A3B (16/4)
gpt-oss 120B (128/4)

← All docs