Skip to content
Tenzro
Documentation menu
Start here

AI models explained

What the main kinds of AI model are, how formats and quantization decide what runs where, what fits in your memory, and what licences, privacy, retrieval, agents and evaluation mean, with an On Tenzro note in each part.

This guide explains durable concepts first, then what they mean on Tenzro. Model names and versions change quickly, so for current lists use the Tenzro catalog (tenzro catalog list) and the vendor pages.

Model types

Language models (LLMs)

A language model predicts the next piece of text, one token at a time. Chat, code help, summaries and extraction all use this one mechanism with different prompts. Size is counted in parameters (7B means seven billion). Larger models are usually more capable and always need more memory and time per token.

Small and edge models

A small language model (roughly under 10B parameters) trades capability for memory and speed. It is the right choice for a laptop, a Raspberry Pi class device, a phone, or when many requests must run cheaply.

Mixture-of-experts (MoE)

A mixture-of-experts model holds many expert sub-networks but runs only a few for each token. The memory needed is that of ALL experts; the compute per token is that of the few active ones. A model named "30B-A3B" has 30 billion parameters in total and about 3 billion active per token: it needs memory for 30B and runs close to the speed of a 3B model.

Embedding models

An embedding model turns text (or an image) into a fixed-length list of numbers so that similar meanings are close together. Used for search, retrieval and clustering. Output is a vector, not text. Vectors from different models, or different dimensions, can never be compared with each other.

Rerankers

A reranker reads a question together with each candidate passage and scores how well the passage answers it. It is slower than vector search, so the usual pattern is: vector or keyword search finds about 20 candidates, the reranker reorders them, and the top few are used.

Vision and multimodal models

A vision-language model accepts images (and sometimes audio or video) as well as text and answers in text. Pure vision models do one fixed task: classification, detection (boxes around objects), or segmentation (a label for every pixel).

Speech, audio and generation

Speech-to-text transcribes audio. Text-to-speech produces audio from text. Audio models can also classify sounds. Image and video generation models are usually diffusion models: they start from noise and remove it step by step, guided by a prompt. 3D generation follows the same idea with 3D outputs.

Forecasting and time-series models

A forecasting model predicts future values of a numeric series from its past. It is a different model family from language models and is evaluated with its own error measures.

Guard and safety models

A guard model classifies a prompt or a response against a safety policy (allowed or not, and why). It runs next to the main model, before or after it.

On Tenzro

The Tenzro catalog seed in the tenzro-network repository (crates/tenzro-catalog/seed, read 2026-11-07) groups entries by engine family: gguf language models (262 entries), media generation (48), ONNX text embedding (10), ONNX vision (9), ONNX detection (9), ONNX audio (8), ONNX forecasting (7), ONNX segmentation (4), rerank (4), decision (4), embed (1), tts (1) and ONNX text segmentation (1). Counts change with each release: tenzro catalog list shows what a running node serves, and tenzro catalog list --engine gguf limits the list to one family.

Sources: Tenzro CLI reference (docs/CLI.md, tenzro catalog), catalog seed files above.

Formats, runtimes and quantization

Formats and runtimes

  • GGUF is the model file format of llama.cpp. One file holds weights, tokenizer and settings, usually already quantized. llama.cpp runs GGUF language models on CPU and, through its backends, on many GPUs.
  • ONNX is an open exchange format for neural networks. ONNX Runtime executes ONNX models through execution providers.
  • Safetensors is a safe weight-storage format used by most Python training and model-hub workflows. It stores weights only; a runtime or conversion step is needed to serve it efficiently.
  • TensorRT is NVIDIA's inference optimizer for NVIDIA GPUs. vLLM-style servers batch many requests for GPU serving of large language models.

What each runtime can use (checked 2026-11-07)

llama.cpp backends listed in its README: CUDA (Nvidia GPU), HIP (AMD GPU), Vulkan (GPU), SYCL (Intel GPU), Metal (Apple Silicon), CANN (Huawei Ascend NPU), MUSA (Moore Threads GPU), OpenCL (Adreno GPU), OpenVINO (Intel CPUs, GPUs, NPUs; marked in progress), Hexagon (Snapdragon), WebGPU, ZenDNN (AMD CPU), plus BLAS, BLIS, RPC, IBM zDNN and VirtGPU. Source: https://github.com/ggml-org/llama.cpp (README, backend table).

ONNX Runtime execution providers listed on its documentation: CUDA, TensorRT, OpenVINO, DNNL, XNNPACK, DirectML, QNN (Qualcomm), MIGraphX (AMD), WebGPU, plus preview providers including CoreML and CANN. There is no Vulkan provider. The ROCm provider was removed in ONNX Runtime 1.23: AMD GPUs use MIGraphX. Source: https://onnxruntime.ai/docs/execution-providers/.

Quantization

Quantization stores weights with fewer bits. A weight in 16-bit floating point takes 2 bytes; an 8-bit format about 1; a 4-bit format about half a byte. Fewer bits mean less memory and usually faster generation, at some loss of quality. The loss is small at 8 bits, noticeable but often acceptable at 4 bits, and grows quickly below that. Quantization changes the weights only: the context window, and the memory for it, are separate.

Context windows

The context window is how many tokens the model can read and write in one conversation. Longer context needs more memory (the KV cache, below) and more time.

On Tenzro

Tenzro serves GGUF models through its llama.cpp-based engine and ONNX models through ONNX Runtime; the engine family appears in the catalog (tenzro catalog list --engine gguf, --engine onnx-vision). Which accelerators a Tenzro node uses on a given machine is stated in the Tenzro hardware coverage pages, not here.

Sources: llama.cpp README and ONNX Runtime documentation above; Tenzro CLI reference (tenzro catalog list).

Will a model fit on my hardware

The rule

Memory needed = weights + KV cache + runtime overhead.

  • Weights = parameters x bits per weight / 8. A 7B model in 16-bit needs 14 GB; in 8-bit about 7.4 GB; in a 4-bit format (Q4_K_M) about 4.2 GB.
  • KV cache = 2 x layers x KV heads x head size x bytes per value x context length. It grows with context. A Llama-2-7B-shaped model (32 layers, 32 KV heads, head size 128) at 4096 tokens in 16-bit uses about 2.1 GB.
  • Overhead: about 1 GB for the runtime and buffers. Memory type: a GPU needs it in its own VRAM (or split across GPUs). Apple silicon and similar unified-memory machines share one pool, and the operating system needs part of it (leave 25 to 35 percent). CPU-only machines use system RAM and run much slower.

Worked examples (general rules of thumb, not Tenzro measurements)

  • 13B, 4-bit (Q4_K_M), full-attention KV (40 layers, 40 KV heads, head size 128): weights about 7.9 GB. At 2048 tokens the total is about 10.6 GB, which fits a 12 GB GPU. At 4096 tokens the total is about 12.3 GB, which does not fit 12 GB. Nearest alternative on 12 GB: shorter context, or a 7B model.
  • 13B in 16-bit needs 26 GB for weights alone: it does not fit a 12, 16 or 24 GB GPU. 4-bit does fit 16 and 24 GB.
  • 8 GB of memory, 4-bit, about 1 GB overhead and no KV: at most about 11.5B parameters of weights; with a realistic context and an operating system, plan for 7B to 8B.
  • Models that use grouped-query attention have fewer KV heads, so their KV cache is several times smaller than the full-attention figure above.

What to ask the user

The memory type and size (GPU VRAM, unified memory, system RAM), the model size and quantization wanted, and the context length. The Tenzro Guide answers "what can I run" from these.

On Tenzro

Measured requirements per role and hardware class (minimum, optimal, best) come from the Tenzro measured tables and the hardware coverage matrix, and those numbers override the rules above for Tenzro. Ask the Guide with your hardware and goal to get the matching row.

Sources: arithmetic above (tested in the Guide repository against the reference values 7B x 16 bit = 14 GB and the Llama-2-7B KV figure); Tenzro hardware coverage pages.

Licences, privacy, RAG, agents and evaluation

Licences: what to check

  • Open-weight means the trained weights can be downloaded; the licence still sets the terms (commercial use, size or revenue limits, required attribution, restricted uses).
  • Open-source in the strict sense also requires the licence to meet an open-source definition, and often the training code and data.
  • Commercial or gated models need an agreement or an access request before download. Read the licence file shipped with the model for each use. Tenzro catalog entries record the licence and whether the model is gated (tenzro catalog show <MODEL> shows the entry and whether a token is needed).

Local versus hosted

  • Local: the data stays on your machine, it works offline, and cost is your hardware. Capability is limited by your memory.
  • Hosted: access to larger models and no hardware to manage; prompts leave your machine and are processed under the provider's terms.
  • Network: Tenzro lets you use models served by other nodes, with a signed receipt per request; see the Tenzro docs for how requests are routed and paid.

Prompting and tool use

Give the model a clear role, the task, the inputs and the output format. Tool use means the model asks for a function call (search, calculator, API) and your program runs it and returns the result. The program, not the model, decides what actually runs.

Retrieval-augmented generation (RAG)

RAG finds relevant passages first (embeddings, keywords, a reranker) and puts them in the prompt so the model answers from them and can cite them. It reduces invented answers but does not remove them: check that every claim traces to a passage.

Agents

An agent is a loop: the model plans, calls tools, reads results and continues until done. Limit what tools can do, cap cost and steps, and keep a person in the loop for actions with side effects.

Evaluation

Build a fixed set of questions with expected answers or expected sources, run it after every change, and track retrieval hit rate, correctness, refusal behaviour on questions that should be declined, latency and cost.

On Tenzro

tenzro chat [MODEL_ID] opens a chat; without a model id it picks one from --use-case (chat, code, reasoning, summarize, extract or embed) and an optional --budget. --max-tokens and --temperature set generation limits. Source: Tenzro CLI reference (tenzro chat, tenzro catalog show).

ExploreJoin the network