Total Cost of AI Ownership: What Inference Really Costs

An enterprise AI budget usually starts with a token price and a usage estimate. That number is real, but it is the smallest honest part of the bill. As an AI programme moves from pilot to production, Gartner expects inference to account for at least 70 percent of a model's lifetime costs and predicts that through 2028 at least half of generative AI projects will overrun their budgets because of poor architectural choices and a lack of operational know-how.

This article lays out a cost model that holds up in production, then the levers that move it.

The token price is the marginal cost, not the total

The FinOps Foundation puts it directly: a token-only view captures the variable, marginal cost of a unit of inference and omits the fixed and semi-fixed costs that decide whether an initiative is viable at scale. In production retrieval and agentic deployments, the Foundation reports that the infrastructure surrounding the model call (vector databases, embedding generation, orchestration runtime, caching layers, data egress) routinely represents 40 to 60 percent of total feature spend.

The takeaway: if your unit cost leaves out the harness, you will optimise the wrong thing.

A cost model with six lines

For each production use case, estimate cost per unit of business output (per resolved ticket, per processed document, per analysed hour of video) from six lines:

1. Model inference. Tokens, images, seconds of audio or frames, at the price of whoever serves them.
2. Harness. Retrieval, embeddings, orchestration, tool calls, caching, logging.
3. Data movement. Egress between clouds and regions, and the cost of keeping copies in several places.
4. Capacity risk. For owned or reserved capacity, the cost of the hours it sits idle. For pay-per-use, the premium paid for elasticity.
5. Assurance. Evaluation runs, red-teaming, audit logging and its retention, incident handling.
6. Switching cost. The engineering needed to change model or supplier. It is zero on a spreadsheet and very real when a supplier changes price or terms.

Own, rent or call: the crossover

The Foundation describes three deployment models with different profiles: SaaS-embedded AI has the lowest activation cost and the highest per-token cost; API consumption gives the most visibility but is sensitive to provider pricing; self-hosted inference requires the most capital and offers the lowest marginal cost per token at sustained scale, with the highest risk when under-used. It also notes the crossover points can be calculated when token demand is modelled early.

In practice the deciding variable is utilisation. Owned accelerators are cheap per token only while they are busy. Two things erode that: demand that is spiky by time of day or season, and fast price declines elsewhere. The 2025 AI Index reports that the inference cost for GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024, and that hardware costs declined about 30 percent a year. A purchase that looks cheap against today's API price may look expensive against next year's.

The takeaway: own the steady base load, buy the peaks, and re-run the crossover every quarter.

Seven levers that reduce cost

1. Route by difficulty. Most request streams contain a large share of easy cases. A router that sends easy requests to a small model and hard ones to a large one can cut cost substantially; RouteLLM reported cost reductions of over two times on public benchmarks in some cases without sacrificing response quality. Measure your own escalation rate rather than assuming one.

2. Right-size the model per step. Classification, extraction, embedding and transcription rarely need a frontier model. A ladder of model sizes per workflow beats one large model for everything.

3. Reuse the prefix. Agents resend the whole conversation on every turn. Serving engines that keep the KV cache of a shared prefix avoid recomputing it; the vLLM documentation describes this for long documents and multi-round chat. Route repeat turns to the machine that already holds the prefix.

4. Meter in the native unit. Pay for pixel-steps for image generation, seconds for speech, points for forecasting, pairs for reranking. Forcing everything into token prices hides waste.

5. Batch what is not interactive. Overnight document processing and evaluation runs do not need interactive latency. Schedule them where capacity is idle.

6. Keep data close to compute. Egress and duplicate storage are line 3. Moving the model to the data is often cheaper than moving the data to the model.

7. Make suppliers compete. A standard interface and an evaluation set per use case make switching a test, not a project. The ability to switch is what keeps every supplier's price honest.

Regional notes

Energy prices, GPU availability and data-transfer rules differ by region, so the same workload can have different optimal placements in the EU, US, UK, APAC and MENA. Residency requirements can also forbid the cheapest option; build them into the cost model as constraints, not afterthoughts.

What this means for CTOs and infrastructure operators

• Report cost per unit of business output, with harness and data movement included.
• Track utilisation of owned capacity weekly; idle hours are the hidden cost of ownership.
• Budget for assurance as a line item, not a contingency.
• Treat switching cost as a risk to be reduced, through interfaces and evaluations, before signing any multi-year commitment.

Where Tenzro fits

Tenzro is an open network where machines of every size sell AI work to agents, apps and people, so it can serve as a pay-per-use source beside owned capacity. Several of the levers above are part of how it works today:

• Competitive quotes. tenzro_quoteInference prices one request at every provider serving a model, cheapest first, from each provider's published rate card. The network sets no price; providers do.
• Native units. Rate cards price metered work per unit (input tokens, pixel-steps, characters, audio milliseconds and others), and a call none of whose billable work is priced is refused rather than served free.
• Pay for what was used. The reference agent payment flow signs a ceiling and settles the actual metered cost after the response; a failed response is not charged.
• Bounded billing. A forwarded call is billed from the provider's signed receipt and capped by what was actually sent and received.
• Prefix reuse and routing. The batch engine keeps computed prefixes for the next request that shares them, and routers can find a provider holding a caller's prefix without revealing what anyone asked. Routing strategies include lowest price, lowest latency and a weighted score.

Planned: edge devices hosting agent harnesses that run the cheap rungs of a model ladder locally and pay GPU providers only for escalations, with each party paid from the caller's budget.

The network is live in preview; governance decides each step towards testnet and mainnet on measured evidence. See tenzro.com.