Training ends. Inference never does.
A language model spends months being trained and then, in principle, nothing more happens to its weights. Every question you ask ChatGPT, Claude or Gemini afterward triggers a different process: the model reads your prompt, runs a forward pass, and emits one token at a time. That process is called inference, and it is where almost all of the money in deployed AI now goes. Deloitte forecast that inference workloads would account for roughly two-thirds of all AI compute in 2026, up from a third in 2023 and half in 2025.
This explainer covers what actually happens on the GPU between your request and the first visible word, why the answer streams out one token at a time instead of all at once, and why providers charge more per output token than per input token. For the pieces that fit into this picture, see our explainers on next-token prediction, the KV cache and attention.
Inference versus training
Training updates the model: it computes gradients and nudges billions of weights, as described in our backpropagation explainer. Inference does none of that. The weights are frozen, no gradients are computed, and the model simply transforms an input sequence into a probability distribution over the next token. The same model that cost hundreds of millions of dollars to build is re-used, unchanged, for every single request.
That re-use is what makes the economics strange. Training is a capital expense that arrives in known bursts. Inference is a recurring cost that scales with usage, which is why enterprises consistently under-budget it.
The two phases: prefill and decode
An inference request runs in two distinct phases, and they behave like different workloads on different hardware.
Prefill processes the entire prompt at once. Every input token is known up front, so the model computes query, key and value vectors for all of them simultaneously and attention runs as large matrix-matrix multiplications. This is compute-bound work: the GPU's arithmetic units are close to saturated. Prefill fills the KV cache and emits the first output token. The metric that captures it is time to first token, TTFT.
Decode generates the rest of the response one token at a time. Each new token depends on the one before it, so the work cannot be parallelized the way prefill can. Per step, the model computes vectors for a single token, reads the cached keys and values of everything that came before, and runs matrix-vector products instead of matrix-matrix products. The bottleneck flips from arithmetic to memory bandwidth. The metric for this phase is inter-token latency, ITL, also called time per output token.
The relationship between them is simple arithmetic: end-to-end latency is roughly TTFT plus the number of remaining output tokens times the per-token interval. Redis measured 472 milliseconds of TTFT at 32,768 input tokens and about 2.2 seconds at 122,880 tokens on the same model, and notes that a 500-token answer streamed at 80 milliseconds per token spends around 40 seconds in decode alone. The blank pause before the first word is prefill. The typing speed you wait out afterward is decode, and for long answers decode dominates the experience.

Why decode is slow: the roofline model
The asymmetry has a formal name: the roofline model, published by Williams, Waterman and Patterson in 2009. It says a kernel's achievable throughput is capped either by the chip's peak arithmetic rate or by memory bandwidth times its arithmetic intensity (FLOPs performed per byte moved), whichever is lower.
On an H100, dense BF16 peak is about 989 TFLOPS and HBM bandwidth about 3.35 TB/s, which puts the crossover, the "ridge point", at roughly 295 FLOPs per byte. Prefill at a 2,000-token prompt sits near 2,000 FLOPs per byte: comfortably compute-bound. Decode at batch size 1 sits at about 1: the GPU must load every weight matrix from memory to perform almost no arithmetic per byte loaded. Spheron's roofline walkthrough puts single-stream decode at 0.3 percent of the chip's compute roof, and notes the batch size needed to cross the ridge is roughly 280 tokens, well beyond what latency-sensitive serving runs. Worse, the attention portion of decode stays near 1 FLOP per byte almost regardless of batch size, as a 2025 measurement study found: throughput plateaus once DRAM reads saturate even when memory still allows larger batches.
The practical consequence is that a chat request uses the most expensive computer in the building as a very slow tape drive. Money is not saved by better arithmetic; it is saved by moving fewer bytes and by batching more streams over the same weight reads.
The KV cache: pay now or pay every step
Decode could, in principle, re-attend over the whole prefix from scratch each token, recomputing keys and values every step. Every production engine instead caches them: the KV cache stores the key and value vectors for every token already processed so each decode step reads them instead of recomputing them.
The price is memory. The cache grows linearly with sequence length, and it competes for the same HBM as the model weights, which caps how many requests fit on a card at once. Chalk's overview makes the point bluntly: prompt length sets both time to first token and serving density. The vLLM project's PagedAttention paper showed that naive cache allocation wastes most of the reserved memory to fragmentation and borrowed the virtual-memory paging idea from operating systems to fix it. That work is why KV cache management, not raw FLOPS, dominates the serving-engine literature.
Batching: the one free lunch
Batching fixes the arithmetic-intensity problem for the matrix products: put 64 requests in one decode step and the weights are still read once, but each weight now does 64 multiply-adds, so intensity scales roughly with batch size (DataInterview's course notes). Throughput per token improves tens of times before the chip nears the ridge.
Scheduling is where the two metrics pull against each other. Static batching makes a new request wait until every request in the running batch finishes. Modern engines use continuous batching: a completed request leaves immediately and a queued one takes its slot. If the engine prioritizes prefills to protect time to first token, in-flight decodes stall mid-stream, and users perceive the answer pausing while someone else's long prompt loads, as Hugging Face's performance series explains. Chunked prefill, from the Sarathi paper, interleaves prompt processing with decode steps to split the difference. Every serving engine you use today is a set of compromises between these two numbers.

Why output tokens cost more than input tokens
Almost every API price list commonly charges two to six times more per output token than per input token, and the ratio is not marketing. It tracks the GPU time. Sarathi's measurements put the cost of one decode token at batch size 1 at up to 200 times the per-token cost of prefill in raw GPU time. An independent at-home measurement on a laptop found one output token took about 29 times longer to produce than one input token at a 2,048-token prompt, and the provider pricing survey notes the 5-6x output premium across OpenAI, Anthropic and Google reflects the difference between a bulk pass and a drip.
The same physics explains two pricing details that look arbitrary:
- Cached input tokens are priced at roughly 10 percent of the normal input rate (Anthropic's prompt caching docs, OpenAI's) because the engine skips prefill compute and just reads KV state from memory.
- Reasoning models are expensive in direct proportion to invisible output. Thinking tokens are decoded tokens. Deloitte estimates long thinking can use more than 100 times the compute of a simple request like an email summary, and that aggregate post-training now uses around 30 times the compute of the original foundation training run.
The price of a token is falling fast
Per-token prices at fixed capability have collapsed. Epoch AI measures the cost of a given level of AI performance falling about 47 percent per quarter, roughly 13x per year, across five benchmarks since 2023, faster than the historical declines of solar, batteries or DNA sequencing. In its price-tracking dataset, reaching GPT-3.5-level MMLU performance cost $20 per million tokens in November 2022, and Stanford's AI Index found the same quality bar reachable for $0.07 per million tokens by October 2024, a 280-fold drop in roughly 18 months. Andreessen Horowitz's LLMflation analysis counted a 1,000x decline over three years to the same effect.
What has not fallen is the bill, because usage grows faster than prices drop. That tension is a topic for a separate post; see our two price curves analysis.
What this means when you feel the lag
You cannot see the GPU, but you can feel both phases. If the model starts slowly, the problem is how much you are sending: long prompts, big retrieved context, and long conversation histories all inflate prefill, and no hardware choice fixes it without changing the input. If the model starts fast but types slowly, the problem is decode: batch economics, KV cache pressure, and per-token server capacity. Application-level fixes map to the phases cleanly: shorten and cache context to cut time to first token, request shorter answers or route to smaller models to cut time per token.
Inference is not an afterthought of training. It is the product. Everything a user experiences, and nearly everything a company pays, happens in the forward pass after the model was already finished.
Sources
- Redis: Prefill vs Decode: LLM Inference Phases Explained
- Chalk: What Is LLM Inference? Prefill, Decode and KV Cache
- Hugging Face (TNG Technology Consulting): Prefill and Decode for Concurrent Requests
- Outcomes School: Prefill vs Decode: LLM Inference Optimization
- Williams, Waterman, Patterson: Roofline: An Insightful Visual Performance Model (CACM 2009)
- Zero Entropy: Arithmetic intensity: FLOPs per byte and the roofline model
- Spheron: Memory Bound vs Compute Bound: Roofline Model for LLM Inference
- DataInterview: Roofline Model for LLM Inference
- Agrawal et al.: Mind the Memory Gap: GPU Bottlenecks in Large-Batch LLM Inference
- Kwon et al.: vLLM: Efficient Memory Management for Large Language Model Serving with PagedAttention
- Agrawal et al.: Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
- Deloitte Insights: More compute for AI, not less (TMT Predictions 2026)
- Epoch AI: The plunging price of thought and LLM inference price trends dataset
- Andreessen Horowitz: LLMflation
- Compresr: Input Token Costs vs Output Token Costs
- Warehows: Are input or output tokens more expensive?
- Bryan Anthoney: One Reason Input Tokens Are Cheaper than Output Tokens
- Stanford AI Index 2025 via LLM Inference Prices Fell 280x
