Subscribe
AI
5 min read

What Are Model Parameters? How Billions of Numbers Make a Language Model

Every headline about a new model leads with a number in the billions: 175B, 405B, 671B. The number is the parameter count, and it is the single most quoted and least understood specification in AI. This explainer covers what a parameter actually is, where the hundreds of billions of them sit inside a model, what they cost, and why the count is a worse yardstick than the industry treats it. A parameter is one number A language model, at the end of training, is a list of ordinary decimal number

What Are Model Parameters? How Billions of Numbers Make a Language Model

Every headline about a new model leads with a number in the billions: 175B, 405B, 671B. The number is the parameter count, and it is the single most quoted and least understood specification in AI. This explainer covers what a parameter actually is, where the hundreds of billions of them sit inside a model, what they cost, and why the count is a worse yardstick than the industry treats it.

A parameter is one number

A language model, at the end of training, is a list of ordinary decimal numbers. Each one is a parameter (the words parameter and weight are used interchangeably in practice). The model turns text into numbers, runs those numbers through a fixed set of arithmetic steps, and produces a probability for the next token. The parameters are the values that fill in that arithmetic: the entries of the matrices the program multiplies vectors by at every step.

The original Transformer paper described a model with 65 million parameters, and a larger variant with 213 million (Vaswani et al., 2017). GPT-3 scaled the same family of equations to 175 billion (Brown et al., 2020). Meta's Llama 3.1 405B, released in July 2024, was the first openly available model past the 400-billion mark (Meta). Nothing about the math changed across that growth. Only the size of the number list did.

Where the parameters live

Three blocks of the transformer architecture hold essentially all of them.

The embedding matrix. One big lookup table that maps each vocabulary token to a vector. For GPT-3's model width, a vocabulary of roughly 50,000 tokens at 12,288 dimensions is several hundred million parameters: real weight, but a rounding error next to the layers.

The attention layers. Each layer stores four matrices per head projection plus output projection (the query, key, value and output mappings). These decide which earlier tokens the model reads when predicting the next one.

The feed-forward networks. Each layer also stores two wide matrices that process every token position after attention. Following the convention set by the original paper, the inner layer is four times the model width. For a model whose number of heads times head width equals its model width, which is the convention in most LLM designs, the feed-forward matrices and the attention matrices contribute roughly equal numbers of parameters, and together they dominate the count.

Where the numbers sit: the three parameter blocks of a transformer, illustrated in Olivetti brochure style

None of this means the model "stores text." It does not. There is no sentence inside the weights. Knowledge in a language model is distributed across hundreds of billions of small numeric values the way a hologram is spread across a plate: any single number means nothing alone, and no number has a fixed meaning like a database field.

What the numbers cost

Parameters are stored as floating point numbers, typically 2 bytes each at inference. Do the arithmetic: 175 billion parameters at 2 bytes is about 350 GB just to hold GPT-3's weights, before any of the text in or out. A 7B model needs about 14 GB, which is why "7B" became the standard size for laptops running quantized models.

Training costs more. To update the numbers, the optimizer keeps additional state per parameter: a full-precision master copy of each weight plus two accumulator values (Adam's first and second moments). The ZeRO paper counts this as 16 bytes per parameter in mixed-precision training, eight times the inference cost of the weights alone (Rajbhandari et al., 2020). That ratio, roughly 7 to 8 times, is why training a frontier model is a datacenter problem and serving one is merely an expensive hosting problem.

What training does to the numbers

Gradient descent nudges every parameter in the direction that slightly lowers prediction error on the training text, billions of times over. The relationship between parameter count and the amount of text needed was formalized by DeepMind's Chinchilla work: for a fixed compute budget, model size and training tokens should grow together, at roughly 20 tokens per parameter (Hoffmann et al., 2022). Their test case was the point of the finding: Chinchilla, with 70 billion parameters trained on 1.4 trillion tokens, beat Gopher, a model with four times as many parameters trained on less data, despite using the same total compute. More parameters is not automatically better. Parameters need to be fed.

Llama 3.1 405B reflects the post-Chinchilla convention: more than 15 trillion training tokens for a 405-billion-parameter model, roughly 38 tokens per parameter (Meta's model card and blog).

Total parameters versus the ones that run

Since 2024, the biggest open models hide a detail in the headline count. Mixture-of-experts models replace each feed-forward block with many parallel "experts" and route each token to only a few of them. DeepSeek-V3 advertises 671 billion parameters, but only 37 billion are used to compute any single token (DeepSeek technical report). Mixtral 8x7B works the same way at smaller scale: 47 billion total, about 13 billion active per token (Mistral AI).

A sparse routing wall: one token lights up a few experts while the rest stay idle

The consequence: the total count predicts memory cost (all experts must be stored), while the active count predicts compute per token (how much math runs to answer you). Two models with the same headline number can have very different inference bills.

Why the count is a bad yardstick

Closed labs do not disclose it. The GPT-4 technical report explicitly declined to publish architecture and parameter details, citing safety and competitive considerations (OpenAI, 2023). When a closed model's count appears in a story, it is a rumor. Open-weight releases like Llama and DeepSeek are where real numbers exist.

Capability does not track size. Chinchilla's result is the canonical counterexample. So is LoRA: fine-tuning can adapt a 175-billion-parameter model by changing a sliver of new matrices, cutting trainable parameters by up to 10,000x versus full fine-tuning (Hu et al., 2021). The parameter count describes the size of the storage object, not the quality of the system.

The useful specs are elsewhere. Training tokens per parameter, active versus total parameters, context window, and measured benchmark performance all tell you more than the raw count.

The short version

A parameter is one learned number, and a model is a list of a few billion to a few hundred billion of them. They sit mostly in the feed-forward matrices, cost 2 bytes apiece to store and about 16 to update, and grow meaningless as a scoreboard once you know that 70B models have beaten 280B models at equal compute. Read the parameter count as an indicator of how big the file is. Read everything else for how good the model is.

Sources

AILLMEducation