In a small language model, most of the parameters are not doing the thinking. They sit in the vocabulary. A new paper asks what that table actually buys, swaps it for a frozen 16 MB table of binary codes, and finds the trade is smaller than most people assume: about 0.20 nats of validation loss, and on three of four common benchmarks, no loss at all.
The paper is Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models by Sai Kiran Bathula of QLNI in New South Wales, Australia, posted to arXiv (cs.CL) with a date of October 15, 2026. The code, weights of every arm, training data, logs and every per-question evaluation answer are on Hugging Face, released for non-commercial research.
The problem: the table is most of the model
The paper uses Gemma 3 270M as its example. From the public configuration, its embedding table (tied to the output head) holds 167.8M of its 268.1M parameters, or 62.6%. The feed-forward layers, where the paper says the model does most of its computing, are 70.8M.
Quantization makes the imbalance worse. The paper notes that BitNet keeps its embedding and head in 16-bit, GPTQ and AWQ leave embed_tokens and lm_head untouched, and llama.cpp stores the output head at 6 to 8 bits. So once the body is compressed, the table is what is left. By the paper's estimate, a Gemma 3 270M with a ternary body and a 16-bit vocabulary would be about 20 MB of body and 335 MB of vocabulary. If you run small models locally (see our guide to running an AI model on your own laptop), the paper's point is that once the body is compressed, the table is most of the file.
The fix: 512 bits per token, frozen
The method takes Gemma 3 270M's own embedding matrix (262,144 tokens by 640), centers it, projects it onto the top 512 principal directions, and keeps only the sign of each projection. Each token gets a fixed 512-bit code. The whole vocabulary is 16.8 MB and never trains.
Around the frozen table, the model has:
- a small trained linear map from the 512-bit code into the body (512 to 640),
- another small map out of the body (640 to 512), whose output is scored against every code by a dot product,
- a fixed per-token prior, the log unigram frequency of each token in the training data, which the paper calls "the seal". It costs 1 MB and has no trainable parameters.
The two maps are 0.65M parameters. The paper reports that the codes keep most of what the float embedding knew: the 512 directions carry 92.3% of its variance, and on WordSim-353 the codes correlate with human similarity ratings at 0.617, against 0.614 for the float embedding.
The experiment
The study is controlled: same Gemma 3 270M body built from random initialization, same 2.05B tokens of FineWeb-Edu, same batches in the same order, same seed. Only the vocabulary changes.
- Arm A: the learned tied table, float body, 268.1M trainable parameters.
- Arm B: the frozen table and fixed seal, same float body, 101.0M trainable.
- Arm C: the frozen table with a ternary BitNet b1.58 body and a trained seal, 101.2M trainable.
Each arm trained on one RTX 5090 in 14 to 16 hours, which the author puts at about $18 per arm at the rental price paid. The whole study, including evaluation, cost under $75.
Results: one benchmark for 168M parameters
The learned table wins on loss: 3.155 nats against 3.358 for the frozen table, about 23% lower perplexity by the paper's count. On zero-shot benchmarks the picture is different. Comparing B to A on the same questions:
- LAMBADA: B is better by 1.6 points (23.2% vs 21.6%).
- HellaSwag: tied (30.4% both).
- PIQA: tied (61.5% vs 60.6%, range includes zero).
- ARC-Easy: A is better by 3.4 points (45.9% vs 42.5%).
The paper's summary: "That is the whole return on 168M trained parameters: one benchmark."
Where the learned table helps is also specific. Splitting validation loss by how often each target token appeared in training, A wins on tokens seen 100 to 100,000 times, where there are enough examples to train a row. On the 161 most common tokens, which are 52% of all text, the gap is 0.035 nats. On rare tokens seen 10 to 99 times, the frozen table wins by 2.3 nats (B) and 1.7 nats (C), because those learned rows never got enough examples while the codes already carried Gemma's meanings.
The 46 MB ternary model on a laptop CPU
Arm C is where this gets practical. The file the author runs is 46.2 MB. Arm A as a 16-bit GGUF is 543 MB. C ties A on LAMBADA and PIQA, trails by 0.9 points on HellaSwag and 4.9 on ARC-Easy, and is 0.28 nats behind in loss.
The author wrote a CPU kernel for C (one C file, AVX2) and ran A through llama.cpp on the same laptop (Intel Xeon W-11955M, 8 threads):
- A, GGUF f16: 543 MB file, 723 MB peak memory, 62 tokens per second.
- A, GGUF Q8_0: 292 MB, 471 MB, 116 tokens per second.
- A, GGUF Q4_K_M: 253 MB, 446 MB, 130 tokens per second.
- C, frozen ternary: 46 MB, 62 MB, 615 to 740 tokens per second.
Against A at Q8_0, which the paper says loses nothing in perplexity, C generates 5.3 to 6.4 times faster in 7.6 times less memory. The reason A cannot shrink further is the vocabulary: llama.cpp keeps the 262,144 by 640 table in 16-bit, and 250 MB of the 4-bit file is that table. For background on why quantizers treat embeddings this way, see our post-training quantization toolkit comparison.
Adding words after training
Because the codes come from a rule, a new word can get a code without training: take the mean of its pieces' Gemma rows and binarize it. The author added the 10,000 most frequent biomedical words from PubMed that Gemma's tokenizer splits into pieces. With no training, the learned table's loss rose 42.9% (nats per character); the frozen tables' loss rose about 3% (3.3% for B, 3.1% for C), while reading the text with 3.2% fewer tokens.
The paper is careful here. No construction made the cost zero, so it calls the claim "about possibility, not improvement." Two tests it expected to favor the frozen table did not: new control tokens such as <tool_call> were placed correctly by every arm from in-context examples, and emoji prediction was at chance for all three.
Limits worth knowing
- One seed per arm. Error bars cover test questions, not training runs.
- 2.05B tokens under-trains A's table. The author expects the loss gap at 20B tokens to be larger than 0.20, and says the benchmark picture may change.
- Arm C differs from B in three ways (ternary body, BitNet schedule, trained prior), and the schedule's share was not isolated.
- The CPU comparison is one laptop and two code bases (a custom kernel against llama.cpp). Only generation speed is compared, because the kernel's prefill runs one token at a time.
- The codes derive from Gemma 3 270M, so the table carries Google's Gemma terms, and the method needs a pretrained embedding to start from.
Why it matters
For anyone shipping sub-billion models to phones and laptops (we compared several in our sub-3B small model guide), the vocabulary is the part quantization does not touch. This paper puts a number on what it costs to drop it: a measurable loss gap, but small benchmark differences at this scale, and a 46 MB model that runs 5.3 to 6.4 times faster on CPU than the 8-bit baseline. It is one author, one seed and one data scale, and everything is released so others can check it. The next test is whether the benchmark ties hold when the learned table gets ten times more data.
