Subscribe
Explainers
8 min read
axc
axc

What Is a Small Language Model? SLM vs LLM Explained

What a small language model is, how SLMs are made from larger models, how they differ from LLMs, where they run, and when a small model fits the task.

Olivetti-style cover reading SMALL LANGUAGE MODELS, SLM vs LLM, with a small black square labeled SLM beside a large one labeled LLM, over teal and red blocks.
Olivetti-style cover reading SMALL LANGUAGE MODELS, SLM vs LLM, with a small black square labeled SLM beside a large one labeled LLM, over teal and red blocks.

A small language model, or SLM, is a language model with far fewer parameters than the large language models behind chatbots such as ChatGPT. IBM's explainer on small language models puts the gap this way: "In terms of size, SLM parameters range from a few million to a few billion, as opposed to LLMs with hundreds of billions or even trillions of parameters."

This guide explains what an SLM is, how developers make one, how it differs from an LLM, what it can and cannot do, and how to decide which type fits a task. Sources were checked on November 12, 2026 (PST).

What is a small language model?

IBM defines the term in one sentence: "Small language models (SLMs) are artificial intelligence (AI) models capable of processing, understanding and generating natural language content." In other words, an SLM does the same basic job as a large language model. It reads text and writes text. If you are new to the larger kind, start with our guide What Is an LLM?

Microsoft's Azure guide to SLMs adds the purpose: "SLMs are trained to perform specific tasks using fewer resources than larger models."

IBM says SLMs use the same building block as LLMs, a neural network design called the transformer. The difference is scale. According to IBM, SLMs need less memory and computing power than large models, which IBM says suits them to resource-constrained settings such as edge devices and mobile apps.

How big is "small"?

There is no single cutoff. The two vendor guides give ranges and examples, not a fixed line.

  • IBM: SLMs have from a few million to a few billion parameters, while LLMs have hundreds of billions or even trillions.
  • Microsoft: "Where LLMs like GPT-4 can contain more than a trillion parameters, an SLM might only contain a few hundred million."

A parameter is one of the numbers a model learns during training. IBM describes parameters as "internal variables, such as weights and biases, that a model learns during training." More parameters give a model more room to store patterns and facts. They also take more memory to load and more computation to run.

Microsoft sorts SLMs into three types:

  • Distilled versions of larger models, where a large "teacher" model trains a smaller "student" model.
  • Task-specific models, fine-tuned for one job such as sentiment analysis, translation or question answering.
  • Lightweight models, built with fewer parameters and architectures designed to keep computing demands low.

IBM lists these model families as examples of SLMs: DistilBERT, Gemma, GPT-4o mini, Granite, Llama, Ministral and Phi. IBM describes its own Granite series as LLM foundation models that come in small sizes, so the SLM label applies to specific model sizes, not to a whole family.

How are small language models made?

Some SLMs are trained from scratch at a small size. Others are made by shrinking a larger model. IBM describes four compression methods:

  • Pruning. This removes parameters that matter less to the result. IBM says pruned models often need fine-tuning afterward, and warns that "overpruning can degrade a model's performance."
  • Quantization. This stores the model's numbers at lower precision. IBM's example: weights "can be represented as 8-bit integers instead of 32-bit floating point numbers." Our guide What Do Q4 and Q8 Mean? explains the labels you see on downloadable model files.
  • Low-rank factorization. This replaces a large table of weights with a smaller approximation. IBM notes that it "can be computationally intensive and more difficult to implement."
  • Knowledge distillation. A larger teacher model trains a smaller student model to copy its answers. IBM calls it "a popular approach for many SLMs." Our explainer What Is Model Distillation? covers the method in more detail.

DistilBERT is a documented example of distillation. Its authors wrote in the DistilBERT paper on arXiv that "it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster." Those figures compare DistilBERT with Google's BERT model, so they describe that one pair of models, not SLMs in general.

SLM vs LLM: what are the differences?

Diagram comparing a small language model and a large language model on size, where each runs, resources and complex tasks: an SLM has a few million to a few billion parameters and runs on edge devices, mobile apps or on premises, while an LLM has hundreds of billions to trillions and needs much more processing power

The vendor guides describe the trade-off along five lines.

Size and hardware. By IBM's ranges, an SLM has a few million to a few billion parameters, and an LLM has hundreds of billions or more. Because an SLM needs less memory and computing power, it can run on a phone, a laptop or a single server.

Speed. IBM links speed to size: "Fewer parameters translate to decreased processing times, allowing SLMs to respond quickly."

Cost and energy. Microsoft's guide says LLMs "offer greater capabilities but require much more processing power." IBM lists "reduced cost" as a benefit and says SLMs "consume less energy" because they need fewer computing resources. Microsoft says the lower computing needs, shorter training time and lower energy use "result in lower overall costs."

Privacy and control. IBM says: "Because of their smaller size, SLMs can be deployed in private cloud computing environments or on premises, allowing for improved data protection and better management and mitigation of cybersecurity threats." When a model runs on your own hardware, your text does not have to go to a vendor's servers.

Complex tasks. Here the LLM has the advantage. Microsoft lists reduced accuracy on complex tasks as a limit and writes that "they may not be the best choice for applications that demand high accuracy, such as scientific research or medical diagnostics." IBM says SLMs "might be less proficient on complex tasks that require knowledge across a comprehensive spectrum of topics."

Microsoft's guide also says a narrow focus can help. It states that SLMs "can provide more accurate responses and information within the areas they're trained in." That claim applies to models trained for a specific domain, not to a general SLM asked an open question.

What are small language models good at?

Both guides point to tasks that are narrow, repeated, or need a fast answer.

IBM lists these use cases:

  • Customer service chatbots that respond to queries in real time.
  • Summarizing discussions, audio recordings and transcripts.
  • Generating, explaining and translating code.
  • Translating between languages.
  • Predictive maintenance, where a model runs on sensors or other edge devices and reads machine data.
  • Sentiment analysis, which sorts large volumes of text such as customer feedback.

Microsoft adds on-device examples: "predictive text input, virtual assistants, and offline translation services." It also names real-time work, such as chatbots and live transcription, where a short response time matters.

Where do small language models fall short?

Fewer parameters mean less room to store facts. IBM quotes Microsoft on its own Phi-3 models: "Phi-3 models do not perform as well on factual knowledge benchmarks as the smaller model size results in less capacity to retain facts."

IBM lists four limits:

  • Bias. "Smaller models can learn from the bias present in their larger counterparts."
  • Complex tasks. Performance drops on tasks that need broad knowledge.
  • Limited generalization. SLMs "lack the wide knowledge base of their expansive equivalents", so IBM says they may suit targeted tasks better.
  • Hallucinations. "Validating the results of SLMs is vital to make sure what they produce is factually correct."

Microsoft adds that SLMs "may struggle to grasp nuances, contextual subtleties, and intricate relationships within language."

How do SLMs run on a phone?

Google's Android documentation shows one working setup. IBM names Gemini Nano as a lightweight Gemini variant designed to operate on mobile devices. The Android developer page for Gemini Nano says the model "runs in Android's AICore system service, which leverages device hardware to enable low inference latency and keeps the model up-to-date."

Google describes what changes when the model runs on the phone: "On-device generative AI executes prompts locally, eliminating server calls." It also states the trade-off: "While this removes network latency, inference speed depends on device hardware." On privacy, Google says AICore "is built to isolate each request and doesn't store any record of the input data or the resulting outputs after processing them to protect user privacy."

The same page lists limits: "Although Gemini Nano is ideal for offline and privacy-sensitive tasks, it has on-device constraints such as token limits, device hardware limitations, and model capability boundaries." For those cases, Google points developers to hybrid inference through the Firebase AI Logic Hybrid API, which can route requests "to Gemini Flash or Gemini Pro models in the cloud when Gemini Nano is unavailable or unsupported on the user's device."

You can also run open SLMs on a computer. Our comparison Ollama vs LM Studio covers two apps that download and run open models on your own machine.

Can SLMs and LLMs work together?

Yes. IBM describes two patterns.

  • Hybrid AI. "A hybrid AI model can have smaller models running on premises and accessing LLMs in the public cloud when a larger corpus of data is required to respond to a prompt."
  • Intelligent routing. A routing module reads each request and picks a model. In IBM's words: "Small language models can handle basic requests, while large language models can tackle more complicated ones."

Google's Android setup, where Gemini Nano handles requests on the phone and cloud Gemini models take over when it cannot, follows the same idea.

AI agents are a newer case. A position paper on arXiv, "Small Language Models are the Future of Agentic AI", states: "Here we lay out the position that small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI." The authors add that "in situations where general-purpose conversational abilities are essential, heterogeneous agentic systems (i.e., agents invoking multiple different models) are the natural choice." This is an argument from the authors, not a measured result.

How to choose between an SLM and an LLM

The points below restate the trade-offs from the IBM, Microsoft and Google pages as questions to ask about your task.

  • Does the data have to stay on your device or network? SLMs can run on premises or on a device, which IBM and Google both describe.
  • Is the task narrow and repeated? Summaries, classification, short replies and translation are on IBM's and Microsoft's SLM use case lists.
  • Does the task need broad factual knowledge or multi-step reasoning? Both guides say SLMs lose accuracy on complex tasks, and Microsoft says LLMs "excel in handling complex tasks."
  • Is a wrong answer costly? Microsoft names scientific research and medical diagnostics as areas that demand high accuracy, where an SLM may not fit. Check any model's output in these cases, small or large.
  • Must it work offline? Google says Gemini Nano works without a network connection, and Microsoft lists offline translation as an SLM use case.
  • What hardware do you have? On-device speed depends on the device, as Google notes. A larger model needs more memory to load.
  • Can you use both? IBM's routing pattern sends basic requests to a small model and harder ones to a large model.

Sources

ExplainersLLMAI

Written by

axc
axc

Edited by the llms.blog desk.

More in Explainers

All Explainers →

Popular this week