This is an opinion piece. The facts come from the linked paper. The argument is mine.
When people build an AI agent, the default choice is one large, general-purpose model behind every step. My view is that this default is mostly habit. A lot of what an agent does all day is narrow and repetitive, and for that kind of work a smaller model, tuned for the job, is the better first thing to try. The case is not airtight, and the best evidence for it comes from a paper that argues a position, not from an experiment. But the argument is worth taking seriously because it changes a decision any team building agents makes early and rarely revisits.
If some of the terms here are new, our AI Glossary defines the basics.
What the paper argues
The paper is Small Language Models are the Future of Agentic AI, by Peter Belcak and colleagues. The authors' affiliations include NVIDIA Research and the Georgia Institute of Technology. The paper argues a view rather than reporting an experiment: the authors say they "lay out the position" that small language models are "sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems".
Their starting observation is about what agents actually ask of a model. In the abstract, they say agentic systems are producing "a mass of applications in which language models perform a small number of specialized tasks repetitively and with little variation". Later they write that "the majority of agentic subtasks in deployed agentic systems are repetitive, scoped, and non-conversational".
They also define the term. Their working definition of a small language model is one "that can fit onto a common consumer electronic device and perform inference with latency sufficiently low to be practical when serving the agentic requests of one user". Anything that is not small by that test counts as a large language model. The definition is about where a model can run, not about a fixed parameter count. The authors say a definition should avoid hardware-specific metrics like parameter count, "which quickly become obsolete as technology advances".
On cost, the authors state that serving a 7bn-parameter small model is 10 to 30 times cheaper, in latency, energy consumption and FLOPs, than serving a 70bn to 175bn-parameter large model. They attribute that comparison to other work, so read it as their summary of the literature, not a result they measured themselves. Our posts on small language models in production and on open-source sub-3B small language models cover the serving side and some candidate models.
The part I find most useful: you do not have to pick one
The paper does not say large models are obsolete. It says that where "general-purpose conversational abilities are essential", mixed systems are "the natural choice", meaning agents that call more than one model. Elsewhere it spells out the shape: "SLMs are used by default and LLMs are invoked selectively and sparingly".
That is a design rule you can apply without believing any big forecast. For each step an agent takes, ask what kind of work it is. A step that always returns the same structured format, or parses a command, or fills a template, is a candidate for a small model. A step that needs open-ended reasoning or conversation stays with a large one.
The paper even sketches how to get there. Its conversion outline starts with logging the agent's model calls, then curating that data, grouping similar tasks, picking candidate small models and fine-tuning them. The authors give "10k-100k examples" as a rule of thumb for fine-tuning small models. Knowledge distillation, where a small model is trained to mimic a larger one, is mentioned as an option in the fine-tuning step. Our explainer What Is Model Distillation? covers how that works.
How much work could move?
The authors run three case studies on open-source agents in an appendix. For MetaGPT, they write "we estimate that about 60% of its LLM queries could be reliably handled by appropriately specialized SLMs". For Open Operator the estimate is about 40%, and for Cradle it is about 70%.
These are estimates, and the authors label them that way. They are not measurements of a deployed system. Still, the spread is informative. Even the lowest figure, about 40%, would mean a sizeable share of calls going to a cheaper model, and the highest, about 70%, is still short of everything. That matches my reading of the paper: the claim is not "replace the big model" but "stop sending every step to it".
Where I would be careful
The authors are open about the weak points. On the economics of running many small specialized models against centralized large ones, they write that the "exact economical considerations" are "highly case-specific", and that "the jury is still out". One counterargument the paper lists is that the costs of inference infrastructure setup, and of acquiring and retaining talent for its upkeep, are often left out of cost calculations. They weigh more if the agent developer becomes responsible for deploying small models.
The paper also names barriers to adoption. It lists "Large amounts of upfront investment in centralized LLM inference infrastructure" as a barrier, and "Lack of popular awareness". The paper's own authors include a disclaimer that their views "do not necessarily reflect the views or positions of any entities they represent". I would apply the same caution to the paper as a whole: it is an argument from people who have a view on where agent costs should go, and it deserves testing, not trust.
What I would do with it
If you build or buy agents, this paper suggests a short, practical audit. List the steps your agent repeats most often. Pick the one that is most narrow and most frequent. Try a small model on that step alone, using real examples from your own logs, and compare it to what you use now. If it holds up, keep it, and keep the large model for the steps that need it. If it does not, you have spent one afternoon and learned where your agent really needs the big model.
The same question applies to coding agents. If you want a refresher on what they do, our guide What Is a Coding Agent? walks through the main ones. Whatever the product, the question underneath is which of its steps actually need the biggest model. The paper's answer, as I read it, is fewer than most teams assume. I think that is a better starting assumption than all of them, and cheap enough to check.
