Natural language processing, or NLP, is the part of AI that deals with human language. Search engines, spam filters, translation tools and customer service chatbots all rely on it. Hugging Face's free course calls large language models (LLMs) such as GPT and Llama "the latest advancement in the field."
This guide explains what NLP is, how it works step by step, what tasks it covers, where it still struggles, and which free courses and books can teach it to you.
What is NLP?
IBM defines natural language processing as "a subfield of computer science and artificial intelligence (AI) that uses machine learning to enable computers to understand and communicate with human language." The same page says NLP lets computers "recognize, understand and generate text and speech."
Hugging Face's free LLM Course puts it slightly differently: "NLP is a field of linguistics and machine learning focused on understanding everything related to human language." It adds that the aim "is not only to understand single words individually, but to be able to understand the context of those words."
Both definitions point at the same idea. Language is messy, full of double meanings, slang and context. NLP is the set of methods that turns that mess into something a computer can work with.
Where do you already use NLP?
According to IBM, NLP already powers:
- Search engines that try to understand the intent behind a query, not only its keywords
- Customer service chatbots
- Voice-operated GPS systems
- Phone assistants such as Amazon's Alexa and Apple's Siri
- Translation tools that convert text from one language to another
- Sentiment analysis on customer reviews and social media posts
Stanford's CS224N course page lists a similar spread: "web search, advertising, emails, customer service, language translation, virtual agents, medical reports, politics, etc." The reason, in the course's words, is that "people communicate almost everything in language."
What tasks does NLP handle?
Hugging Face groups common NLP tasks into five kinds:
- Classifying whole sentences: getting the sentiment of a review, or detecting whether an email is spam.
- Classifying each word: finding grammatical parts (noun, verb, adjective) or named entities (person, location, organization).
- Generating text: completing a prompt, or filling in masked words.
- Extracting an answer: given a question and a passage, pulling out the answer.
- Generating a new sentence from input text: translation and summarization.
Hugging Face also notes that NLP "isn't limited to written text." It covers speech recognition and some computer vision work, such as writing a transcript of audio or a description of an image.
IBM's explainer names four lower-level tasks that help a computer make sense of a sentence:
- Coreference resolution: working out who or what a word refers to, such as linking "she" to "Mary."
- Named entity recognition (NER): tagging "London" as a location or "Maria" as a person's name.
- Part-of-speech tagging: deciding that "make" is a verb in "I can make a paper plane" but a noun in "What make of car do you own?"
- Word sense disambiguation: choosing the right meaning of a word with several meanings, based on context.
How does NLP work?
IBM describes a typical NLP pipeline in four stages.
Text preprocessing
Raw text is cleaned and split up. This starts with tokenization, which breaks text into smaller units such as words or phrases. (Modern AI models work in tokens too.) Other common moves are lowercasing so "Apple" and "apple" match, removing stop words such as "is" and "the," and stemming or lemmatization, which cuts "running" down to "run."
Feature extraction
Computers need numbers, not words. Feature extraction turns text into numerical form. Older methods such as Bag of Words and TF-IDF count how often words appear and how important they are in a document. Newer methods such as Word2Vec and GloVe represent words as vectors that capture meaning. Contextual embeddings go further and take the surrounding words into account. Our explainer on embeddings covers this idea in more depth.
Text analysis
With text in numerical form, the system can run analysis tasks: part-of-speech tagging, named entity recognition, dependency parsing (how words relate grammatically), sentiment analysis, and topic modeling. IBM describes natural language understanding (NLU) as "a subset of NLP that focuses on analyzing the meaning behind sentences."
Model training
The processed data trains a machine learning model, which adjusts its parameters to reduce errors. IBM says the model is then refined "through evaluation, validation and fine-tuning." If fine-tuning is new to you, see our guide on what fine-tuning is.
Stanford's CS224N page notes that deep learning brought "single end-to-end neural models that did not require traditional, task-specific feature engineering." Our reading: modern systems do not always run these stages as separate steps, but the stages are still a useful map of what has to happen for a computer to handle language.
What are the main approaches to NLP?
IBM sorts NLP into three approaches that arrived roughly in order.
Rules-based NLP
The earliest systems were "simple if-then decision trees, requiring preprogrammed rules," IBM says, citing the original version of Moviefone. Because they have no machine learning, IBM calls this approach "highly limited and not scalable."
Statistical NLP
Statistical NLP uses machine learning to label parts of text and assign "a statistical likelihood to each possible meaning." IBM credits it with mapping words and grammar to vector representations, and with early tools such as spellcheckers and T9 texting on phone keypads.
Deep learning NLP
Deep learning, using neural networks, is now the dominant approach, according to IBM. It includes:
- Sequence-to-sequence models, based on recurrent neural networks, used mostly for machine translation.
- Transformer models, which use tokenization and self-attention to work out how parts of a text relate. IBM names Google's BERT as "a landmark in transformer models." Our plain-English guide to transformers explains attention.
- Autoregressive models, transformers trained to predict the next word. IBM's examples are GPT, Llama, Claude and Mistral.
Stanford's CS224N page sums up the shift: in the 2010s, deep learning "obtained very high performance across many different NLP tasks, using single end-to-end neural models that did not require traditional, task-specific feature engineering." In the 2020s, "amazing further progress was made through the scaling of Large Language Models, such as ChatGPT."
How are NLP and LLMs related?
A large language model is one kind of NLP system. Hugging Face describes LLMs as "a significant advancement in the field of natural language processing (NLP)."
The bigger change is in how NLP gets built. Hugging Face says LLMs "shifted the paradigm from building specialized models for specific NLP tasks to using a single, large model that can be prompted or fine-tuned to address a wide range of language tasks." Our reading: tasks such as spam detection and translation, which once each called for a specialized model, can now be handled by prompting one general model.
For a fuller picture, read what an LLM is. If you want to see where NLP fits among AI, machine learning and deep learning, our AI vs machine learning vs deep learning explainer lays out the layers.
What are the limits of NLP?
Language is hard for machines, and both sources list open problems.
IBM names four risks:
- Biased training data: biased data skews answers, and training datasets scraped from the web "are prone to bias."
- Misinterpretation: speech systems can get confused by obscure dialects, mumbling, slang, homonyms or background noise.
- New vocabulary: new words and broken grammar rules leave the system to make a best guess or admit it is unsure.
- Tone of voice: sarcasm, exaggeration and stress can change meaning in ways the words alone do not show.
Hugging Face lists limits specific to LLMs: they can generate incorrect information confidently (see our guide to AI hallucinations), they can reproduce bias, they have limited context windows, and they need significant computing power. It adds that ambiguity, cultural context, sarcasm and humor remain hard, and that LLMs "still often fall short of human-level understanding in many complex scenarios."
How can you learn NLP for free?
These resources are checked as of November 2026. Each claim about cost comes from the resource's own page.
- Hugging Face LLM Course. Teaches LLMs and NLP "using libraries from the Hugging Face ecosystem," and says it began as an NLP course before shifting its focus to LLMs. The course page says: "It's completely free and without ads."
- Stanford CS224N: Natural Language Processing with Deep Learning. Stanford's NLP course. The course page says complete lecture videos are free on the CS224N 2024 YouTube playlist. Slides and assignments are posted on the course site. The page notes the course uses the PyTorch framework.
- Speech and Language Processing (3rd edition draft) by Dan Jurafsky and James H. Martin. A textbook covering words and tokens, embeddings, transformers, RAG, speech recognition and more. The authors post draft chapters and a single PDF online; the latest release on the page is dated August 19, 2026.
- Natural Language Processing with Python (NLTK book) by Steven Bird, Ewan Klein and Edward Loper. A hands-on book built on the Natural Language Toolkit, readable online under a Creative Commons license. IBM describes NLTK as "a suite of libraries and programs for English that is written in the Python programming language."
- Advanced NLP with spaCy. The course page calls it a "free and interactive online course" on using spaCy "to build advanced natural language understanding systems, using both rule-based and machine learning approaches."
Most of these assume some Python. If you need to start there, see our list of free Python courses for AI beginners. For courses focused on large language models specifically, see free LLM courses.
Bottom line
NLP is the branch of AI that lets computers read, sort and write human language. It grew from hand-written rules, to statistical models, to the deep learning systems behind today's chatbots. LLMs are the latest step, but the older ideas (tokenization, embeddings, tagging, sentiment) still explain what those models are doing. For more terms like these, see our AI glossary and our explainer on RAG.
Sources
- IBM Think: What is NLP (natural language processing)?
- Hugging Face LLM Course, Chapter 1: Natural Language Processing and Large Language Models
- Hugging Face LLM Course, Introduction
- Stanford CS224N: Natural Language Processing with Deep Learning
- CS224N 2024 lecture playlist on YouTube
- Jurafsky and Martin, Speech and Language Processing (3rd ed. draft)
- NLTK book: Natural Language Processing with Python
- Advanced NLP with spaCy
