A neural network is a computer program that learns a task from examples instead of following rules a person wrote by hand. IBM defines it this way: "A neural network is a machine learning model that stacks simple "neurons" in layers and learns pattern-recognizing weights and biases from data to map inputs to outputs." The same idea sits under image recognition, voice assistants and the chatbots in the news.
This guide explains what the parts of a neural network are, why layers alone are not enough, how a network learns, where it goes wrong, and where to study it further. Every claim links to the page it comes from, and where two sources differ we say so.
The short version
- A neural network is made of nodes, also called neurons, arranged in layers: an input layer, one or more hidden layers and an output layer, as AWS and IBM both describe.
- Each connection carries a number called a weight, and each neuron adds a number called a bias. Those weights and biases are what the network learns.
- Without an extra nonlinear step called an activation function, stacking layers adds nothing. Google's Machine Learning Crash Course says: "Linear operations performed on linear operations are still linear."
- Training adjusts the weights and biases to shrink the error. The usual method is backpropagation.
- Modern chatbots are neural networks too. IBM says transformers follow the same core principles: learned weights and biases, stacked layers, nonlinear activations and end-to-end training by backpropagation.
What a neural network is for
Google's course starts from a problem. Some data cannot be separated by a straight line, and fixing that by hand with feature crosses takes effort and experimentation. Google says neural networks are "a family of model architectures designed to find nonlinear patterns in data," and that during training the model automatically learns the optimal feature crosses to perform on the input data to minimize loss.
MIT News describes the everyday version. A computer learns to perform a task by analyzing training examples, and usually the examples have been hand-labeled in advance. An object recognition system might be fed thousands of labeled images of cars, houses and coffee cups, and it finds visual patterns in the images that consistently correlate with particular labels.
The parts: nodes, layers, weights and biases
A basic network has three kinds of layer, according to AWS:
- Input layer. Information from the outside world enters here, and the input nodes pass it to the next layer.
- Hidden layers. They take input from the input layer or from other hidden layers, process it further and pass it on. A network can have many of them.
- Output layer. It gives the final result. A yes or no question needs one output node. A question with many possible answers, such as which of ten digits an image shows, needs more than one.
Connections between nodes carry weights. AWS says a weight is positive if one node excites another and negative if it suppresses it, and nodes with higher weight values have more influence on the others. IBM compares weights to dials that control how strongly each input feature influences the decision, and biases to built-in values that shift the decision threshold.
IBM's example is spam detection. Words such as "prize," "money," "dear" or "win" go in as inputs. Early neurons weigh how important each signal is, later layers combine that into higher-level cues, and the final layer computes a probability that the email is spam.
What one neuron does
MIT News gives the plainest description of a single node. Each incoming connection has a weight. The node multiplies each incoming number by its weight and adds the results into one number. If that number is below a threshold value, the node passes no data to the next layer. If it exceeds the threshold, the node "fires," which in today's networks generally means sending the sum along its outgoing connections.
IBM writes the same recipe as two lines of math: a weighted sum of the inputs plus a bias, then an activation function applied to that sum. Google's nodes and hidden layers page says the value of each hidden neuron is calculated like the output of a linear model: the sum of each input times its own weight, plus the bias.
Why layers need an activation function
This is the step most beginner explanations skip. Google's course has readers explore a model with a hidden layer and then asks whether it can learn nonlinear patterns. The answer on the page is no. Every calculation is multiplication and addition, and the page says: "Linear calculations performed on the output of linear calculations are also linear." So the model still cannot learn nonlinearities.
The fix is an activation function, a nonlinear transform of a neuron's output before it goes to the next layer. Google names three common ones on its activation functions page:
- Sigmoid squashes any input to a value between 0 and 1.
- Tanh squashes any input to a value between -1 and 1.
- ReLU returns 0 for negative inputs and returns the input unchanged otherwise.
Google says ReLU often works a little better than sigmoid or tanh because it is less susceptible to the vanishing gradient problem during training, and it is significantly easier to compute. The course adds: "we still recommend starting with ReLU." The 3Blue1Brown lesson makes the same point from the other side: its author says nowadays people tend to use ReLU instead of sigmoid, and that without such a function the whole network would reduce to a linear function.
How many knobs a network has
Google's course gives a small counting exercise. A network with three inputs, one hidden layer of four neurons and one output has 21 parameters. Each of the four hidden neurons has 3 weights and a bias, which is 16 parameters, and the output has 4 weights and a bias, which is 5 more.
3Blue1Brown's lesson counts a bigger example built to recognize handwritten digits. Each digit image has 28 x 28 = 784 pixels, so the input layer has 784 neurons. The network has 2 hidden layers of 16 neurons each, which the lesson calls "kind of an arbitrary choice," and an output layer of 10 neurons. Its total is 13,002 weights and biases. The lesson says the entire network is just a function that takes in 784 numbers and returns 10.
How a network learns
MIT News says that at the start of training all of a network's weights and thresholds are set to random values. Training data goes in at the input layer, passes through the layers and arrives transformed at the output layer. During training the weights and thresholds are continually adjusted until training data with the same labels consistently yields similar outputs.
IBM breaks one round of that into four named stages, and the diagram below shows the loop:
- Forward pass. Inputs flow through the network and produce a prediction.
- Error calculation. A loss function measures the difference between the prediction and the true answer.
- Backward pass. The error is propagated backward through the network, and the algorithm works out how much each weight and bias contributed to it, using the chain rule of calculus.
- Weight update. Weights and biases are adjusted slightly in the direction that reduces the error, using an optimization method such as gradient descent.
IBM says this repeats many times over the training dataset, and that backpropagation coupled with gradient descent makes it possible for networks with millions or even billions of parameters to learn from massive datasets. Google's backpropagation page calls backpropagation the most common training algorithm for neural networks and notes that libraries such as Keras handle it automatically, so you do not need to do the calculus yourself.
Where neural networks go wrong
- Overfitting. IBM says that when a network has too many parameters it can fit its training data too closely and predict poorly on new data. MIT News describes the same thing as a network becoming so attuned to its training data that it fails to generalize to other instances of the same categories.
- Vanishing gradients. Google says gradients for the lower layers can become very small in deep networks, and layers with vanishing gradients train very slowly or not at all.
- Exploding gradients. If weights are very large, gradients can grow too large to converge. Google says batch normalization or a lower learning rate can help.
- Dead ReLU units. A ReLU unit can get stuck outputting 0, and gradients can no longer flow through it. Google suggests a lower learning rate or a variant such as LeakyReLU.
- Dropout. This is a fix rather than a failure. Google describes dropout regularization as randomly dropping out unit activations for a single gradient step, so that the model does not overfit.
- Opacity. MIT News says the networks' opacity is still unsettling to theorists. Enough training may let a network classify data usefully, but looking at the weights of individual connections will not answer what image features an object recognizer is looking at.
- Data hunger. AWS says deep networks need millions of examples of training data, where a simpler network might need perhaps hundreds or thousands.
Kinds of neural networks
IBM lists three architectures that sit on top of the basic design:
- Convolutional neural networks (CNNs) are designed for grid-like data such as images.
- Recurrent neural networks (RNNs) have feedback loops that let information persist across time steps, and suit speech and other sequences.
- Transformers use attention and, in IBM's words, replaced RNNs for many sequence tasks. IBM says they power models like GPT.
IBM says that regardless of architecture, all of them rely on artificial neurons, nonlinear activations and optimization algorithms. If a chatbot is the reason you are reading, that is the link: a large language model is a network of this kind at much larger scale. Our explainer on AI, machine learning and deep learning shows how the three terms nest, and What Is Generative AI? covers what these networks are used to generate.
A short history, with a date caveat
MIT News dates the first proposal of neural networks to 1944, by Warren McCulloch and Walter Pitts (the MIT article spells the surname McCullough). IBM gives 1943 for their first mathematical model of a neuron, so the two sources differ by a year. MIT says the first trainable network, the Perceptron, was demonstrated by Frank Rosenblatt in 1957, and IBM says he introduced it in 1958.
MIT's article, published April 14, 2017, says neural networks have been going in and out of fashion for more than 70 years. It says they have since returned, fueled largely by the increased processing power of graphics chips, and that modern GPUs let the one-layer networks of the 1960s and the two- to three-layer networks of the 1980s grow into the 10-, 15-, even 50-layer networks of its day. MIT says that is what the "deep" in "deep learning" refers to: the depth of the network's layers. Those layer counts are from 2017.
Where neural networks are used
IBM lists computer vision, natural language processing, speech recognition, forecasting and time series, reinforcement learning and pattern recognition such as fraud detection. AWS adds recommendation engines and lists medical diagnosis by medical image classification, financial predictions and electrical load forecasting among its use cases.
Where to learn more
- Google Machine Learning Crash Course, Neural networks module. The module gives an estimated length of 75 minutes. It assumes you have covered Introduction to Machine Learning, Linear regression, Logistic regression, Classification, Working with numerical data, Working with categorical data, and Datasets, generalization, and overfitting.
- 3Blue1Brown, Neural Networks, Chapter 1. The lesson, titled "But what is a Neural Network?", is a text adaptation by Josh Pullen of Grant Sanderson's video, dated October 5, 2017. It introduces the topic through recognizing handwritten digits.
- MIT News, Explained: Neural networks. The article is an explainer by Larry Hardesty, published April 14, 2017.
- AWS, What is a neural network? The page covers uses, architecture, types and training in short sections, and also describes AWS's own services.
- IBM, What is a neural network? The explainer, by IBM developer advocate Fangfang Lee, includes the math behind a neuron and the training loop.
For the math behind all of this, see our list of math for machine learning courses, books and videos. For a study order, try our 12-week plan to learn AI from scratch. Once the basics click, these posts show what is built on them: What Is Fine-Tuning?, What Is LoRA?, What Is RLHF?, What Is a Mixture of Experts Model? and What Are Embeddings?
Bottom line
A neural network is layers of simple units joined by weights, with an activation function between layers so the stack can learn nonlinear patterns. It learns by guessing, measuring the error and nudging every weight and bias to shrink that error, repeated many times. Our reading: if you can follow the single-neuron recipe and the four-stage training loop, you have the core idea that IBM says transformers share with the older designs.
Sources
- IBM: What is a neural network?
- Google Machine Learning Crash Course: Neural networks
- Google Machine Learning Crash Course: Nodes and hidden layers
- Google Machine Learning Crash Course: Activation functions
- Google Machine Learning Crash Course: Training using backpropagation
- MIT News: Explained: Neural networks
- AWS: What is a neural network?
- 3Blue1Brown: But what is a Neural Network?
