For most of the 2010s, every serious language model was an LSTM. Translation systems, speech recognition, the first contextual word embeddings: all recurrent, all gated, none of them transformers. The architecture was proposed in 1997 to fix a specific mathematical failure in earlier networks, and it held the field for roughly twenty years before Attention Is All You Need retired it. Understanding why LSTMs worked, and why they lost, explains a lot about how modern LLMs manage memory today.
The problem: gradients that die on the way backward
A recurrent neural network (RNN) processes a sequence one element at a time. It keeps a hidden state, a fixed-size vector, and at each step it updates that state from the previous state plus the current input, using the same weight matrix every step. That weight sharing is the whole idea: the network carries a summary of everything it has read so far inside one small vector.
Training such a network meant backpropagation through time, Paul Werbos's 1990 extension of backpropagation to recurrent graphs. The network is conceptually unrolled across time steps, and the error signal flows backward through every unrolled copy of the weight matrix, multiplying by the same Jacobian at each step.
That multiplication is the flaw. An error signal traveling back 50 steps passes through 50 matrix products. Depending on the weights, it either shrinks toward zero or blows up, exponentially in the number of steps. In 1991 Sepp Hochreiter formalized the failure in his diploma thesis, and the 1994 paper Learning Long-Term Dependencies with Gradient Descent is Difficult by Yoshua Bengio, Patrick Simard and Paolo Frasconi proved the dilemma is structural: a recurrent network that remembers inputs for a long time is precisely a network whose gradient signal vanishes over the same span. Plain RNNs in practice failed to learn dependencies more than five to ten steps apart.
The fix: a conveyor belt with three doors
Hochreiter and Schmidhuber's 1997 paper Long Short-Term Memory, published in Neural Computation, added one component: a memory cell whose state evolves additively rather than multiplicatively. Inside the cell, error signals flow backward unchanged, what the paper called a Constant Error Carousel. A value written into the cell can persist indefinitely because keeping it costs nothing; the decay only happens where a gate says so.
Three gates, sigmoid units that output a value between 0 and 1 and multiply it against a signal, control the cell:
- An input gate decides how much of the current candidate value gets written into the cell.
- An output gate decides how much of the cell state is exposed to the hidden state and the next layer.
- A forget gate, added by Felix Gers, Schmidhuber and Fred Cummins in Learning to Forget (1999), decides how much of the old cell state to keep. The original 1997 cell had no forget gate and could not reset itself, which broke down on continuous input streams where the state grew without bound.
The intuitive picture: the cell is a conveyor belt running through time, and the gates are workers allowed to add items, remove items, or read the belt, but never to shake it. Information can ride the belt hundreds of steps while gradients stay stable.
A common simplification is the gated recurrent unit (GRU), published by Kyunghyun Cho and colleagues in 2014, which merges the cell state and hidden state into one and uses two gates instead of three. It is cheaper to train and usually within a point or two of LSTM quality, which made it the default in many production systems of the mid-2010s.

The empire: 1999 to 2017
LSTMs became the workhorse of sequence learning once they scaled. Three results marked the peak.
In 2013, Graves, Mohamed and Hinton showed deep bidirectional LSTMs trained end to end hit 17.7% error on the TIMIT phoneme recognition benchmark, state of the art at the time, and pushed LSTM-based acoustic models into commercial speech recognition shortly after.
In 2014, Sutskever, Vinyals and Le built the dominant neural machine translation system of the era from two LSTMs, one reading a sentence and compressing it into a fixed 1,000-dimensional vector, the other generating the translation from that vector. The paper itself documents the architecture's ceiling: quality degraded on sentences longer than roughly 30 words, and the authors recovered accuracy by reversing the order of the target sentence so the first output word sat near the last input word.
The bottleneck of that fixed-length vector is why attention exists at all. Bahdanau, Cho and Bengio added a mechanism letting the decoder LSTM peek at every encoder state rather than a single summary, and translation quality jumped. Attention, the thing that would later define transformers, was invented as an accessory to an LSTM.
The last great standalone LSTM result in NLP was ELMo in 2018: a two-layer bidirectional LSTM that produced context-dependent word representations, so the word bank in a river sentence got a different embedding than the word bank in a finance sentence. ELMo briefly topped almost every NLP benchmark before being displaced by BERT within months.
Why transformers won
Two properties ended the LSTM era, and both are about parallelism.
An LSTM cannot compute step t until step t-1 is done. The recurrence is a serial dependency chain exactly as long as the sequence, so training on a book chapter means tens of thousands of sequential updates per layer, regardless of how many GPUs you own. The transformer dropped recurrence entirely: every token in the sequence is processed in the same matrix multiply, and the 2017 paper cited reduced sequential operations and longer backpropagation paths as its core advantages.
The second property is path length. For two words 200 positions apart to interact in an LSTM, information must survive 200 gated updates. In a transformer, attention connects them in one step. Long-range dependencies stopped being a memory problem and became a retrieval problem.

The price the transformer pays is that it never summarizes. Every past token's key-value pair must be kept verbatim in the KV cache at inference time, which is why the KV cache dominates memory cost in long-context serving and why the industry spent 2024-2026 inventing cache eviction, quantization and prefix sharing. The LSTM had solved the same economics the other way: a hidden state of fixed size, no matter the context length. The trade is one vector that loses detail versus a cache that loses nothing but grows without bound.
The revival
The original authors never stopped. Hochreiter's group published xLSTM in 2024, extending the cell with exponential gating and a matrix-valued memory (the mLSTM, whose update rule is a covariance sum and therefore fully parallelizable at training time). xLSTM language models scaled past a billion parameters and reported competitive perplexities against same-size transformers and state space models. No xLSTM system has come close to displacing the transformer stack in production, but the line of work keeps the recurrent bet alive: inference-time memory that does not grow with context, on hardware that modern kernels can finally parallelize.
If you have ever wondered why an LLM's context window is a hard product boundary with a price tag attached, the LSTM is the counterfactual. It was the architecture that made context free and lossy, and it lost to the one that makes context exact and expensive.
Sources
- Hochreiter & Schmidhuber, Long Short-Term Memory, Neural Computation 1997
- Bengio, Simard & Frasconi, Learning Long-Term Dependencies with Gradient Descent is Difficult, IEEE TNN 1994
- Werbos, Backpropagation through time: what it does and how to do it, Proceedings of the IEEE 1990
- Gers, Schmidhuber & Cummins, Learning to Forget: Continual Prediction with LSTM, Neural Computation 2000
- Cho et al., Learning Phrase Representations using RNN Encoder-Decoder (GRU), 2014
- Graves, Mohamed & Hinton, Speech Recognition with Deep Recurrent Neural Networks, ICASSP 2013
- Sutskever, Vinyals & Le, Sequence to Sequence Learning with Neural Networks, 2014
- Bahdanau, Cho & Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, 2014
- Peters et al., Deep Contextualized Word Representations (ELMo), 2018
- Vaswani et al., Attention Is All You Need, 2017
- Beck et al., xLSTM: Extended Long Short-Term Memory, 2024
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023
