Subscribe
AI
6 min read

What Is an LSTM? The Architecture That Ran Language Models Before Transformers

For most of the 2010s, every serious language model was an LSTM. Translation systems, speech recognition, the first contextual word embeddings: all recurrent, all gated, none of them transformers. The architecture was proposed in 1997 to fix a specific mathematical failure in earlier networks, and it held the field for roughly twenty years before Attention Is All You Need retired it. Understanding why LSTMs worked, and why they lost, explains a lot about how modern LLMs manage memory today. Th

What Is an LSTM? The Architecture That Ran Language Models Before Transformers

For most of the 2010s, every serious language model was an LSTM. Translation systems, speech recognition, the first contextual word embeddings: all recurrent, all gated, none of them transformers. The architecture was proposed in 1997 to fix a specific mathematical failure in earlier networks, and it held the field for roughly twenty years before Attention Is All You Need retired it. Understanding why LSTMs worked, and why they lost, explains a lot about how modern LLMs manage memory today.

The problem: gradients that die on the way backward

A recurrent neural network (RNN) processes a sequence one element at a time. It keeps a hidden state, a fixed-size vector, and at each step it updates that state from the previous state plus the current input, using the same weight matrix every step. That weight sharing is the whole idea: the network carries a summary of everything it has read so far inside one small vector.

Training such a network meant backpropagation through time, Paul Werbos's 1990 extension of backpropagation to recurrent graphs. The network is conceptually unrolled across time steps, and the error signal flows backward through every unrolled copy of the weight matrix, multiplying by the same Jacobian at each step.

That multiplication is the flaw. An error signal traveling back 50 steps passes through 50 matrix products. Depending on the weights, it either shrinks toward zero or blows up, exponentially in the number of steps. In 1991 Sepp Hochreiter formalized the failure in his diploma thesis, and the 1994 paper Learning Long-Term Dependencies with Gradient Descent is Difficult by Yoshua Bengio, Patrick Simard and Paolo Frasconi proved the dilemma is structural: a recurrent network that remembers inputs for a long time is precisely a network whose gradient signal vanishes over the same span. Plain RNNs in practice failed to learn dependencies more than five to ten steps apart.

The fix: a conveyor belt with three doors

Hochreiter and Schmidhuber's 1997 paper Long Short-Term Memory, published in Neural Computation, added one component: a memory cell whose state evolves additively rather than multiplicatively. Inside the cell, error signals flow backward unchanged, what the paper called a Constant Error Carousel. A value written into the cell can persist indefinitely because keeping it costs nothing; the decay only happens where a gate says so.

Three gates, sigmoid units that output a value between 0 and 1 and multiply it against a signal, control the cell:

  • An input gate decides how much of the current candidate value gets written into the cell.
  • An output gate decides how much of the cell state is exposed to the hidden state and the next layer.
  • A forget gate, added by Felix Gers, Schmidhuber and Fred Cummins in Learning to Forget (1999), decides how much of the old cell state to keep. The original 1997 cell had no forget gate and could not reset itself, which broke down on continuous input streams where the state grew without bound.

The intuitive picture: the cell is a conveyor belt running through time, and the gates are workers allowed to add items, remove items, or read the belt, but never to shake it. Information can ride the belt hundreds of steps while gradients stay stable.

A common simplification is the gated recurrent unit (GRU), published by Kyunghyun Cho and colleagues in 2014, which merges the cell state and hidden state into one and uses two gates instead of three. It is cheaper to train and usually within a point or two of LSTM quality, which made it the default in many production systems of the mid-2010s.

An LSTM memory cell as a conveyor belt with gate mechanisms

The empire: 1999 to 2017

LSTMs became the workhorse of sequence learning once they scaled. Three results marked the peak.

In 2013, Graves, Mohamed and Hinton showed deep bidirectional LSTMs trained end to end hit 17.7% error on the TIMIT phoneme recognition benchmark, state of the art at the time, and pushed LSTM-based acoustic models into commercial speech recognition shortly after.

In 2014, Sutskever, Vinyals and Le built the dominant neural machine translation system of the era from two LSTMs, one reading a sentence and compressing it into a fixed 1,000-dimensional vector, the other generating the translation from that vector. The paper itself documents the architecture's ceiling: quality degraded on sentences longer than roughly 30 words, and the authors recovered accuracy by reversing the order of the target sentence so the first output word sat near the last input word.

The bottleneck of that fixed-length vector is why attention exists at all. Bahdanau, Cho and Bengio added a mechanism letting the decoder LSTM peek at every encoder state rather than a single summary, and translation quality jumped. Attention, the thing that would later define transformers, was invented as an accessory to an LSTM.

The last great standalone LSTM result in NLP was ELMo in 2018: a two-layer bidirectional LSTM that produced context-dependent word representations, so the word bank in a river sentence got a different embedding than the word bank in a finance sentence. ELMo briefly topped almost every NLP benchmark before being displaced by BERT within months.

Why transformers won

Two properties ended the LSTM era, and both are about parallelism.

An LSTM cannot compute step t until step t-1 is done. The recurrence is a serial dependency chain exactly as long as the sequence, so training on a book chapter means tens of thousands of sequential updates per layer, regardless of how many GPUs you own. The transformer dropped recurrence entirely: every token in the sequence is processed in the same matrix multiply, and the 2017 paper cited reduced sequential operations and longer backpropagation paths as its core advantages.

The second property is path length. For two words 200 positions apart to interact in an LSTM, information must survive 200 gated updates. In a transformer, attention connects them in one step. Long-range dependencies stopped being a memory problem and became a retrieval problem.

Serial recurrent chain versus fully connected attention grid

The price the transformer pays is that it never summarizes. Every past token's key-value pair must be kept verbatim in the KV cache at inference time, which is why the KV cache dominates memory cost in long-context serving and why the industry spent 2024-2026 inventing cache eviction, quantization and prefix sharing. The LSTM had solved the same economics the other way: a hidden state of fixed size, no matter the context length. The trade is one vector that loses detail versus a cache that loses nothing but grows without bound.

The revival

The original authors never stopped. Hochreiter's group published xLSTM in 2024, extending the cell with exponential gating and a matrix-valued memory (the mLSTM, whose update rule is a covariance sum and therefore fully parallelizable at training time). xLSTM language models scaled past a billion parameters and reported competitive perplexities against same-size transformers and state space models. No xLSTM system has come close to displacing the transformer stack in production, but the line of work keeps the recurrent bet alive: inference-time memory that does not grow with context, on hardware that modern kernels can finally parallelize.

If you have ever wondered why an LLM's context window is a hard product boundary with a price tag attached, the LSTM is the counterfactual. It was the architecture that made context free and lossy, and it lost to the one that makes context exact and expensive.

Sources

AILLMEducation