Recurrent Neural Networks
Reading a sentence means remembering what came before. A recurrent neural network (RNN) keeps a fixed-length vector as its notebook, rewrites it after each input, and uses the same parameters for every rewrite:
Here is the notebook up to the previous step, called the hidden state, and is the current input. Each is multiplied by a matrix, the results are added, and the sum passes through the nonlinearity to give the new state ; holds this step's output scores. Because the notebook has a fixed length, can only be a finite-dimensional summary learned for the objective; it cannot store every detail of the raw history. That compression enables streaming and also loses long-range detail.
Read horizontally to follow the hidden state through time, then vertically to connect an input to its output. Repeated boxes represent repeated use of shared parameters, not separately trained networks. A later output can depend on earlier inputs through this state path.
One Number Can Serve as a State: Streaming Smoothing
Start with a state that is a single number; the gradient section below reuses it. Consider the scalar recurrence
Each new value keeps a share of the old state and gives the new value a share . This is an exponential moving average. Each update uses constant memory. The newest observation has weight , and an observation steps old has weight .
Take , , and inputs :
Expanded with the weight formula, : the earliest input, 2, keeps only one eighth of its weight.
The example shows both sides of recurrence: an RNN can update online without storing the whole sequence, but early information decays according to a fixed rule, and one old value cannot be reread on demand. A learned RNN is more flexible, yet remains constrained by its finite state and training signal. Attention changes this trade-off by retaining multiple represented positions and accessing them by content.
Sequence Objectives Come in Several Forms
Autoregressive modeling factorizes
That is, the probability of the whole sequence is the product, step by step, of the probability of each value given the values before it. Training often uses teacher forcing, which feeds the true prefix at every step. At inference the model instead sees its own earlier outputs, which may contain errors, so low training loss does not guarantee stable free generation. This gap between training and generation is called exposure bias; longer training does not remove it, and the remedy depends on the task.
Gradients Multiply Through Time, So They Vanish or Explode
Training an RNN unrolls the loop through time and differentiates backward from the end; this is backpropagation through time (BPTT). The path from time to contains a Jacobian product:
In words, each step back multiplies the gradient once more by “activation slope × .” The smoothing example is the simplest case: every factor is , so the sensitivity to after three steps is , and after fifty steps it is ; early inputs receive almost no training signal. An ordinary RNN multiplies by a similar factor at every step. Repeated contraction causes vanishing gradients; repeated expansion causes exploding gradients. Sequence length, weight spectra, activation saturation, and the trajectories in the data together decide which happens.
- Gradient clipping limits large updates but cannot restore vanished gradients.
- Truncated BPTT backpropagates only a fixed number of steps, reducing memory and computation while also cutting long-range credit assignment, that is, deciding which earlier input the final error should be attributed to.
- Suitable initialization, normalization, and gates can improve training, but memory length remains finite.
For clarity, define . The ordered product is ; matrix factors cannot generally be reversed. The same recurrent parameters appear at every step, so their gradient sums contributions from all uses, including indirect paths through later states. This is the chain-rule mechanism behind BPTT's gradient difficulties.
Code usually truncates with detach: the value still flows forward, but the gradient path is cut at that point. In the smoothing example, detaching the state after step two keeps its value for the next forward step but removes the gradient path to the earlier chunk. Detaching is not resetting: a reset would replace with the initial value. For padded sequences, masking only the loss is insufficient if a final state is consumed: skip padded state updates or select the last valid state.
LSTM Uses an Additive Path to Carry Information Forward Unchanged
A long short-term memory network (LSTM) maintains a cell state alongside the hidden state . The forget gate scales the previous cell state, the input gate scales the candidate update , and the output gate controls how the cell state contributes to . Here is the sigmoid function, means elementwise multiplication, and concatenates the two vectors. In these equations, is the output gate, a different quantity from the output scores in the opening RNN equation. A common form is:
The key is that the line is an addition. With the forget gate near 1 and the input gate near 0, : the cell state is copied to the next step unchanged, and the derivative along this path is also close to 1, instead of being shrunk by a multiplication with at every step as in an ordinary RNN. Information and gradients can therefore travel across many more steps. Implementations vary in biases, projections, peepholes, and gate order.
In Distill’s Visualizing memorization in RNNs, select characters in the autocomplete examples and inspect their connections to earlier input. The displayed connections come from gradient-based diagnostics on those examples. Comparing GRU and LSTM shows that similar aggregate scores can hide different ways of using context.
State and Direction Must Match the Task
- Causal RNNs use only current and past inputs, so they support streaming prediction.
- Bidirectional RNNs also use future context, so they do not suit online decisions where that context is unavailable.
- Reset hidden state between independent entities unless carrying it across the boundary has explicit meaning.
State-Based Modeling Continued After RNNs
State-based modeling kept developing after classical RNNs. Structured state-space models (SSMs) start from continuous- or discrete-time linear state equations and use structured parameters and parallel algorithms to combine long-sequence training with recurrent inference. S4 demonstrated the long-sequence capabilities of structured SSMs; Mamba made parts of the update input-dependent and introduced a hardware-aware scan; xLSTM revisited LSTM memory and gates. A classical RNN's state transition is nonlinear, while an SSM uses structured state equations; these names refer to distinct architectures.
RNNs, SSMs, and Full Self-Attention Pay Different Costs
The choice depends on latency, memory, sequence length, training scale, kernels, and task structure. Paper benchmarks remain tied to data, scale, kernels, and training budgets, and need replication on the target task.
Hybrid Models Now Ship With a KV Cache in Only a Few Layers
By October 2026, models that mix state-based layers with a few full-attention layers ship as open weights. The Qwen3-Next-80B-A3B model card gives its layer layout as 12 * (3 * (Gated DeltaNet -> MoE) -> 1 * (Gated Attention -> MoE)): of 48 layers, three in every four are Gated DeltaNet and one is full attention. Its native context is 262,144 tokens, and full_attention_interval: 4 in its configuration file records that spacing.
Gated DeltaNet is a form of linear attention. Like the RNNs on this page, it compresses what it has read into a fixed-size state that does not grow with length. Only the 12 full-attention layers store a key and a value for every token read. In this model, then, the KV cache that grows with context comes from a quarter of the layers. How much memory the KV cache takes, and how to estimate it, is worked out in Attention Variants and KV-Cache Compression.