Skip to main content

Recurrent Neural Networks

Reading a sentence means remembering what came before. A recurrent neural network (RNN) keeps a fixed-length vector as its notebook, rewrites it after each input, and uses the same parameters for every rewrite:

ht=ϕ(Wxhxt+Whhht−1+bh),ot=Whoht+bo.\mathbf{h}_t =\phi(W_{xh}\mathbf{x}_t+W_{hh}\mathbf{h}_{t-1}+\mathbf{b}_h), \qquad \mathbf{o}_t=W_{ho}\mathbf{h}_t+\mathbf{b}_o.

Here ht−1\mathbf h_{t-1} is the notebook up to the previous step, called the hidden state, and xt\mathbf x_t is the current input. Each is multiplied by a matrix, the results are added, and the sum passes through the nonlinearity ϕ\phi to give the new state ht\mathbf h_t; ot\mathbf o_t holds this step's output scores. Because the notebook has a fixed length, ht\mathbf h_t can only be a finite-dimensional summary learned for the objective; it cannot store every detail of the raw history. That compression enables streaming and also loses long-range detail.

Inputs feed hidden states across time, and each hidden state produces an output.Open full-size image

Read horizontally to follow the hidden state through time, then vertically to connect an input to its output. Repeated boxes represent repeated use of shared parameters, not separately trained networks. A later output can depend on earlier inputs through this state path.

One Number Can Serve as a State: Streaming Smoothing​

Start with a state that is a single number; the gradient section below reuses it. Consider the scalar recurrence

ht=αht−1+(1−α)xt,0<α<1.h_t=\alpha h_{t-1}+(1-\alpha)x_t, \qquad 0<\alpha<1.

Each new value keeps a share α\alpha of the old state and gives the new value a share 1−α1-\alpha. This is an exponential moving average. Each update uses constant memory. The newest observation has weight 1−α1-\alpha, and an observation mm steps old has weight (1−α)αm(1-\alpha)\alpha^m.

Take α=1/2\alpha=1/2, h0=0h_0=0, and inputs [2,0,4][2,0,4]:

StepComputationState
10.5⋅0+0.5⋅20.5\cdot0+0.5\cdot2h1=1h_1=1
20.5⋅1+0.5⋅00.5\cdot1+0.5\cdot0h2=0.5h_2=0.5
30.5⋅0.5+0.5⋅40.5\cdot0.5+0.5\cdot4h3=2.25h_3=2.25

Expanded with the weight formula, h3=0.125⋅2+0.25⋅0+0.5⋅4=2.25h_3=0.125\cdot2+0.25\cdot0+0.5\cdot4=2.25: the earliest input, 2, keeps only one eighth of its weight.

The example shows both sides of recurrence: an RNN can update online without storing the whole sequence, but early information decays according to a fixed rule, and one old value cannot be reread on demand. A learned RNN is more flexible, yet remains constrained by its finite state and training signal. Attention changes this trade-off by retaining multiple represented positions and accessing them by content.

Sequence Objectives Come in Several Forms​

FormOutputExampleBoundary
many-to-onefinal or pooled statesequence classificationcan early evidence survive?
aligned many-to-manyevery steptagging, forecastingmasks and padding matter
autoregressivenext token/valuelanguage, time seriestrain and generation inputs differ
encoder–decoderconditional sequencetranslationone context can bottleneck history

Autoregressive modeling factorizes

p(x1:T)=∏t=1Tp(xt∣x<t).p(x_{1:T})=\prod_{t=1}^{T}p(x_t\mid x_{<t}).

That is, the probability of the whole sequence is the product, step by step, of the probability of each value given the values before it. Training often uses teacher forcing, which feeds the true prefix at every step. At inference the model instead sees its own earlier outputs, which may contain errors, so low training loss does not guarantee stable free generation. This gap between training and generation is called exposure bias; longer training does not remove it, and the remedy depends on the task.

Gradients Multiply Through Time, So They Vanish or Explode​

Training an RNN unrolls the loop through time and differentiates backward from the end; this is backpropagation through time (BPTT). The path from time tt to kk contains a Jacobian product:

∂ht∂hk=∏j=k+1tdiag⁡ ⁣(ϕ′(zj))Whh.\frac{\partial\mathbf{h}_t}{\partial\mathbf{h}_k} =\prod_{j=k+1}^{t} \operatorname{diag}\!\left(\phi'(\mathbf{z}_j)\right)W_{hh}.

In words, each step back multiplies the gradient once more by “activation slope × WhhW_{hh}.” The smoothing example is the simplest case: every factor is α=1/2\alpha=1/2, so the sensitivity to h0h_0 after three steps is α3=1/8\alpha^3=1/8, and after fifty steps it is (1/2)50≈9×10−16(1/2)^{50}\approx9\times10^{-16}; early inputs receive almost no training signal. An ordinary RNN multiplies by a similar factor at every step. Repeated contraction causes vanishing gradients; repeated expansion causes exploding gradients. Sequence length, weight spectra, activation saturation, and the trajectories in the data together decide which happens.

  • Gradient clipping limits large updates but cannot restore vanished gradients.
  • Truncated BPTT backpropagates only a fixed number of steps, reducing memory and computation while also cutting long-range credit assignment, that is, deciding which earlier input the final error should be attributed to.
  • Suitable initialization, normalization, and gates can improve training, but memory length remains finite.

For clarity, define Jj=diag⁡(ϕ′(zj))WhhJ_j=\operatorname{diag}(\phi'(z_j))W_{hh}. The ordered product is JtJt−1⋯Jk+1J_tJ_{t-1}\cdots J_{k+1}; matrix factors cannot generally be reversed. The same recurrent parameters appear at every step, so their gradient sums contributions from all uses, including indirect paths through later states. This is the chain-rule mechanism behind BPTT's gradient difficulties.

Code usually truncates with detach: the value still flows forward, but the gradient path is cut at that point. In the smoothing example, detaching the state after step two keeps its value 0.50.5 for the next forward step but removes the gradient path to the earlier chunk. Detaching is not resetting: a reset would replace 0.50.5 with the initial value. For padded sequences, masking only the loss is insufficient if a final state is consumed: skip padded state updates or select the last valid state.

LSTM Uses an Additive Path to Carry Information Forward Unchanged​

A long short-term memory network (LSTM) maintains a cell state ct\mathbf c_t alongside the hidden state ht\mathbf h_t. The forget gate ft\mathbf f_t scales the previous cell state, the input gate it\mathbf i_t scales the candidate update c~t\tilde{\mathbf c}_t, and the output gate ot\mathbf o_t controls how the cell state contributes to ht\mathbf h_t. Here σ\sigma is the sigmoid function, ⊙\odot means elementwise multiplication, and [ht−1,xt][\mathbf h_{t-1},\mathbf x_t] concatenates the two vectors. In these equations, ot\mathbf o_t is the output gate, a different quantity from the output scores in the opening RNN equation. A common form is:

ft=σ(Wf[ht−1,xt]+bf),it=σ(Wi[ht−1,xt]+bi),\mathbf{f}_t=\sigma(W_f[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_f), \quad \mathbf{i}_t=\sigma(W_i[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_i), c~t=tanh⁡(Wc[ht−1,xt]+bc),ct=ft⊙ct−1+it⊙c~t,\tilde{\mathbf{c}}_t=\tanh(W_c[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_c), \quad \mathbf{c}_t=\mathbf{f}_t\odot\mathbf{c}_{t-1}+\mathbf{i}_t\odot\tilde{\mathbf{c}}_t, ot=σ(Wo[ht−1,xt]+bo),ht=ot⊙tanh⁡(ct).\mathbf{o}_t=\sigma(W_o[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_o), \quad \mathbf{h}_t=\mathbf{o}_t\odot\tanh(\mathbf{c}_t).

The key is that the ct\mathbf c_t line is an addition. With the forget gate near 1 and the input gate near 0, ct≈ct−1\mathbf c_t\approx\mathbf c_{t-1}: the cell state is copied to the next step unchanged, and the derivative along this path is also close to 1, instead of being shrunk by a multiplication with WhhW_{hh} at every step as in an ordinary RNN. Information and gradients can therefore travel across many more steps. Implementations vary in biases, projections, peepholes, and gate order.

In Distill’s Visualizing memorization in RNNs, select characters in the autocomplete examples and inspect their connections to earlier input. The displayed connections come from gradient-based diagnostics on those examples. Comparing GRU and LSTM shows that similar aggregate scores can hide different ways of using context.

State and Direction Must Match the Task​

  • Causal RNNs use only current and past inputs, so they support streaming prediction.
  • Bidirectional RNNs also use future context, so they do not suit online decisions where that context is unavailable.
  • Reset hidden state between independent entities unless carrying it across the boundary has explicit meaning.

State-Based Modeling Continued After RNNs​

State-based modeling kept developing after classical RNNs. Structured state-space models (SSMs) start from continuous- or discrete-time linear state equations and use structured parameters and parallel algorithms to combine long-sequence training with recurrent inference. S4 demonstrated the long-sequence capabilities of structured SSMs; Mamba made parts of the update input-dependent and introduced a hardware-aware scan; xLSTM revisited LSTM memory and gates. A classical RNN's state transition is nonlinear, while an SSM uses structured state equations; these names refer to distinct architectures.

RNNs, SSMs, and Full Self-Attention Pay Different Costs​

DimensionClassical RNN / some SSMsFull self-attention
training pathRNN serial; structured SSMs may use parallel scans or convolutionspositions parallel within a layer
inference statefixed or controlled sizeKV cache usually grows with context
history accesscompressed statedirect access to represented positions
long-range coststate dynamics and selectionscores usually quadratic in length
useful counterexamplestreaming sensors, low-memory settings, long sequenceslarge-scale parallel training and flexible content retrieval

The choice depends on latency, memory, sequence length, training scale, kernels, and task structure. Paper benchmarks remain tied to data, scale, kernels, and training budgets, and need replication on the target task.

Hybrid Models Now Ship With a KV Cache in Only a Few Layers​

By October 2026, models that mix state-based layers with a few full-attention layers ship as open weights. The Qwen3-Next-80B-A3B model card gives its layer layout as 12 * (3 * (Gated DeltaNet -> MoE) -> 1 * (Gated Attention -> MoE)): of 48 layers, three in every four are Gated DeltaNet and one is full attention. Its native context is 262,144 tokens, and full_attention_interval: 4 in its configuration file records that spacing.

Gated DeltaNet is a form of linear attention. Like the RNNs on this page, it compresses what it has read into a fixed-size state that does not grow with length. Only the 12 full-attention layers store a key and a value for every token read. In this model, then, the KV cache that grows with context comes from a quarter of the layers. How much memory the KV cache takes, and how to estimate it, is worked out in Attention Variants and KV-Cache Compression.

Explore connectionsOpen network