Recurrent Neural Networks

An RNN reuses parameters while carrying state through time:

$$\mathbf{h}_t=\phi(W_{xh}\mathbf{x}_t+W_{hh}\mathbf{h}_{t-1}+\mathbf{b}_h), \qquad \mathbf{o}_t=W_{ho}\mathbf{h}_t+\mathbf{b}_o.$$

The finite-dimensional state is a learned summary for the objective, not a lossless archive. That compression enables streaming and also loses long-range detail.

Sequence Objectives

Form	Output	Example	Boundary
many-to-one	final or pooled state	sequence classification	can early evidence survive?
aligned many-to-many	every step	tagging, forecasting	masks and padding matter
autoregressive	next token/value	language, time series	train and generation inputs differ
encoder–decoder	conditional sequence	translation	one context can bottleneck history

Autoregressive modeling factorizes

$$p(x_{1:T})=\prod_{t=1}^{T}p(x_t\mid x_{<t}).$$

Teacher forcing supplies the true prefix in training; free generation consumes prior model errors. Low teacher-forced loss therefore does not guarantee stable rollout.

Why Backpropagation Through Time Fails

The path from time $t$ to $k$ contains a Jacobian product:

$$\frac{\partial\mathbf{h}_t}{\partial\mathbf{h}_k} =\prod_{j=k+1}^{t}\operatorname{diag}(\phi'(\mathbf{z}_j))W_{hh}.$$

Repeated contraction causes vanishing gradients; expansion causes exploding gradients. Length, weight spectra, activation saturation, and encountered trajectories all matter.

Gradient clipping limits large updates but cannot restore vanished gradients. Truncated BPTT saves memory while truncating credit assignment. Initialization, normalization, and gates help without creating unlimited memory.

LSTM Gating

A common LSTM form is

$$\mathbf{f}_t=\sigma(W_f[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_f), \quad \mathbf{i}_t=\sigma(W_i[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_i),$$ $$\tilde{\mathbf{c}}_t=\tanh(W_c[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_c), \quad \mathbf{c}_t=\mathbf{f}_t\odot\mathbf{c}_{t-1}+\mathbf{i}_t\odot\tilde{\mathbf{c}}_t,$$ $$\mathbf{o}_t=\sigma(W_o[\mathbf{h}_{t-1},\mathbf{x}_t]+\mathbf{b}_o), \quad \mathbf{h}_t=\mathbf{o}_t\odot\tanh(\mathbf{c}_t).$$

The additive cell path improves information and gradient flow. Gates are learned soft controls, not automatically interpretable memory switches; implementations vary in biases, projections, peepholes, and gate order.

Worked Example: Streaming Smoothing

For scalar state

$$h_t=\alpha h_{t-1}+(1-\alpha)x_t, \qquad 0<\alpha<1,$$

an observation $m$ steps old has weight $(1-\alpha)\alpha^m$. The update uses constant memory, but cannot retrieve one old value on demand. Learned recurrence is more flexible and retains the same compression trade-off; attention changes it by keeping and addressing multiple represented positions.

State and Data Boundaries

Causal RNNs support streaming; bidirectional RNNs use future context and cannot be deployed where the future is unavailable. Padding must be masked. State must reset between independent entities unless cross-boundary carry has explicit meaning. Randomly splitting overlapping windows can leak adjacent sequence content across train and test—an evaluation flaw, not architecture quality.

Modern State-Based Sequence Models

Classical RNNs are not the end of state-based modeling. Structured state-space models (SSMs) use structured state equations and parallel algorithms to combine long-sequence training with recurrent inference. S4 demonstrated structured SSMs; Mamba made parts of the update input-dependent and introduced a hardware-aware scan; xLSTM revisited recurrent memory and gates.

These are not one architecture. Mamba's sequence-scaling claim describes an algorithmic path, not guaranteed wall-clock superiority at every length, batch, or device. xLSTM does not establish that LSTMs universally beat Transformers. Paper benchmarks remain tied to data, scale, kernels, and budgets.

Dimension	Classical RNN / some SSMs	Full self-attention
training path	RNN serial; structured SSMs may parallelize	positions parallel within a layer
inference state	fixed or controlled size	KV cache usually grows with context
history access	compressed state	direct access to represented positions
long-range cost	state dynamics and selection	scores usually quadratic in length
useful counterexample	streaming, low-memory sequence	flexible retrieval and parallel training

As of 2026-08-11, S4, Mamba, and xLSTM are important competing designs, not a new permanent default.

Minimal Experiment

Overfit a short sequence; test that changing masked padding leaves output unchanged; deliberately mix state boundaries and ensure tests fail; report teacher-forced and free-running results; split by entity and time; compare against stateless, seasonal, or moving-average baselines; and record gradient norms, truncation length, hidden width, kernels, and failed seeds.

This note favors supervised discrete sequences. Continuous-time systems, all SSM variants, reinforcement learning, and online feedback require separate treatment.