Deep Learning: From Function Approximation to Sequence Models
Deep learning learns representations with layered parameterized functions. Architecture is only one part of the system: the data-generating process, objective, optimization, split design, and deployment distribution determine whether a result is credible.
Learning Outcomes
A reader should be able to explain why nonlinear layers do not collapse into one linear map; what inductive biases MLPs, recurrence, and attention add; what a Transformer block contains besides attention; and why a falling training loss does not rule out leakage, shortcut learning, or distribution shift.
Architecture Path
| Order | Note | Question | Minimal exercise |
|---|---|---|---|
| 1 | Linear and Softmax regression | how parameters, losses, and gradients connect | match output heads to losses |
| 2 | Multilayer Perceptron | what nonlinear hidden representations add | express XOR with two ReLUs |
| 3 | Activations and Gated FFNs | where ReLU, GELU, SiLU, and SwiGLU differ | compare them under equal budgets |
| 4 | Recurrent Neural Networks | how state compresses history, and what modern SSMs change | unroll an exponential moving average |
| 5 | Attention | how content-addressed retrieval works | compute a two-key Softmax mixture |
| 6 | Transformers | how attention, residuals, norms, FFNs, and positions form a network | derive Q/K/V and score shapes |
CNNs form a separate path through locality and translation sharing. The Generative Models map covers latent-variable and diffusion objectives.
MLPs remain inside Transformer blocks. ReLU is not universally obsolete, although modern Transformers often use GELU, SiLU, or gated FFNs. RNNs remain useful for streaming and bounded state, while S4, Mamba, and xLSTM revisit state-based sequence computation. Attention predates the Transformer and can be combined with recurrence or convolution. The order above expresses conceptual dependency, not a permanent ranking.
Evidence Path
Read every architecture experiment alongside:
- Data Splits and Leakage, which fixes prediction time, entities, and available information before fitting;
- Evaluation Under Distribution Shift, which aligns splits, metrics, thresholds, and uncertainty with use;
- Evidence and Bias, which separates paper results, reproductions, inference, and preference.
A random split that leaks users or future data, or a test set repeatedly used for tuning, invalidates an otherwise polished architecture comparison.
question and prediction time
→ unit, provenance, license, and split keys
→ simple baseline
→ one controlled architectural change
→ fixed budget and multiple seeds
→ central result, variation, failures, and resources
→ temporal, group, or external validation
This path is biased toward supervised gradient training, sequences, and English-language textbooks. It does not constitute a complete treatment of causality, reinforcement learning, graphs, governance, energy, or social impact.