Neural and generative models

Architecture determines how information is represented and combined. A training objective says what behavior is rewarded; a decoding procedure says how outputs are obtained. These are connected choices, not synonyms.

Follow MLPs and activations into convolutional and recurrent alternatives. For text, read tokenization, attention, Transformers and encoder/decoder families before generation. VAE and diffusion explain other generative routes; multimodal models and MoE extend representations and computation.

Reading Order

Step	Article	What it explains
1	Multilayer Perceptron	Understand MLP capacity, backpropagation, optimization failures, and inductive bias through tensor shapes and a worked XOR construction.
2	Activations and Gated Feed-Forward Networks	Distinguish ReLU, GELU, SiLU/Swish, and GLU variants, then select nonlinearities with controlled modern-network experiments.
3	Convolutional Neural Networks	Convolutional inductive bias, channels, padding, stride, pooling, and the distinction between translation equivariance and invariance.
4	Recurrent Neural Networks	Understand recurrent state compression, backpropagation through time, LSTM gating, and the boundary with modern state-space sequence models.
5	Tokenization and Text Representations Inside a Model	Follow text through token IDs, embeddings, and contextual states, distinguishing tokenization, position, padding, and retrieval vectors.
6	Attention Mechanism	Understand content addressing, Q/K/V, masks, multi-head variants, efficient implementations, and interpretation limits through a numerical example.
7	Transformers: Original Architecture and Modern Blocks	Distinguish the 2017 encoder–decoder from common Pre-Norm, RMSNorm, RoPE, GQA, and SwiGLU blocks, including training and inference costs.
8	Encoder, Decoder, and Encoder–Decoder: How Information Flows	Distinguish model families by visible context, training objectives, and output heads, from BERT classifiers to generators and sequence converters.
9	Autoregressive Generation and Decoding	Follow next-token distributions to complete answers, including temperature, top-k, top-p, search, stopping, and structured output.
10	Variational Autoencoders	Latent-variable generative models trained with amortized variational inference, the ELBO, and reparameterized gradients.
11	Diffusion Models	Forward noising, learned reverse transitions, noise-prediction training, iterative sampling, conditioning, and computational trade-offs.
12	Multimodal Models: Alignment, Fusion, and Visual Questions	Use image–text retrieval and visual question answering to distinguish objectives, patches, modality connectors, information loss, and evaluation.
13	Mixture of Experts: Sparse Computation, Routing, and Deployment Cost	Follow a token through MoE routing to distinguish total parameters, active parameters, memory, speed, load balancing, and communication.

Use What You Read

Track tensor shapes and information access through one example. For a new architecture, ask what can attend to what, which parameters are shared, and what the forward pass actually returns. Training and inference costs have their own companion topics.

Return to the AI reading paths to choose a neighboring topic.

My reading · Opened notes are remembered in this browser.