Activations and Gated Feed-Forward Networks
Distinguish ReLU, GELU, SiLU/Swish, and GLU variants, then select nonlinearities with controlled modern-network experiments.
Distinguish ReLU, GELU, SiLU/Swish, and GLU variants, then select nonlinearities with controlled modern-network experiments.
Distinguish model families by visible context, training objectives, and output heads, from BERT classifiers to generators and sequence converters.
A section-by-section reading of Vaswani et al. (2017) covering the parallelism and long-range-dependency problem, scaled dot-product and multi-head attention, positional encoding, the training setup, and the results in Tables 1–4.
Distinguish the 2017 encoder–decoder from common Pre-Norm, RMSNorm, RoPE, GQA, and SwiGLU blocks, including training and inference costs.