Skip to main content

Information Theory and Entropy

Information theory provides a language for uncertainty and the limits of communication. Build this branch in the following order:

  1. self-information and entropy;
  2. joint and conditional entropy;
  3. mutual information and independence;
  4. cross-entropy and Kullback–Leibler divergence;
  5. source coding and compression limits;
  6. noisy channels and channel capacity;
  7. connections to probabilistic modeling and machine-learning objectives.

This is currently a seed map. Use MIT OpenCourseWare 6.050J as the coherent external route rather than collecting isolated definitions.