Description

Introductory course in information theory. This course explores the fundamental limits of information acquisition, transmission and reception initially described by Claude Elwood Shannon in 1948. Although primary emphasis will be on digital communication schemes, analog methods are also encompassed by the theory and will be discussed.

Prerequisites: Knowledge of stochastic processes and probability.

• Fundamental quantities of information theory
• Entropy
• Mutual information
• Kullback-Leibler divergence.
• Source Coding Theorem
• Entropy and compression
• Noisy Channel Coding Theorem
  • Channel capacity
• Network Information Theory
• Broadcast and Multiple-Access Channels
• Correlated source coding and the Slepian-Wolf Theorem
• Rate-Distortion Theory
  • Ultimate limits of compression, communication and signal processing
  • Measure matching
  • Blahut-Arimoto Algorithm
• Information Theory and Signal Processing
• Estimation: links between mutual information and optimal estimators
• Detection: optimal detector performance and Kullback-Leibler divergence

Course material

• Cover & Thomas. Elements of Information Theory, Wiley, Second Edition, 2006.