Lectures

Tuesday, Thursday: 2:30-3:50, Duncan Hall 1046

Class Schedule for Spring 2008
Week Topic

HW
Due Thursday
1/7

Class organization.
What is information theory?
1/14

Properties of Entropy, mutual information and Kullback Leibler distance

 2.2, 2.4, 2.9, 2.12, 2.14, NP1, NP2
1/21 Asymptotic Equipartition Property (AEP). Theory of types.
Source Coding Theorem. Uniquely decodeable codes. Kraft inequality. Huffman coding.		 3.12, NP3, NP4, NP5, NP6
1/28

Universal coding. Generating random bits. Channel capacity.

 4.2, 4.6, 4.8, 5.24, 5.28
2/4 Noisy channel coding theorem.
Capacity of discrete channels; feedback.
 5.12, 5.14, 5.20, 5.29, 13.5
2/11

Source-channel separation theorem.
Capacity of continuous channels; parallel channels.
Capacity in the frequency domain.

7.3, 7.5, 7.7, 7.8, 7.11
2/18

Feedback and capacity.
Network information theory: multi-access and broadcast channels.

  
2/25

Network information theory.
Slepian-Wolf Theorem.

 Quiz I due
3/3 Midterm Recess
  
3/10

Rate distortion theory.
Fundamental limits of analog communication.
Fundamental limts of digital communication.
Measure matching theorem.

 9.1, 9.6, 15.1, NP7, NP11
3/17

"Exact" solution to the rate distortion function; Blahut algorithm.
Information Theory and Estimation:
Minimum mean-squared error estimators and mutual information

 10.1, 10.4, 10.6, NP12
3/24

Minimum mean-squared error estimators and mutual information

 10.8, NP8, NP9, NP10
3/31

Basics of estimation theory.
Model-based estimation: maximum likelihood, Cramér-Rao bound, Fisher information.

  
4/7

Linear signal models: Minimum Description Length (MDL) and A Information Criterion (AIC).
Spectral estimation and the maximum entropy principle.

  
4/14

Entropy estimation.
Basics of detection theory.
Large deviation theory, Sanov's theorem, Stein-Chernoff Lemma.

 Quiz II Due
4/21 Type-based detectors.  

Extra Problems: