COMP / ELEC / STAT 502
Spring 2018


Schedule

The schedule of class topics, home works, tests, due dates, and handouts will come on-line in this table in a timely manner. The actual materials (Lecture Slides, other handouts, home work assignments) will be posted in Canvas for download. Should you have trouble with this web page please notify me and copy the TAs.

The The Lecture Notes and Handouts constitute the mandatory reading, and consist of two main ingredients:

Key to notation:
Lx-"title" = set #x of my lecture slides titled "title". EM-"other" = "other" electronic material that I may post.
CFi/k = Chapter k of Colin Fyfe's Volume i. Note that since I am using chapters from two different volumes two printed chapters may have the same number. Mark the volume # on your hardcopies.

Additional, optional reading is also listed here to quench your thirst for more knowledge than contained in the Lecture Notes. The references to those are given in the Course materials.
>>18

No.

Date

Topic

Lecture Notes & Handouts
mandatory reading

Homework

Upload an electronic copy to Canvas
before 2:15pm on the due date,
unless otherwise indicated.

Optional Reading

1

Jan 9

Orientation, Introduction

CF2/1
L1-Introduction-502.pdf

(not a real) HW01 due 1/11

Haykin Ch1
or Ham & Kostanic Ch 1, Ch2.1-3

2

Jan 11

Train a single PE
Associative memory
Review of basics, on your own. Exercises in HW02.
Review of HW02 solutions in class, Jan 18.

CF2/1, CF2/2
Train-a-single-PE.pdf
L2-ReviewOfBasics.pdf
CF1/2
L4-AssociativeMemory.pdf

HW02 due 1/18
most of the problems will be graded
only for turning in and attempting them.
See details in the posted HW02.

Haykin Ch10
or Cover & Thomas Ch1 and Ch9

Jan 16

CLASS CANCELED, Rice closed for winter storm

3

Jan 18

Associative memory (Josh Taylor)
Review of basics, cont'd
CF1/2
L4-AssociativeMemory.pdf
Haykin Ch2.11
or Ham & Kostanic Ch 3.2

4

Jan 23

Associative memory, wrap-up.
Discussion of HW02 solutions (Zhengjia Wang)
You will be assumed to have studied
L2-ReviewOfBasics, to have attempted
solutions, and to ask questions.
CF1/2
L4-AssociativeMemory.pdf
HW03 due 2/1
MAMerrcorr-summary.pdf
Haykin Ch2.11
or Ham & Kostanic Ch 3.2

5

Jan 25 Simple Perceptron L5-Perceptron.pdf
CF1/3

Ham & Kostanic Ch 2.5-2.6; or Haykin Ch3

6

Jan 30 Simple Perceptron
Batch vs on-line learning
L5-Perceptron.pdf
CF1/3

Ham & Kostanic Ch 2.5-2.6; or Haykin Ch3

7

Feb 1 Multilayer Perceptron (Back Propagation) L6-MultiLayerPerceptron.pdf
CF1/4

HW04 Part I due 2/8 Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4

8

Feb 6 Multilayer Perceptron cont'd
Issues With Backprop: Scaling, Momentum
L7-IssuesWithMLPs.pdf
CF1/4

Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8

Feb 8 Spring recess, NO CLASS

9

Feb 13 Issues With Backprop: Other practical items L7-IssuesWithMLPs.pdf
L7.1-BP-practical-items.pdf
L7.2-Vector-matrix-BP.pdf CF1/4

HW4 Part I and Part II together; due Feb 20 Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8

10

Feb 15 Network Pruning
(Start) # training samples; Vapnik-Chervonenkis dimension
L8-VC-Dimension.pdf
L9-NetworkPruning.pdf
Review PCA for lecture on Feb 20
Material in Canvas, L2, and in Supplements/PCA-Ham&Kostanic-pp396-399.pdf
or any other standard text
Haykin Ch4.5-4.8, Haykin pp 94 - 102, Bishop

11

Feb 20 VC-dimension; Start PCA nets: Oja's Symmetric Subspace Algorithm L8-VC-Dimension.pdf
L10-UnsupervisedLearning-PCAnets.h.pdf
HW05 due 3/1 Ham & Kostanic Ch 9.1-9.3; Haykin CH8.1-5

12

Feb 22 PCA nets: Oja's Symmetric Subspace Algorithm, GHA L10-UnsupervisedLearning-PCAnets.h.pdf
CF1/5.6-8
Ham and Kostanic Ch 4.1-4.2; or Haykin CH8.1-5, CH9.1-6

13

Feb 27 Wrap up GHA;
Competitive learning: SOM

L11.1-Self_Organizing_Maps-EM502.part1.pdf CF1/5.6-8

HW06 due 3/8

Haykin CH8.1-5, CH9.1-6

14

Mar 1 QUIZ1

15

Mar 6 Competitive learning: SOM, cont'd, case studies; LVQ L11-CompetitiveUnsupervisedLearning.pdf
L11.1-Self_Organizing_Maps-EM502.pdf CF1/5.6-8

Haykin CH8.1-5, CH9.1-6

16

Mar 8 Competitive Learning: SOM case studies';
LVQ (LVQ1, LVQ2, LVQ3) paradigms
(Read ART in L11 on your own, 2 slides.);

L11-CompetitiveUnsupervisedLearning.pdf updated
L11.1-Self_Organizing_Maps-EM502.pdf updated
L11.2-LVQ_illustration.pdf
L11.3-Self_organized_hand-RitterSchulten.pdf
HW07 due 3/20 (last HW; a little heavy, start early)

Project proposals due March 20, upload to Canvas.
Review requirements under Course project
Project proposal reviews / feedback on March 20 (in office hour) and March 22 (in class period); in my office (or room TBA). Sign up for 15-minute time slots that will be posted on the door. Group members should come together.
Suggestions for project themes coming soon
Feel free to consult with me!
Ham & Kostanic Ch 9.1-9.3; Haykin CH8.1-5

Mar 13

NO CLASS, spring break

Mar 15

NO CLASS, spring break

17

Mar 20 Recurrent nets: Hopfield network
Better, faster learning: Simulated Annealing
L12-RecurrentNetworks-HopfieldNet.h.pdf
L13-RecurrentNetworks-SAandBM.h.Part1.pdf
CF1/7.1-7.2, CF2/5.4-5.6
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11

18

Mar 22

Project proposal reviews in my office (or room TBA). Sign up for 15-min windows posted on the door. Those who do their proposal review in office hour on Tue will not need to come today.
Review requirements under Course project
Exam 1 handed out
Exam 1 guidelines
Exam 1 due no later than 12:00pm (noon) on Sunday, 3/25 (as posted in Canvas);
Material for midterm up to March 6 (including HW exercises up to HW06).

19

Mar 27 Better, faster learning: Network Growing with Cascade Correlation
Cross-entropy as objective function
L14.1-NetworkGrowing_CC-illustration.pdf
L14-NetworkGrowing.h.pdf
L15-InfoTheoreticalObjectiveFunctions.h.pdf
Read cross-entropy part of L15 Ham& Kostanic A.5.5

20

Mar 29 Better, faster learning: Conjugate Gradients
SOM-hybrid supervised network; Conscience SOM;
case studies
L11.11-SOM-hybrid-classification.h.pdf
L11.12-Conscience-SOM.h.pdf
L16-ConjugateGradients.h.pdf
L16-ConjugateGradients-annotated.pdf
Ham& Kostanic A.5.5

21

Apr 3 Blind Source Separation
Neural factorial codes
L18-ANN-NLPCA-and-ICA.h.pdf
L18.1-FactorialCodes.pdf
CF 2/6

22

Apr 5 Better, faster learning: Information theoretical objective functions L15-InfoTheoreticalObjectiveFunctions.h.pdf

23

Apr 10 Recurrent nets: Boltzman Machine L13-RecurrentNetworks-SAandBM.pdf
CF1/7.1-7.2, CF2/5.4-5.6
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11

24

Apr 12 QUIZ2 Material for QUIZ2: from QUIZ1 to April 5;
QUIZ2 will be similar in nature to QUIZ1

25

Apr 17 NO CLASS, allowance for project presentations

26

Apr 19 Project presentations instructions to come

Return to COMP / ELEC 502 / STAT 502 home page