COMP / ELEC / STAT 502
Spring 2019


Schedule

The schedule of class topics, home works, tests, due dates, and handouts will come on-line in this table in a timely manner. The actual materials (Lecture Slides, other handouts, home work assignments) will be posted in Canvas for download. Should you have trouble with this web page please notify me and copy the TAs.

The The Lecture Notes and Handouts constitute the mandatory reading, and consist of two main ingredients:

Key to notation:
Lx-"title" = set #x of my lecture slides titled "title". EM-"other" = "other" electronic material that I may post.
CFi/k = Chapter k of Colin Fyfe's Volume i. Note that since I am using chapters from two different volumes two printed chapters may have the same number. Mark the volume # on your hardcopies.

Additional, optional reading is also listed here to quench your thirst for more knowledge than contained in the Lecture Notes. The references to those are given in the Course materials.
>>

No.

Date

Topic

Lecture Notes & Handouts
mandatory reading

Homework

Upload an electronic copy to Canvas
before 2:15pm on the due date,
unless otherwise indicated.

Optional Reading

1

Jan 8

CLASS canceled. Instructor sick.
Introduction to Neural Networks - postponed (hopefully to Thu).
For Orientation, please do Part II of HW01 (familiarize yourself thoroughly with posted course logistics and requirements), come with questions for any clarification next time.

(not a real) HW01 due 1/10

2

Jan 10


Introduction to Neural Networks - (instructor hopefully back).
Train a single PE (time permitting).
Q & A about course logistics and requirements.

On your own: Review of basics (L2-ReviewOfBasics.pdf). This will help you, along with exercises in HW02, gauge your preparedness and to catch up on background, if needed.


L1-Introduction-502.pdf


CF2/1
L2-ReviewOfBasics.pdf

HW02 due 1/17
Most HW02 problems will be graded
only for turning in and earnestly attempting them.
See details in the posted HW02.
Now do Parts III and IV of HW01 (nothing to turn in, but I will assume the knowledge).

Haykin Ch1
or Ham & Kostanic Ch 1, Ch2.1-3

3

Jan 15

Train a single PE (finish)
Associative memory
Continue to review the basics, on your own.

Train-a-single-PE.pdf
L4-AssociativeMemory.pdf
CF2/1, CF2/2, CF1/2
L2-ReviewOfBasics.pdf

Haykin Ch10
or Cover & Thomas Ch1 and Ch9

4

Jan 17

Associative memory, cont'd
Review of basics, cont'd
CF1/2
L4-AssociativeMemory.pdf
Haykin Ch2.11
or Ham & Kostanic Ch 3.2

5

Jan 22

Associative memory, wrap-up
Simple Perceptron
CF1/2
L4-AssociativeMemory.pdf
L5-Perceptron.pdf
CF1/3

HW03 due 1/29 MAMerrcorr-summary.pdf
Haykin Ch2.11
or Ham & Kostanic Ch 3.2
Ham & Kostanic Ch 2.5-2.6; or Haykin Ch3

6

Jan 24 Simple Perceptron L5-Perceptron.pdf
CF1/3

Ham & Kostanic Ch 2.5-2.6; or Haykin Ch3

7

Jan 29 Simple perceptron, batch learning, wrap-up
Multilayer Perceptron (Back Propagation)
L6-MultiLayerPerceptron.pdf
CF1/4

HW04 Part I due 2/6 (yes, Wed) Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4

8

Jan 31 Multilayer Perceptron cont'd
Issues With Backprop: Scaling, convergence, performance monitoring, overtraining
L7-IssuesWithMLPs.pdf
CF1/4

Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8

9

Feb 5 Issues With Backprop: More scaling, momentum L7-IssuesWithMLPs.pdf
L7.2-Vector-matrix-BP.pdf CF1/4

HW4 Part I and Part II together (will post after 5pm, Feb 6); due Feb 14 11pm Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8

Feb 7 Spring recess, NO CLASS Review PCA for lecture on Feb 14, Feb 19
Material in Canvas, L2, and in Supplements/PCA-Ham&Kostanic-pp396-399.pdf
or any other standard text

10

Feb 12 Issues With Backprop: Other practical items; wrap-up L7-IssuesWithMLPs.pdf
L7.1-BP-practical-items.pdf
L7.2-Vector-matrix-BP.pdf CF1/4

Review PCA
Material in Canvas, L2, and in Supplements/PCA-Ham&Kostanic-pp396-399.pdf
or any other standard text
Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8

11

Feb 14 Network Pruning L9-NetworkPruning.pdf HW05 due 2/21 Haykin Ch4.5-4.8, Haykin pp 94 - 102, Bishop
Ham & Kostanic Ch 9.1-9.3; Haykin CH8.1-5

12

Feb 19 PCA nets: Oja's Symmetric Subspace Algorithm, GHA L10-UnsupervisedLearning-PCAnets.h.pdf CF2/3.1-3.6 Ham and Kostanic Ch 9.1-9.3; or Haykin CH8.1-5

13

Feb 21 Oja's Symmetric Subspace Algorithm, GHA, cont'd
Competitive learning: SCL, SOM
L10-UnsupervisedLearning-PCAnets.h.pdf
L11.1-Self_Organizing_Maps-EM502.pdf
CF1/5.6-8

HW06 due 2/28

Ham & Kostanic Ch 4.1 - 4.2, Haykin CH8.1-5, CH9.1-6

14

Feb 26 QUIZ1
SOM cont'd
L11.1-Self_Organizing_Maps-EM502.pdf
L11-CompetitiveUnsupervisedLearning.h
CF1/5.6-8
Ham & Kostanic Ch 4.1 - 4.2, CH9.1-6

15

Feb 28 Competitive learning: SOM L11.1-Self_Organizing_Maps-EM502.pdf
L11-CompetitiveUnsupervisedLearning.h
CF1/5.6-8

HW07 due Mar 19

Ham & Kostanic Ch 4.1 - 4.2, CH9.1-6

16

Mar 5 SOM wrap-up; LVQ (LVQ1, LVQ2, LVQ3) paradigms
Read ART in L11 on your own, 2 slides.
Project discussion. Please review Project requirements in preparation.

Exam 1 postponed. It will be posted by 5pm on March 21, due 11:00pm Sat, March 23. Material remains the same as indicated in this EXAM 1 guide.

Project proposals due 12noon, March 19.
Proposal requirements under Course project
Ham & Kostanic Ch 4.1 - 4.2, CH9.1-6

17

Mar 7 SOM clustering, SOM-hybrid classification and regression: case studies
Network Growing with Cascade Correlation
L14.1-NetworkGrowing_CC-illustration.pdf
L14-NetworkGrowing.h.pdf
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11

Mar 12

NO CLASS, spring break Project group declarations due by 5pm today. Email me the following items for each group member, one group member per line: Group #, NetID, last name, first name

Mar 14

NO CLASS, spring break

18

Mar 19

Project proposal reviews from 2:30pm to 5pm, in my office, or room TBA. Sign up sheet will be provided for 15-min slots per group. Please come with a hard copy of your proposal.
Proposal requirements under Course project

19

Mar 21

TBA (potentially Recurrent nets: Hopfield network
Better, faster learning: Simulated Annealing)
L12-RecurrentNetworks-HopfieldNet.h.pdf
L13-RecurrentNetworks-SAandBM.h.Part1.pdf
CF1/7.1-7.2, CF2/5.4-5.6
Exam 1 posted this evening. Material up to Feb 26 (including HW exercises up to HW06), as per this
Exam 1 guidelines Exam 1 due no later than 11:00pm on Sat, 3/23 (as posted in Canvas);
Ham& Kostanic 5.1 - 5.3, 5.4

20

Mar 26

TBA (potentially
Better, faster learning: Simulated Annealing; Conjugate Gradients)
Cross-entropy as objective function
L13-RecurrentNetworks-SAandBM.h.Part1.pdf
L16-ConjugateGradients.h.pdf
CF1/7.1-7.2, CF2/5.4-5.6
Ham& Kostanic 5.4, A.5

21

Mar 28

Better, faster learning: Conjugate Gradients;
Potentially: Information Theoretical Objective Functions - Cross-entropy as objective function
L16-ConjugateGradients.h.pdf
L15-InfoTheoreticalObjectiveFunctions.h.pdf
CF1/7.1-7.2, CF2/5.4-5.6
Ham& Kostanic 5.4, A.5

22

Apr 2 Vapnik-hervonenkis dimension (the capacity of learning machines / ANNs) L8-VC-Dimension.pdf

23

Apr 4 Blind Source Separation:
Neural ICA and Factorial codes
Recurrent nets: Boltzman Machine - optional read, L13
L18-ANN-NLPCA-and-ICA.h.pdf
L18.1-FactorialCodes.pdf
CF 2/6
L13-RecurrentNetworks-SAandBM.pdf
CF1/7.1-7.2, CF2/5.4-5.6
Ham & Kostanic Ch 10.8.3, Ch 5.4 - 5.6; Haykin Ch11

24

Apr 9 QUIZ2 (given by Josh Taylor)
short lecture by Josh Taylor, ANN outputs approximate Bayesian posterior (EM out of town)
Material for QUIZ2: from QUIZ1 to April 4;
QUIZ2 will be similar in nature to QUIZ1

25

Apr 11 NO CLASS, make-up time for Exam01

25

Apr 16 NO CLASS, pooling this class period with class on April 19, for project presentations

26

Apr 19 Project presentations, 2:30pm - 5pm
room TBA
Your presence is mandatory. We will peer grade the presentations. Instructions to come, along with grading sheets.

Return to COMP / ELEC 502 / STAT 502 home page