COMP / ELEC / STAT 502
Fall 2024


Schedule

The schedule of class topics, home works, tests, due dates, and handouts will come on-line in this table in a timely manner. The actual materials (lecture slides, other handouts, home work assignments) will be posted in Canvas for download. Should you have trouble with this web page please notify me and copy the TA(s).

The Lecture Notes and Handouts constitute the mandatory reading, and consist of two main ingredients:

Key to notation:
Lx-"title" = set #x of my lecture slides titled "title". EM-"other" = "other" electronic material that I may post.
CFi/k = Chapter k of Colin Fyfe's Volume i. Note that since I am using chapters from two different volumes two printed chapters may have the same number. Mark the volume # on your hardcopies.

Additional, highly recommended reading from standard text books is also listed. The references to those are given in the Course materials.
>>

No.

Date

Topic

Lecture Notes & Handouts
mandatory reading

Homework

Upload an electronic copy to Canvas by 10:35am on the
due date unless otherwise indicated in Canvas.

Recommended Reading

1

Aug 27

CLASS CANCELLED

HW01 due 8/29. Part I accepted until the late deadline w/o late penalty.
Complete parts III and IV by 9/5. (Nothing to turn in but I expect you to know.)
This is a very light (not real) home work. You will only have to submit answers to my survey to assess backgrounds. Do this HW individually.

Form HW groups for HW2 and on. We will discuss HW groups in class.

2

Aug 29

Orientation, Introduction

On your own: Review of basics (L2-ReviewOfBasics.pdf). This will help you, along with exercises in HW02, to gauge your preparedness and to catch up on background if needed. Notions / formulae that I collected in Lecture 2 (L2) will come up in various contexts at various times in the course. Alternatively, you can look these up when the time comes :-)

L1-Introduction-502.pdf
L2-ReviewOfBasics.pdf
CF2/1



Form HW groups for HW2 and on. I will send an announcement with some detail.

Haykin Ch1
or Ham & Kostanic Ch 1, Ch2.1-3

3

Sep 3

Orientation, cont'd (logistics, HW groups, etc.)
Train a single PE

L2-ReviewOfBasics.pdf
L1.1-Training-a-single-PE.pdf
CF2/1, CF2/2

HW02 due 9/10
Part of HW02 problems may be fully graded,
others only for turning in and earnestly attempting them. See details in the posted HW02. The exact manner of grading will be determined according to grading help (TBD).

Haykin Ch10
or Cover & Thomas Ch1 and Ch9

4

Sep 5

Associative memory L4-AssociativeMemory.h.pdf and L4.x files
CF1/2
Haykin Ch2.11
or Ham & Kostanic Ch 3.2

5

Sep 10

Associative memory, cont'd updated L4-AssociativeMemory.h.pdf and L4.x files
CF1/2

Haykin Ch2.11
or Ham & Kostanic Ch 3.2

6

Sep 12

Associative memory, cont'd (wrap-up)
Simple perceptron
updated L4-AssociativeMemory.h.pdf and L4.x files
CF1/2
L5-Perceptrons.h.pdf and L.5x files
CF1/3

HW03 due 9/24


Haykin Ch2.11
or Ham & Kostanic Ch 3.2
Ham & Kostanic Ch 2.5-2.6; or Haykin Ch3

7

Sep 17 Preliminary to Multilayer Perceptron
Multilayer Perceptron with Back Propagation
L6.0-Train-a-2layer-network.h.pdf
L6-MultiLayerPerceptron.h.part1.pdf
CF1/4

Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4

8

Sep 19 Multilayer Perceptron, cont'd. Back Propagation rule.
Issues With Backprop
L6-MultiLayerPerceptron.h.pdf
L7-IssuesWithMLPs.h.part1.pdf
CF1/4
Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4

9

Sep 24 Issues With Backprop: Scaling, convergence, performance monitoring
Matrix BP; practical items (time permitting)
L7-IssuesWithMLPs.h.pdf
L7.1-Vector-matrix-BP.h.pdf CF1/4

HW04 Part I due Oct 1
Part I will have to be re-submitted together with and Part II (posted ~Oct 1, due ~Oct 8), with optional revision to Part I based on "free" feedback we give you. See details with the Assignment(s) in Canvas.
Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8

10

Sep 26 Issues With Backprop: More practical items;
HW04 samples; code structure overview
L7-IssuesWithMLPs.h.pdf
L7.1-Vector-matrix-BP.h.pdf
L7.0x-HW04_sample-plots_xxx.pdf samples
L7.3_Sample-parameter-table_502.pdf

>Recommended: experiment with Jesse Hellem's HW03 code now that you are done with your HW03. Nothing to turn in, just enjoy running it and exploring limits with the characters.ascii data set (all letters plus 10 digits). The code is in the Files/Code folder. If you use the code to produce reults or write-up of discussion, remember to give credit to Jesse for the code. Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8

11

Oct 1 HW04 Part II discussion, samples;
Code structure overview;
Cross-validation; VC Dimension (time permitting)
All L7-related files as listed in previous postings;
irisrun_sample.EM.m from Files/Code
L7.4-Cross-validation.h.pdf

If VC Dimension covered:
L8.0-Interrelated-issues&VC-Primer.h.pdf
L8-VC-Dimension.h.annnot.pdf
L8.2_VC-dim-example.pdf
HW04 Part II due Oct 8
Part I will have to be re-submitted together with Part II (as one HW04), with optional revision to Part I. See details with the Assignment(s) in Canvas.
Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8

12

Oct 3 Network Pruning: Optimal Brain Damage and Optimal Brain Surgeon
L9-NetworkPruning.pdf
L9.1-note_OBS-derivations.pdf
L9.2-RecursiveEvaluation-of-InverseHessian.pdf
Haykin Ch 4.15 pp 222 - 226

13

Oct 8 Finish Network Pruning;
Unsupervised learning. Competitive learning: SCL, SOM
L9 files (listed on Oct 3);
L11-CompetitiveUnsupervisedLearning.h.pdf
L11.1-Self_Organizing_Maps-EM502.pdf
CF1/5.6-8
HW05 due Oct 16 (BP application exercises including cross-validation) Ham and Kostanic Ch 4.1-4.2; or Haykin CH9.1-6

14

Oct 10 Competitive learning: SOM, cont'd; visualization, interpretation L11-CompetitiveUnsupervisedLearning.h.pdf
update of
L11.1-Self_Organizing_Maps-EM502.pdf
CF1/5.6-8
Ham and Kostanic Ch 4.1-4.2; or Haykin CH9.1-6

Oct 15 FALL RECESS, no classes HW06 due Oct 26 noon
(Posting a little early to allow spacing yourself.)

15

Oct 17 QUIZ1
Material up to Oct 10, excluding competitive learning.

Competitive learning: wrap up SOMs

Review PCA in preparation for PCA networks on Oct 24
PCA material in Canvas, in Supplements/PCA-Ham&Kostanic-pp396-399.pdf or in any other standard text

16

Oct 22 Competitive Learning
SOM visualization, case studies, wrap-up.
Learning Vector Quantization (LVQ)
L11-CompetitiveUnsupervisedLearning.h.pdf
L11.2-Self-organized-hand-RitterSchulten.pdf
L11.3-CompetitiveSupervisedLearning.LVQ.h.pdf
L11.3-LVQ_illustration.pdf
CF1/5.6-8
Ham and Kostanic Ch 4.1-4.2, 4.3, Haykin CH9.1-6

17

Oct 24 Unsupervised learning: Oja's PCA net, GHA net L10-whiteboard.PCAillustration.pdf
L10-UnsupervisedLearning-PCAnets.pdf
CF2/3
Review of PCA (Principle Component Analysis), from Canvas, Files/Supplements/PCA-Ham&Kostanic-pp396-399.pdf
or from any other standard text
HW07 due March 19. See further instructions with the Assignment. Ham & Kostanic Ch 9.1-9.3.4; Haykin CH8.1-5

18

Oct 29 HW05, HW06 discussion, practical items Reminder of what we discussed, and general feedback is in Files/General Feedback, HW05-comments.2024F, HW05.discussion.iris-experiments.pdf, HW05.PP&discussion.pdf, HW06.discussion.pdf, HW06.SOMIllustratioins.pdf TBA

19

Oct 31 Discussion of class project, including potential topics. See Files/Supplements/NML502_Project-topics.2024F.pdf for a reminder of suggested topics we discussed, and for a recap of major points.

SOM case studies;
Network growing with Cascade Correlation
update of L11.1-Self_Organizing_Maps-EM502.pdf
L14-NetworkGrowing.h.pdf

Form project groups. You may keep your HW group or re-group.
Email me your group information, similarly as you did for HW groups, by 8pm, Nov 1.
If you are re-grouping from your HW group and have difficulty finalizing it, you may send me your Project group information by noon, Sunday Nov 3 THE LATEST.
I must have your Project groups by Sunday so that I can set up a "Project Proposal" assignment in Canvas before the proposal are due on Monday.
Project proposals will be due 10pm, Nov 4.
Project proposal reviews will take place Nov 5 - Nov 7 (with Nov 8 fallback). Groups can start working on their project as soon as I approved their proposal.
Schedule a 30-min meeting with me sometime during these days. Schedule well ahead of time.
TBA

Nov 1
Exam 1 posts tonight in Canvas. Exam 1 information, see also in Canvas instructions. Approximate schedule is as at right. Material for midterm: up to (including) October 24 (GHA), and HW07 exercise.
You will have a 36-hour allowance that you can spend in multiple sittings between the opening of Exam 1 and the ultimate deadline, 11:59pm on Nov 3. Exam 1 is due when your 36 hours have been spent, but no later than 11:59pm on Nov 3, whichever comes first. No extension, no exceptions.

Nov 5 NO CLASS
Cancelled by Rice, for Election Day
You may, however, schedule class project proposal review with me on this day. TBA

20

Nov 7 Discussion, HW06 PP, and general issues from HW06, HW07
Information theoretical objective functions:
Start Cross-entropy for classification in BP;
L15-InfoTheoreticalObjectiveFunctions.Cross-entropy.h.pdf Slides 1 - 9 were covered.
CH 5.1 from CF2/5.1-5.3
Ham& Kostanic A.5.5
Haykin Ch10.1 - 10.5 (refresher)

21

Nov 12 Information theoretical objective functions: Finish Cross-entropy for classification in BP;
Discussion, HW07 and PP;
TBA
L15-InfoTheoreticalObjectiveFunctions.Cross-entropy.h.pdf
CH 5.1 from CF2/5.1-5.3
Ham& Kostanic A.5.5
Haykin Ch10.1 - 10.5 (refresher)

22

Nov 14 Recurrent networks: Hopfield Net L12-RecurrentNetworks-HopfieldNet.h.pdf
CF1/7.1-7.2, CF2/5.4-5.6
From now on, topics listed may or may not be covered on the date indicated, depending on how fast we can go and what else we may have to discuss with higher priority.
If a topic does not get covered on the indicated date expect it to come up soon after, and apply the posted lecture notes as appropriate.
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11

23

Nov 19 Better, faster learning:
Conjugate Gradients
Simulated Annealing as time permits
Alternatively: recommended slides (on your own) for Simulated Annealing
Supporting lecture slides
L13-SimulatedAnnealing.h.pdf
L16-ConjugateGradients.h.pdf
CF1/7.1-7.2, CF2/5.4-5.6
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11

24

Nov 21 Exam01 discussion
Project presentation logistics;
TBA lecture topic
Follow instruction that I will send in Announcement, for presentation preparation. Instructions will include downloading / reading details for presentation logistics.

25

Nov 26 QUIZ2, on-line. (No physical presence needed in the class room.) Google form URL releaased at 10:50am, nominal start at 10:52am, form closed 45 minutes after nominal start.
No lecture after quiz.
Material for QUIZ2: from QUIZ1 to Nov 19;
QUIZ2 will be similar in nature to QUIZ1.

Nov 28 NO CLASS, Thanksgiving

26

Dec 3 Blind Source Separation
Neural factorial codes
L18-ANN-NLPCA-and-ICA-andFA.h.pdf
CF 2/6
Optional:
Neural FA demo code "Bars.McDemo" in Canvas/Code (instructions included)
L18.4_FA-net-learns.mp4
L18.4_FA-net-recall.mp4
L18.2_Factorial-codes-java-annotated.pdf Some insight to code.
L18.1_Neural-ICA-code(Ham&Kostanic).pdf

26

Apr 20 Project presentations Logistics as discussed in class and as (will have been) posted in announcement.