The Lecture Notes and Handouts constitute the mandatory reading, and consist of two main ingredients:
Additional, highly recommended reading from standard text books is also listed. The references to those are given in the Course materials.
No. |
Date |
Topic |
Lecture Notes & Handouts | >
Homework | >
Recommended Reading |
|||||
1 |
Aug 27 |
CLASS CANCELLED |
HW01 due 8/29. Part I accepted until the late deadline w/o late penalty.
| |||||||
2 |
Aug 29 |
Orientation,
Introduction
|
L2-ReviewOfBasics.pdf CF2/1 |
Form HW groups for HW2 and on. I will send an announcement with some detail. |
Haykin Ch1 |
|||||
3 |
Sep 3 |
Orientation, cont'd (logistics, HW groups, etc.)
|
L2-ReviewOfBasics.pdf
|
HW02 due 9/10
|
Haykin Ch10 |
|||||
4 |
Sep 5 | Associative memory | L4-AssociativeMemory.h.pdf and L4.x files
CF1/2 |
Haykin Ch2.11 or Ham & Kostanic Ch 3.2 |
||||||
5 |
Sep 10 | Associative memory, cont'd | updated L4-AssociativeMemory.h.pdf and L4.x files
CF1/2 |
Haykin Ch2.11 or Ham & Kostanic Ch 3.2 |
||||||
6 |
Sep 12 | Associative memory, cont'd (wrap-up)
Simple perceptron |
updated L4-AssociativeMemory.h.pdf and L4.x files
CF1/2 L5-Perceptrons.h.pdf and L.5x files CF1/3 |
HW03 due 9/24 |
Haykin Ch2.11 or Ham & Kostanic Ch 3.2 Ham & Kostanic Ch 2.5-2.6; or Haykin Ch3 |
|||||
7 |
Sep 17 | Preliminary to Multilayer Perceptron
Multilayer Perceptron with Back Propagation |
L6.0-Train-a-2layer-network.h.pdf
L6-MultiLayerPerceptron.h.part1.pdf CF1/4 |
/td> | Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4 | |||||
8 |
Sep 19 | Multilayer Perceptron, cont'd. Back Propagation rule.
Issues With Backprop |
L6-MultiLayerPerceptron.h.pdf
L7-IssuesWithMLPs.h.part1.pdf CF1/4 |
Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4 | ||||||
9 |
Sep 24 | Issues With Backprop: Scaling, convergence, performance monitoring
Matrix BP; practical items (time permitting) |
L7-IssuesWithMLPs.h.pdf
L7.1-Vector-matrix-BP.h.pdf CF1/4 |
HW04 Part I due Oct 1
Part I will have to be re-submitted together with and Part II (posted ~Oct 1, due ~Oct 8), with optional revision to Part I based on "free" feedback we give you. See details with the Assignment(s) in Canvas. |
Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8 | |||||
10 |
Sep 26 | Issues With Backprop: More practical items;
HW04 samples; code structure overview |
L7-IssuesWithMLPs.h.pdf
L7.1-Vector-matrix-BP.h.pdf L7.0x-HW04_sample-plots_xxx.pdf samples L7.3_Sample-parameter-table_502.pdf |
>Recommended: experiment with Jesse Hellem's HW03 code now that you are done with your HW03. Nothing to turn in, just enjoy running it and exploring limits with the characters.ascii data set (all letters plus 10 digits). The code is in the Files/Code folder. If you use the code to produce reults or write-up of discussion, remember to give credit to Jesse for the code. | Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8 | |||||
11 |
Oct 1 |
HW04 Part II discussion, samples;
Code structure overview; Cross-validation; VC Dimension (time permitting) |
All L7-related files as listed in previous postings;
irisrun_sample.EM.m from Files/Code L7.4-Cross-validation.h.pdf If VC Dimension covered: L8.0-Interrelated-issues&VC-Primer.h.pdf L8-VC-Dimension.h.annnot.pdf L8.2_VC-dim-example.pdf |
HW04 Part II due Oct 8
Part I will have to be re-submitted together with Part II (as one HW04), with optional revision to Part I. See details with the Assignment(s) in Canvas. |
Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8 | |||||
12 |
Oct 3 | Network Pruning: Optimal Brain Damage and Optimal Brain Surgeon |
L9-NetworkPruning.pdf L9.1-note_OBS-derivations.pdf L9.2-RecursiveEvaluation-of-InverseHessian.pdf |
Haykin Ch 4.15 pp 222 - 226 | ||||||
13 |
Oct 8 | Finish Network Pruning;
Unsupervised learning. Competitive learning: SCL, SOM |
L9 files (listed on Oct 3);
L11-CompetitiveUnsupervisedLearning.h.pdf L11.1-Self_Organizing_Maps-EM502.pdf CF1/5.6-8 |
HW05 due Oct 16 (BP application exercises including cross-validation) | Ham and Kostanic Ch 4.1-4.2; or Haykin CH9.1-6 | |||||
14 |
Oct 10 | Competitive learning: SOM, cont'd; visualization, interpretation | L11-CompetitiveUnsupervisedLearning.h.pdf
update of L11.1-Self_Organizing_Maps-EM502.pdf CF1/5.6-8 |
Ham and Kostanic Ch 4.1-4.2; or Haykin CH9.1-6 | ||||||
|
Oct 15 | FALL RECESS, no classes | HW06 due Oct 26 noon
(Posting a little early to allow spacing yourself.) |
|||||||
15 |
Oct 17 | QUIZ1
Material up to Oct 10, excluding competitive learning. Competitive learning: wrap up SOMs |
Review PCA in preparation for PCA networks on Oct 24 PCA material in Canvas, in Supplements/PCA-Ham&Kostanic-pp396-399.pdf or in any other standard text |
|||||||
16 |
Oct 22 | Competitive Learning
SOM visualization, case studies, wrap-up. Learning Vector Quantization (LVQ) |
L11-CompetitiveUnsupervisedLearning.h.pdf
L11.2-Self-organized-hand-RitterSchulten.pdf L11.3-CompetitiveSupervisedLearning.LVQ.h.pdf L11.3-LVQ_illustration.pdf CF1/5.6-8 |
Ham and Kostanic Ch 4.1-4.2, 4.3, Haykin CH9.1-6 | ||||||
17 |
Oct 24 | Unsupervised learning: Oja's PCA net, GHA net | L10-whiteboard.PCAillustration.pdf
L10-UnsupervisedLearning-PCAnets.pdf CF2/3 Review of PCA (Principle Component Analysis), from Canvas, Files/Supplements/PCA-Ham&Kostanic-pp396-399.pdf or from any other standard text |
HW07 due March 19. See further instructions with the Assignment. | Ham & Kostanic Ch 9.1-9.3.4; Haykin CH8.1-5 | |||||
18 |
Oct 29 | HW05, HW06 discussion, practical items | Reminder of what we discussed, and general feedback is in Files/General Feedback, HW05-comments.2024F, HW05.discussion.iris-experiments.pdf, HW05.PP&discussion.pdf, HW06.discussion.pdf, HW06.SOMIllustratioins.pdf | TBA | ||||||
19 |
Oct 31 |
Discussion of class project, including potential topics.
See Files/Supplements/NML502_Project-topics.2024F.pdf for a reminder of suggested topics we discussed, and for a recap of major points.
SOM case studies; Network growing with Cascade Correlation |
update of L11.1-Self_Organizing_Maps-EM502.pdf
L14-NetworkGrowing.h.pdf Form project groups. You may keep your HW group or re-group. Email me your group information, similarly as you did for HW groups, by 8pm, Nov 1. If you are re-grouping from your HW group and have difficulty finalizing it, you may send me your Project group information by noon, Sunday Nov 3 THE LATEST. I must have your Project groups by Sunday so that I can set up a "Project Proposal" assignment in Canvas before the proposal are due on Monday. Project proposals will be due 10pm, Nov 4. |
Project proposal reviews will take place Nov 5 - Nov 7 (with Nov 8 fallback). Groups can start working on their project as soon as I approved their proposal.
Schedule a 30-min meeting with me sometime during these days. Schedule well ahead of time. |
TBA | |||||
|
Nov 1 |
Exam 1 posts tonight in Canvas. Exam 1 information, see also in Canvas instructions. Approximate schedule is as at right. Material for midterm: up to (including) October 24 (GHA), and HW07 exercise. |
You will have a 36-hour allowance that you can spend in multiple sittings between the opening of Exam 1 and the ultimate deadline, 11:59pm on Nov 3. Exam 1 is due when your 36 hours have been spent, but no later than 11:59pm on Nov 3, whichever comes first. No extension, no exceptions. | |||||||
Nov 5 |
NO CLASS
Cancelled by Rice, for Election Day | You may, however, schedule class project proposal review with me on this day. | TBA | |||||||
20 |
Nov 7 |
Discussion, HW06 PP, and general issues from HW06, HW07
Information theoretical objective functions: Start Cross-entropy for classification in BP; |
L15-InfoTheoreticalObjectiveFunctions.Cross-entropy.h.pdf
Slides 1 - 9 were covered.
CH 5.1 from CF2/5.1-5.3 |
Ham& Kostanic A.5.5
Haykin Ch10.1 - 10.5 (refresher) |
||||||
21 |
Nov 12 |
Information theoretical objective functions:
Finish Cross-entropy for classification in BP;
Discussion, HW07 and PP; TBA |
L15-InfoTheoreticalObjectiveFunctions.Cross-entropy.h.pdf
CH 5.1 from CF2/5.1-5.3 |
Ham& Kostanic A.5.5
Haykin Ch10.1 - 10.5 (refresher) |
||||||
22 |
Nov 14 | Recurrent networks: Hopfield Net |
L12-RecurrentNetworks-HopfieldNet.h.pdf
CF1/7.1-7.2, CF2/5.4-5.6 |
From now on, topics listed may or may not be covered on the date indicated, depending on how fast we can go and what else we may have to discuss with higher priority.
If a topic does not get covered on the indicated date expect it to come up soon after, and apply the posted lecture notes as appropriate. |
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11 | |||||
23 |
Nov 19 |
Better, faster learning:
Conjugate Gradients Simulated Annealing as time permits Alternatively: recommended slides (on your own) for Simulated Annealing |
Supporting lecture slides
L13-SimulatedAnnealing.h.pdf L16-ConjugateGradients.h.pdf CF1/7.1-7.2, CF2/5.4-5.6 |
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11 | ||||||
24 |
Nov 21 |
Exam01 discussion
Project presentation logistics; TBA lecture topic |
Follow instruction that I will send in Announcement, for presentation preparation. Instructions will include downloading / reading details for presentation logistics. | |||||||
25 |
Nov 26 | QUIZ2, on-line. (No physical presence needed in the class room.)
Google form URL releaased at 10:50am, nominal start at 10:52am, form closed 45 minutes after nominal start.
No lecture after quiz. |
Material for QUIZ2: from QUIZ1 to Nov 19;
QUIZ2 will be similar in nature to QUIZ1. |
|||||||
Nov 28 | NO CLASS, Thanksgiving | |||||||||
26 |
Dec 3 | Blind Source Separation Neural factorial codes |
L18-ANN-NLPCA-and-ICA-andFA.h.pdf
CF 2/6 Optional: Neural FA demo code "Bars.McDemo" in Canvas/Code (instructions included) L18.4_FA-net-learns.mp4 L18.4_FA-net-recall.mp4 L18.2_Factorial-codes-java-annotated.pdf Some insight to code. L18.1_Neural-ICA-code(Ham&Kostanic).pdf |
|||||||
26 |
Apr 20 | Project presentations | Logistics as discussed in class and as (will have been) posted in announcement. |