The Lecture Notes and Handouts constitute the mandatory reading, and consist of two main ingredients:
Additional, highly recommended reading from standard text books is also listed. The references to those are given in the Course materials.
No. |
Date |
Topic |
Lecture Notes & Handouts | >
Homework | >
Recommended Reading |
|||||
1 |
Jan 9 |
Orientation,
Introduction
|
L2-ReviewOfBasics.pdf CF2/1 |
HW01 due 1/11.
|
Haykin Ch1 |
|||||
2 |
Jan 11 |
Introduction, orientation, cont'd
|
L2-ReviewOfBasics.pdf
|
HW02 due 1/22 (Monday) 2pm (change due to class lost on Jan 16)
|
Haykin Ch10 |
|||||
|
Jan 16 | NO CLASS, due to freeze | ||||||||
3 |
Jan 18 | Associative memory | L4-AssociativeMemory.h.pdf and L4.x files
CF1/2 |
Haykin Ch2.11 or Ham & Kostanic Ch 3.2 |
||||||
4 |
Jan 23 | Associative memory, cont'd | updated L4-AssociativeMemory.h.pdf and L4.x files
CF1/2 |
Haykin Ch2.11 or Ham & Kostanic Ch 3.2 |
||||||
5 |
Jan 25 | Associative memory, cont'd (wrap-up) | updated L4-AssociativeMemory.h.pdf and L4.x files
CF1/2 |
HW03 due 2/1 |
Haykin Ch2.11 or Ham & Kostanic Ch 3.2 |
|||||
6 |
Jan 31 | Simple Perceptron | L5-Perceptrons.h.pdf and L.5x files
CF1/3 |
Ham & Kostanic Ch 2.5-2.6; or Haykin Ch3 | ||||||
7 |
Feb 1 | Preliminary to Multilayer Perceptron
Multilayer Perceptron with Back Propagation |
L6.0-Train-a-2layer-network.h.pdf
L6-MultiLayerPerceptron.h.part1.pdf CF1/4 |
Recommended: experiment with Jesse Hellem's HW03 code after you are done with your HW03. Nothing to turn in, just enjoy running it and exploring limits with the characters.ascii data set (all letters plus 10 digits). Code will be available from after the late HW03 deadline passed, noon on Feb 3 | Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4 | |||||
8 |
Feb 6 | Multilayer Perceptron, cont'd
Issues With Backprop |
L6-MultiLayerPerceptron.h.pdf
L7-IssuesWithMLPs.h.part1.pdf CF1/4 |
HW04 Part I due Feb 15
This is Part I of HW04. Part II will post on Feb 13 (yes, with overlap) and will be due on Feb 20 (so two weeks for HW04 altogether). We will give you feedback on Part I before Part II is due and, in addition to using the feedback to imrpove Part II, you will have a chance to revise and resubmit Part I along with Part II. See details with the Assignment in Canvas. |
Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4 | |||||
|
Feb 8 | NO CLASS, registrar cancelled | ||||||||
9 |
Feb 13 | Issues With Backprop: Scaling, convergence, performance monitoring
Matrix BP; practical items (time permitting) |
L7-IssuesWithMLPs.h.pdf
L7.1-Vector-matrix-BP.h.pdf L7.0x-HW04_sample-plots_xxx.pdf sample plots L7.3_Sample-parameter-table_502.pdf CF1/4 |
HW04 Part II due Feb 20
Part I will have to be re-submitted together with and Part II, with optional revision to Part I. See details with the Assignment(s) in Canvas. |
Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8 | |||||
10 |
Feb 15 | Issues With Backprop: More practical items; cross-validation
TBA |
L7-IssuesWithMLPs.h.pdf
L7.1-Vector-matrix-BP.h.pdf L7.2-BP-practical-items.h.pdf L7.0x-HW04_sample-plots_xxx.pdf samples L7.3_Sample-parameter-table_502.pdf |
Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8 | ||||||
11 |
Feb 20 | Cross-validation;
Network Pruning: Optimal Brain Damage and Optimal Brain Surgeon |
L7.4-Cross-validation.h.pdf
L9-NetworkPruning.pdf L9.1-note_OBS-derivations.pdf L9.2-RecursiveEvaluation-of-InverseHessian.pdf |
HW05 due Feb 29 (BP application exercises including cross-validation) | Haykin Ch 4.15 pp 222 - 226 | |||||
12 |
Feb 22 | Finish Network Pruning;
Unsupervised learning. Competitive learning: SCL, SOM |
L9 files (listed on Feb 20)
L11-CompetitiveUnsupervisedLearning.h.pdf L11.1-Self_Organizing_Maps-EM502.pdf CF1/5.6-8 |
Ham and Kostanic Ch 4.1-4.2; or Haykin CH9.1-6 | ||||||
13 |
Feb 27 | QUIZ1
Material up to Feb 22, excluding competitive learning. Competitive learning, cont'd |
||||||||
14 |
Feb 29 | Competitive learning: SOM, cont'd; visualization, interpretation | L11-CompetitiveUnsupervisedLearning.h.pdf
update of L11.1-Self_Organizing_Maps-EM502.pdf CF1/5.6-8 |
HW06 due March 7, 11:59pm;
Deadline extended to March 9 due to unrelated delays; HW06 will also be accepted w/o late penalty until March 14, 11:59pm Please see more details with the HW instructions in Canvas. Review PCA in preparation for PCA networks on March 7. PCA material in Canvas, in Supplements/PCA-Ham&Kostanic-pp396-399.pdf or in any other standard text |
Ham and Kostanic Ch 4.1-4.2; or Haykin CH9.1-6 | |||||
15 |
Mar 5 | Competitive Learning
SOM visualization, case studies, wrap-up. Learning Vector Quantization (LVQ) |
L11-CompetitiveUnsupervisedLearning.h.pdf
L11.2-Self-organized-hand-RitterSchulten.pdf L11.3-CompetitiveSupervisedLearning.LVQ.h.pdf L11.3-LVQ_illustration.pdf CF1/5.6-8 |
Ham and Kostanic Ch 4.1-4.2, 4.3, Haykin CH9.1-6 | ||||||
16 |
Mar 7 | Unsupervised learning: Oja's PCA net, GHA net | L10-whiteboard.PCAillustration.pdf
L10-UnsupervisedLearning-PCAnets.pdf CF2/3 Review of PCA (Principle Component Analysis), from Canvas, Files/Supplements/PCA-Ham&Kostanic-pp396-399.pdf or form any other standard text |
HW07 due March 19. See further instructions with the Assignment. | Ham & Kostanic Ch 9.1-9.3.4; Haykin CH8.1-5 | |||||
Project proposals are due in Canvas at 10pm, Monday, Mar 18. Strict deadline because I have to read the proposals in order to discuss with you on Tue, Mar 19 (see below).
Declare your project group as soon as possible but no later than Sat, March 16 (email me the names of group members) so I can set up the project proposal groups in Canvas for the upload of your proposals. Project groups can be the same as the HW groups, or different. |
Please set up a 30-min meeting with me this week as per my Announcement on Mo, Mar 4 to have a preliminary discussion of what you may want to do for a project, what data you are planning to use. Preferably come as a group. For a reminder, before meeting with me please read point 3. (Course Project) at Grading policies and the Course Project requirements . After this meeting, if you have any question about your potential project while drafting your proposal please ask me in email or schedule a short Zoom visit. | |||||||||
|
Mar 11 - Mar 17 | SPRING BREAK, no classes | ||||||||
17 |
Mar 19 | Project proposal reviews, in my office, starting at 2:30pm, continuing through my office hour. | Logistics TBA in the instructions under Project Proposal assignment (opening on March 17) in Canvas. | |||||||
18 |
Mar 21 |
Oja's normalized Hebbian rule (leftover from 3/7) - postponed;
SOM applications, case studies;
HW06 discussion | L11.1-Self_Organizing_Maps.apps.pdf | Ham& Kostanic A.5.5 | ||||||
|
Mar 21 |
Exam 1 open at 11:59pm today in Canvas, a 4-hour shift compared to the preview schedule in Exam 1 information, see also in Canvas instructions. Approximate schedule is as at right. Material for midterm: up to (including) March 7 (GHA), and HW07 exercise. |
You will have a 36-hour allowance that you can spend in multiple sittings within a contiguous 60-hour window that you choose between the 11:59pm on March 21 and 11:59pm on March 25. Exam 1 is due when your 36 hours have been spent, but no later than 11:59pm on March 25, whichever comes first. No extension, no exceptions. | |||||||
19 |
Mar 26 |
Plans for the rest of the semester;
HW06 PP competition; Revised Project Proposals; Discussion of learning and experiment design issues; Oja's normalized Hebbian rule (leftover from 3/7); A few more SOM case studies; Recurrent networks: Hopfield Net (Start, time permitting) |
L12-RecurrentNetworks-HopfieldNet.h.pdf if covered
CF1/7.1-7.2, CF2/5.4-5.6 |
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11 | ||||||
20 |
Mar 28 | PP competition (HW07);
Recurrent networks: Hopfield Net |
L12-RecurrentNetworks-HopfieldNet.h.pdf
CF1/7.1-7.2, CF2/5.4-5.6 |
From now on, topics listed may or may not be covered on the date indicated, depending on how fast we can go and what else we may have to discuss with higher priority.
If a topic does not get covered on the indicated date expect it to come up soon after, and apply the posted lecture notes as appropriate. |
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11 | 21 |
Apr 2 | Project presentation logistics;
VC dimension OR Better, faster learning: Network Growing with Cascade Correlation; |
L8.0-Interrelated-issues&VC-Primer.h.pdf
L8-VC-Dimension.h.annnot.pdf L8.2_VC-dim-example.pdf L14-NetworkGrowing.h.pdf CH 8.3 - 8.5 from CF1/8 |
Haykin Ch 2.14; Haykin Ch 2.15; Ham& Kostanic A.5.5 |
22 |
Apr 4 |
Information theoretical objective functions:
Cross-entropy for classification in BP; Network Growing with Cascade Correlation OR VC Dimension (whichever was not covered on Apr 2); possibly ANN outputs vs Bayes posterior |
L15-InfoTheoreticalObjectiveFunctions.h.pdf
CH 5.1 from CF2/5.1-5.3 Recommended (download from Canvas Files/Other, Optional Texts): L15.2-ANNoutput-and-Bayes.pdf (My notes to guide through the Bishop book pages below.) - if covered L15.2-ANN-classificiation-approximates-Bayes-posterior.pdf (Pages from Bishop, 1995.) |
Ham& Kostanic A.5.5
Haykin Ch10.1 - 10.5 (refresher) |
||||||
23 |
Apr 9 | QUIZ2, on-line. (No physical presence needed in the class room.) The Quiz may be in the form of a Canvas Quiz this time. Fallback is the usual Google form. TBD. | Material for QUIZ2: from QUIZ1 to April 4;
QUIZ2 will be similar in nature to QUIZ1. |
|||||||
24 |
Apr 11 |
Conjugate Gradients
Also: recommended slides (on your own) for Simulated Annealing |
Supporting lecture slides
L13-SimulatedAnnealing.h.pdf L16-ConjugateGradients.h.pdf CF1/7.1-7.2, CF2/5.4-5.6 |
Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11 | ||||||
25 |
Apr 16 | Blind Source Separation Neural factorial codes |
L18-ANN-NLPCA-and-ICA-andFA.h.pdf
CF 2/6 Optional: Neural FA demo code "Bars.McDemo" in Canvas/Code (instructions included) L18.4_FA-net-learns.mp4 L18.4_FA-net-recall.mp4 L18.2_Factorial-codes-java-annotated.pdf Some insight to code. L18.1_Neural-ICA-code(Ham&Kostanic).pdf |
|||||||
26 |
Apr 20 | Project presentations | Logistics as discussed in class and posted in announcement. |
Return to COMP 502 / ELEC 502 / STAT 502 home page