COMP / ELEC / STAT 502
Spring 2024

Schedule

Additional, highly recommended reading from standard text books is also listed. The references to those are given in the Course materials.
>>

No.	Date		Topic		Lecture Notes & Handouts mandatory reading		Homework Upload an electronic copy to Canvas by 2:15pm on the due date unless otherwise indicated in Canvas.		Recommended Reading
1	Jan 9		Orientation, Introduction On your own: Review of basics (L2-ReviewOfBasics.pdf). This will help you, along with exercises in HW02, to gauge your preparedness and to catch up on background if needed. Notions / formulae that I collected in Lecture 2 (L2) will come up in various contexts at various times in the course. Alternatively, you can look these up when the time comes :-)		L1-Introduction-502.pdf L2-ReviewOfBasics.pdf CF2/1		HW01 due 1/11. This is a very light (not real) home work. You will only have to submit answers to my survey to assess backgrounds. Do this HW individually. Form HW groups for HW2 and on. We will discuss HW groups in class.		Haykin Ch1 or Ham & Kostanic Ch 1, Ch2.1-3
2	Jan 11		Introduction, orientation, cont'd Train a single PE		L2-ReviewOfBasics.pdf L1.1-Training-a-single-PE.pdf CF2/1, CF2/2		HW02 due 1/22 (Monday) 2pm (change due to class lost on Jan 16) Part of HW02 problems may be fully graded, others only for turning in and earnestly attempting them. See details in the posted HW02. The exact manner of grading will be determined according to grading help (TBD). The longer time (than one week, slighty overlapping with HW03) is given so that HW groups can form and establish their collaboration.		Haykin Ch10 or Cover & Thomas Ch1 and Ch9
	Jan 16		NO CLASS, due to freeze
3	Jan 18		Associative memory		L4-AssociativeMemory.h.pdf and L4.x files CF1/2				Haykin Ch2.11 or Ham & Kostanic Ch 3.2
4	Jan 23		Associative memory, cont'd		updated L4-AssociativeMemory.h.pdf and L4.x files CF1/2				Haykin Ch2.11 or Ham & Kostanic Ch 3.2
5	Jan 25		Associative memory, cont'd (wrap-up)		updated L4-AssociativeMemory.h.pdf and L4.x files CF1/2		HW03 due 2/1		Haykin Ch2.11 or Ham & Kostanic Ch 3.2
6	Jan 31		Simple Perceptron		L5-Perceptrons.h.pdf and L.5x files CF1/3				Ham & Kostanic Ch 2.5-2.6; or Haykin Ch3
7	Feb 1		Preliminary to Multilayer Perceptron Multilayer Perceptron with Back Propagation		L6.0-Train-a-2layer-network.h.pdf L6-MultiLayerPerceptron.h.part1.pdf CF1/4		Recommended: experiment with Jesse Hellem's HW03 code after you are done with your HW03. Nothing to turn in, just enjoy running it and exploring limits with the characters.ascii data set (all letters plus 10 digits). Code will be available from after the late HW03 deadline passed, noon on Feb 3		Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4
8	Feb 6		Multilayer Perceptron, cont'd Issues With Backprop		L6-MultiLayerPerceptron.h.pdf L7-IssuesWithMLPs.h.part1.pdf CF1/4		HW04 Part I due Feb 15 This is Part I of HW04. Part II will post on Feb 13 (yes, with overlap) and will be due on Feb 20 (so two weeks for HW04 altogether). We will give you feedback on Part I before Part II is due and, in addition to using the feedback to imrpove Part II, you will have a chance to revise and resubmit Part I along with Part II. See details with the Assignment in Canvas.		Ham & Kostanic Ch 3.3.1; or Haykin Ch4.1-4.4
	Feb 8		NO CLASS, registrar cancelled
9	Feb 13		Issues With Backprop: Scaling, convergence, performance monitoring Matrix BP; practical items (time permitting)		L7-IssuesWithMLPs.h.pdf L7.1-Vector-matrix-BP.h.pdf L7.0x-HW04_sample-plots_xxx.pdf sample plots L7.3_Sample-parameter-table_502.pdf CF1/4		HW04 Part II due Feb 20 Part I will have to be re-submitted together with and Part II, with optional revision to Part I. See details with the Assignment(s) in Canvas.		Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8
10	Feb 15		Issues With Backprop: More practical items; cross-validation TBA		L7-IssuesWithMLPs.h.pdf L7.1-Vector-matrix-BP.h.pdf L7.2-BP-practical-items.h.pdf L7.0x-HW04_sample-plots_xxx.pdf samples L7.3_Sample-parameter-table_502.pdf				Ham & Kostanic Ch 3.3.2 - 3.3.7; or Haykin Ch4.5-4.8
11	Feb 20		Cross-validation; Network Pruning: Optimal Brain Damage and Optimal Brain Surgeon		L7.4-Cross-validation.h.pdf L9-NetworkPruning.pdf L9.1-note_OBS-derivations.pdf L9.2-RecursiveEvaluation-of-InverseHessian.pdf		HW05 due Feb 29 (BP application exercises including cross-validation)		Haykin Ch 4.15 pp 222 - 226
12	Feb 22		Finish Network Pruning; Unsupervised learning. Competitive learning: SCL, SOM		L9 files (listed on Feb 20) L11-CompetitiveUnsupervisedLearning.h.pdf L11.1-Self_Organizing_Maps-EM502.pdf CF1/5.6-8				Ham and Kostanic Ch 4.1-4.2; or Haykin CH9.1-6
13	Feb 27		QUIZ1 Material up to Feb 22, excluding competitive learning. Competitive learning, cont'd
14	Feb 29		Competitive learning: SOM, cont'd; visualization, interpretation		L11-CompetitiveUnsupervisedLearning.h.pdf update of L11.1-Self_Organizing_Maps-EM502.pdf CF1/5.6-8		HW06 due March 7, 11:59pm; Deadline extended to March 9 due to unrelated delays; HW06 will also be accepted w/o late penalty until March 14, 11:59pm Please see more details with the HW instructions in Canvas. Review PCA in preparation for PCA networks on March 7. PCA material in Canvas, in Supplements/PCA-Ham&Kostanic-pp396-399.pdf or in any other standard text		Ham and Kostanic Ch 4.1-4.2; or Haykin CH9.1-6
15	Mar 5		Competitive Learning SOM visualization, case studies, wrap-up. Learning Vector Quantization (LVQ)		L11-CompetitiveUnsupervisedLearning.h.pdf L11.2-Self-organized-hand-RitterSchulten.pdf L11.3-CompetitiveSupervisedLearning.LVQ.h.pdf L11.3-LVQ_illustration.pdf CF1/5.6-8				Ham and Kostanic Ch 4.1-4.2, 4.3, Haykin CH9.1-6
16	Mar 7		Unsupervised learning: Oja's PCA net, GHA net		L10-whiteboard.PCAillustration.pdf L10-UnsupervisedLearning-PCAnets.pdf CF2/3 Review of PCA (Principle Component Analysis), from Canvas, Files/Supplements/PCA-Ham&Kostanic-pp396-399.pdf or form any other standard text		HW07 due March 19. See further instructions with the Assignment.		Ham & Kostanic Ch 9.1-9.3.4; Haykin CH8.1-5
			Project proposals are due in Canvas at 10pm, Monday, Mar 18. Strict deadline because I have to read the proposals in order to discuss with you on Tue, Mar 19 (see below). Declare your project group as soon as possible but no later than Sat, March 16 (email me the names of group members) so I can set up the project proposal groups in Canvas for the upload of your proposals. Project groups can be the same as the HW groups, or different.				Please set up a 30-min meeting with me this week as per my Announcement on Mo, Mar 4 to have a preliminary discussion of what you may want to do for a project, what data you are planning to use. Preferably come as a group. For a reminder, before meeting with me please read point 3. (Course Project) at Grading policies and the Course Project requirements . After this meeting, if you have any question about your potential project while drafting your proposal please ask me in email or schedule a short Zoom visit.
	Mar 11 - Mar 17		SPRING BREAK, no classes
17	Mar 19		Project proposal reviews, in my office, starting at 2:30pm, continuing through my office hour.				Logistics TBA in the instructions under Project Proposal assignment (opening on March 17) in Canvas.
18	Mar 21		Oja's normalized Hebbian rule (leftover from 3/7) - postponed; SOM applications, case studies; HW06 discussion		L11.1-Self_Organizing_Maps.apps.pdf				Ham& Kostanic A.5.5
	Mar 21		Exam 1 open at 11:59pm today in Canvas, a 4-hour shift compared to the preview schedule in Exam 1 information, see also in Canvas instructions. Approximate schedule is as at right. Material for midterm: up to (including) March 7 (GHA), and HW07 exercise.				You will have a 36-hour allowance that you can spend in multiple sittings within a contiguous 60-hour window that you choose between the 11:59pm on March 21 and 11:59pm on March 25. Exam 1 is due when your 36 hours have been spent, but no later than 11:59pm on March 25, whichever comes first. No extension, no exceptions.
19	Mar 26		Plans for the rest of the semester; HW06 PP competition; Revised Project Proposals; Discussion of learning and experiment design issues; Oja's normalized Hebbian rule (leftover from 3/7); A few more SOM case studies; Recurrent networks: Hopfield Net (Start, time permitting)		L12-RecurrentNetworks-HopfieldNet.h.pdf if covered CF1/7.1-7.2, CF2/5.4-5.6				Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11
20	Mar 28		PP competition (HW07); Recurrent networks: Hopfield Net		L12-RecurrentNetworks-HopfieldNet.h.pdf CF1/7.1-7.2, CF2/5.4-5.6		From now on, topics listed may or may not be covered on the date indicated, depending on how fast we can go and what else we may have to discuss with higher priority. If a topic does not get covered on the indicated date expect it to come up soon after, and apply the posted lecture notes as appropriate.		Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11
21	Apr 2		Project presentation logistics; VC dimension OR Better, faster learning: Network Growing with Cascade Correlation;		L8.0-Interrelated-issues&VC-Primer.h.pdf L8-VC-Dimension.h.annnot.pdf L8.2_VC-dim-example.pdf L14-NetworkGrowing.h.pdf CH 8.3 - 8.5 from CF1/8				Haykin Ch 2.14; Haykin Ch 2.15; Ham& Kostanic A.5.5
22	Apr 4		Information theoretical objective functions: Cross-entropy for classification in BP; Network Growing with Cascade Correlation OR VC Dimension (whichever was not covered on Apr 2); possibly ANN outputs vs Bayes posterior		L15-InfoTheoreticalObjectiveFunctions.h.pdf CH 5.1 from CF2/5.1-5.3 Recommended (download from Canvas Files/Other, Optional Texts): L15.2-ANNoutput-and-Bayes.pdf (My notes to guide through the Bishop book pages below.) - if covered L15.2-ANN-classificiation-approximates-Bayes-posterior.pdf (Pages from Bishop, 1995.)				Ham& Kostanic A.5.5 Haykin Ch10.1 - 10.5 (refresher)
23	Apr 9		QUIZ2, on-line. (No physical presence needed in the class room.) The Quiz may be in the form of a Canvas Quiz this time. Fallback is the usual Google form. TBD.		Material for QUIZ2: from QUIZ1 to April 4; QUIZ2 will be similar in nature to QUIZ1.
24	Apr 11		Conjugate Gradients Also: recommended slides (on your own) for Simulated Annealing		Supporting lecture slides L13-SimulatedAnnealing.h.pdf L16-ConjugateGradients.h.pdf CF1/7.1-7.2, CF2/5.4-5.6				Ham & Kostanic Ch 5.1 - 5.3, Ch 5.4 - 5.6; Haykin Ch11
25	Apr 16		Blind Source Separation Neural factorial codes		L18-ANN-NLPCA-and-ICA-andFA.h.pdf CF 2/6 Optional: Neural FA demo code "Bars.McDemo" in Canvas/Code (instructions included) L18.4_FA-net-learns.mp4 L18.4_FA-net-recall.mp4 L18.2_Factorial-codes-java-annotated.pdf Some insight to code. L18.1_Neural-ICA-code(Ham&Kostanic).pdf
26	Apr 20		Project presentations		Logistics as discussed in class and posted in announcement.

Return to COMP 502 / ELEC 502 / STAT 502 home page