05 Neural Networks

Warning

🚧 Դասերի սլայդերը պատրաստ են (L14, L15), իսկ գործնական-տնայինը (ներքևում) պատրաստ է լուծումով։

🎲 Random

TBD

📚 Նյութը

Երկու դասախոսություն (սլայդերը ml/ch5_neural_networks/ պանակում)․

  • L14 — Neural network fundamentals: նեյրոնը որպես ռեգրեսիա, ակտիվացիաներ, թաքնված շերտեր, forward pass, խորությունը (folding). PDF
  • L15 — Training neural networks: backpropagation (ձեռքով + վեկտորիզացված), training loop, dropout / weight decay / augmentation / init+BatchNorm, training curves. PDF

Օպտիմիզատորների (SGD, momentum, Adam) մանրամասների համար՝ Optim մոդուլը (09 Optim: Prerequisites and Gradient Descent, 10 Optim: Momentum + First Order Methods)։

📝 Թեմայի վերաբերյալ հարցաշար (Google Form): TBD

🏡 Տնային

Practical — Train your first neural network (Fashion-MNIST) 🧀🧀

Train a multilayer perceptron on Fashion-MNIST (28×28 grayscale clothing images, 10 classes) and run the experiments from L15. Framework: PyTorch; everything runs on a laptop CPU in a couple of minutes (subsample the data - the starter notes how).

The arc mirrors the lectures: build the net (L14) → train it and fight overfitting (L15). At every step, plot and read the training curves - that habit is the real lesson.

Tasks (1-5 graded; bonuses optional):

  1. Data + a linear baseline 🧀 — load Fashion-MNIST, flatten to 784-vectors, scale to [0, 1], split off a validation set. Fit logistic regression (a single softmax “neuron”, L14) and report its test accuracy - the floor your network must beat.
  2. Build + train an MLP 🧀🧀 — define an nn.Module MLP and write the training loop (forward → loss → zero_grad → backward → step). Train with Adam and beat the baseline.
  3. Learning-rate sweep 🧀🧀 — train the same architecture at three learning rates, plot the training loss, and say which is too high, too low, and just right.
  4. Dropout + weight decay 🧀🧀 — add dropout and L2 weight decay (= Ridge, L01b). Compare the train/val gap with and without. Does regularization improve validation accuracy?
  5. BatchNorm + early stopping 🧀🧀🧀 — on a deeper net, compare training with vs without BatchNorm; then add early stopping on validation loss and report the epoch it picks.
  • Bonus 🎁 — backprop from scratch: reimplement the L15 worked example (the 2-2-1 sigmoid net) in numpy - forward, backward, and a gradient check against finite differences. You should recover f - y = -0.31 and a falling loss.
  • Bonus 🎁 — tuning push: get test accuracy as high as you can (wider net, dropout, weight decay, early stopping; for a real search reuse grid / random / Optuna from lecture [08]). On the 10k subsample an MLP tops out around 85%; the biggest lever is more data - the full 60k training set gets an MLP to roughly 88%.

Solution: nn_practical_solution.ipynb (download) · view on GitHub

Flag Counter