Deep learning lecture

Comprehensive overview and discussion on modern deep learning architectures and recent advancements

Impt notes: citations and references are not properly included yet. please do not distribute.

List of Topics

1. Basics of Neural Network
   • Basic functional form
   • Universal function approximator
2. Technical difficulties (and solutions)
   • Vanishing gradient problems
   • Vanishing variance problems
   • Ill-posed problem and overfitting
3. Programming exercise 01
   • Pseudo-code
     • Framework comparision
   • PyTorch hands-on
4. Exploiting data properties and structure
   • Recurrent neural network (RNN)
     • Vanilla, LSTM and variants
   • Convolutional neural network (CNN)
     • Notable architectures (AlexNet, VGGNet, GoogLeNet, ResNet, xception)
5. Programming exercise 02
   • RNN and CNN implementation
6. NeuralNet zoo
   • Google AutoML, Caffe model zoo
7. Advanced usage 01 (Into the deep layers)
   • Transfer learning
   • AutoEncoder
   • Neural style transfer
   • U-net
8. Advanced usage 02 (Generative models)
   • Generative adversarial network (GAN)
   • Variational autoencoder (VAE), conditional VAE
     • variational inference
9. Programming exercise 03
   • Implementation templates for Lec07 and Lec08
10. Research Trend 01: AlphaZero
    • Monte Carlo Tree Search, Reinforcement learning
11. Research Trend 02: Deep Query Network (DQN)
    • Algoritm "cocktail" (convLTSM, net-merging)
12. Research Trend 03: NEAT
    • Dynamic neural network
    • model avering
13. Final remarks and tips
    • "Brainless ML" / Automating machine learning
      • step-by-step guide
    • Research trend
    • Unresolved topcis / problems