09. Deep Neural Networks

Linear classifier

Loss function & regularization

  • Cross-entropy

Activation function

  • Sigmoid, tanh, ReLU, Leaky, ReLU, Maxout, ELU
  • Biological neuron vs. artificial neuron

Back-propagation

  • Computational graph
  • Chain rule
  • Simple examples of forward pass and backward pass
  • Vector/matrix gradient derivatives

Optimization

  • Stochastic gradient descent
  • Momentum, AdaGrad, Adam
  • Learning rate schedules

Training neural networks (by TA)

Previous
Next