09. Deep Neural Networks
Linear classifier
Loss function & regularization
- Cross-entropy
Activation function
- Sigmoid, tanh, ReLU, Leaky, ReLU, Maxout, ELU
- Biological neuron vs. artificial neuron
Back-propagation
- Computational graph
- Chain rule
- Simple examples of forward pass and backward pass
- Vector/matrix gradient derivatives
Optimization
- Stochastic gradient descent
- Momentum, AdaGrad, Adam
- Learning rate schedules