Course and lecture info

• Course Website
• Video
• Slide

Intro

Deep Learning: Extract patterns from data using neural network.

Finding patterns are usually hand engineered features. Not scalable in practice. Deep learning resurgenced now because of Big Data, hardware, software.

• Stochastic Gradient Descent (1952)
• Perceptron: Learanable Weights (1958)
• Backpropagation: Multi-layer Perceptron (1986)
• Deep Convolutional NN: Digit Recognition (1995)

The perceptron: Forward Propagation

1  - w0 --.   (bias)
x1 - w1 --.
          .---> sum -> non-linear activation function -> output
x2 - w2 --.
xm - wm --.

Common Activation Functions

Probability distribution between 0 and 1. Non-linear functions. The purpose is to introduce non-linearities into the network. Non-linearities allow us to approximate arbitrarily complex functions.

Sigmoid function:

Hyperbolic Tangent:

Rectified Linear Unit (RELU):

Example

We have $w_0=1$ and $W=\begin{bmatrix} 3\\ -2 \end{bmatrix}$.

Which is a 2-D line.

Test with $X = \begin{bmatrix} -1\\ 2 \end{bmatrix}$.

In the result,

$\hat{y}= g(1+3x_1-2x_2)$ is a straightforward formula. In reality, the formula is hard to identify due to the size of the input and result.

Building Neural Networks with Perceptrons

$\hat{y} = g(w_0 + X^TW)$

Simplified:

Multi Output Perceptron like $y_1 = g(z_1), y_2 = g(z_2)$:

Because all inputs are densely connected to all outputs, these layers are called Dense layers.

class MyDenseLayer(tf.keras.layers.Layer):
  def __init__(self, input_dim, output_dim):
    super(MyDenseLayer, self).__init__()

    # Initialize weights and bias
    self.w = self.add_weight([input_dim, output_dim])
    self.b = self.add_weight([1, output_dim])
  
  def call(self, inputs):
    # Forward propagate the inputs
    z = tf.matmul(inputs, self.W) + self.b

    # Feed through a non-linear activation
    output = tf.math.sigmoid(z)

    return output

import tensorflow as tf

layer = tf.keras.layers.Dense(units=2)

Single Layer Neural Network

"Hidden" layers are underlying between the input and the final output.

import tensorflow as tf

model = tf.keras.Sequential([
  tf.keras.layers.Dense(n),
  tf.keras.layers.Dense(2)
])

Deep Neural Network

Make many layers in the network.

import tensorflow as tf

model = tf.keras.Sequential([
  tf.keras.layers.Dense(n_1),
  tf.keras.layers.Dense(n_2),
  # ...
  tf.keras.layers.Dense(2)
])

Applying Neural Networks

Example: "Will I pass this class?"

A simple two feature model.

Need to train the network first.

Quantifying Loss

The loss of our network measures the cost incurred from incorrect predictions.

Fix answers to move closer towards to the true answers.

Empirical Loss

The empirical loss measures the total loss over out entire dataset. Average of all individual losses.

Binary Cross Entropy Loss

Cross entropy loss can be used with models that output a probability between 0 and 1. Introduced by Claude Shannon.

loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y, predicted))

Compares how different these two distributions.

Mean Squared Error Loss

Mean squared error loss can be used with regression models that output continuous real numbers. Possible numbers rather than true or false.

loss = tf.reduce_mean(tf.square(tf.subtract(y, predicted)))

Training Neural Networks

Loss Optimization

We want to find the network weights that achieve the lowest loss.

Find the each of the weights W.

1. Pick random place as an initial $(w_0, w_1)$.
2. Compute gradient. $\frac {\partial J(W)} {\partial W}$ to find maximum ascent.
3. Take small step in opposite direction of gradient
4. Repeat until convergence to local minimum

Graident Descent

Algorithm:

1. Initialize weights randomly $~N(0, \sigma^2)$
2. Loop until convergence:
   1. Compute gradient. $\frac {\partial J(W)} {\partial W}$
   2. Update weights, $W \gets W - \eta \frac {\partial J(W)} {\partial W}$ ($\eta$: learning rate, how much of a step to repeat each iteration)
3. Return weights

import tensorflow as tf
weights = tf.Variable([tf.random.normal()])

while True:
  with tf.GradientTape() as g:
    loss = compute_loss(weights)
    gradient = g.gradient(loss, weights) # Backpropagation

  weights = weights - lr * gradient

Computing Gradients: Backpropagation

Gradients shows how does a small change in one wieght (ex. $w_2$) affect the final loss $J(W)$.

Use chain rule here.

Repeat this for every weight in the network using gradients from later layers. The most frameworks provide the function to calculate this under the hood.

Neural Networks in Practice: Optimization

Training neural networks is difficult.

Loss functions can be difficult to optimize. Optimization through gradient descent: $W \gets W - \eta \frac {\partial J(W)} {\partial W}$. How to decide the learning rate $\eta$?

• Small learning rates: converges slowly and gets stuck in false local minima
• Large learning rates: overshoot, become unstable and diverge
• Stable learning rates: converge smoothly and avoid local minima

Approaches

1. Try many different learning rate. Or,
2. Adaptive learning rate that adapts to the landscape
   • no longer fixed, many algorithms

Adaptive algorithm

• SGD tf.keras.optimizers.SGD
• Adam tf.keras.optimizers.Adam
• Adadelta tf.keras.optimizers.Adadelta
• Adagrad tf.keras.optimizers.Adagrad
• RMSProp tf.keras.optimizers.RMSProp

Ref.:

• Kiefer & Wolfowitz. "Stochastic Estimation of the Maximum of a regression Function." 1952.
• Kingma et al. "Adam: A Method for Stochastic Optimization." 2014.
• Zeiler et al. "ADADELTA: An Adaptive Learning Rate Method." 2012.
• Duchi et al. "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization." 2011.

import tensorflow as tf
model = tf.keras.Sequential([...])

# pick your favorite optimizer
optimizer = tf.keras.optimizer.SGD()

while True:
  # forward pass through the network
  prediction = model(x)

  with tf.GradientTape() as tape:
    # compute the loss
    loss = compute_loss(y, prediction)
  
  # update the weights using the gradient
  grads = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(grads, model.trainable_variables))

Neural Networks in Practice: Mini-batches

Calculating every partials are expensive tasks. Pick single data point $i$ and compute the gradient.

Pick single data point. Easy to compute but very noisy (stochastic):

1. Initialize weights randomly $~N(0, \sigma^2)$
2. Loop until convergence:
   1. Pick single data point $i$
   2. Compute gradient. $\frac {\partial J_i(W)} {\partial W}$
   3. Update weights, $W \gets W - \eta \frac {\partial J(W)} {\partial W}$
3. Return weights

Mini batch of points. Fast to compute and a much better estimate of the true gradient.

1. Initialize weights randomly $~N(0, \sigma^2)$
2. Loop until convergence:
   1. Pick batch of $B$ data points
   2. Compute gradient. $\frac {\partial J(W)} {\partial W} = \frac 1 B \sum_{k=1}^B \frac {\partial J_k(W)} {\partial W}$
   3. Update weights, $W \gets W - \eta \frac {\partial J(W)} {\partial W}$
3. Return weights

• More accurate estimation of gradient
  • Smoother convergence
  • Allows for larger learning rates
• Mini-batches lead to fast training
  • Can parallelize computation
  • achieve significant speed increases on GPU's

Neural Networks in Practice: Overfitting

Overfitting is a general problem in machine learning.

• Underfitting: Model does not have capacity to fully learn the data
• Ideal fit
• Overfitting: Too complex, extra parameters, does not generalize well

Regularization

Technique that constrains our optimization problem to discourage complex models. Improve generalization of of our model on unseen data.

Regularization 1: Dropout

• During training, randomly set some activations to 0
• Typically 'drop' 50% of activations in layer
• Forces network to not rely on any 1 node
• Repeat every iteration
• Build more robust representation of its prediction
• Generalize better to new test data

tf.keras.layers.Dropout(p=0.5)

Regularization 2: Early Stopping

• Stop training before we have a chance to overfit.
• Find the inflection point that diverge the loss from the testing data. The point will be between underfitting and overfitting.

Summary

• The Perceptron
  • Structural building blocks
  • Nonlinear activation functions
• Neural Networks
  • Stacking Perceptrons to form neural networks
  • Optimization through backpropagation
• Training in Practice
  • Adaptive learning
  • Batching
  • Regularization