Neural Networks

01Biological perspective

The basic computational unit of the brain is a neuron. Approx 86 billion in the human nervous system, connected with approx \(10^{15}\) synapses. Each neuron receives input signals from its dendrites and produces output signals along its axon. The axon connects to the dendrites of other neurons via synapses.

If we model the neuron, we can say that the signal that travels along the axon \((x_0)\) interacts with the dendrites of other neurons \((w_0 \cdot x_0)\) based on the synaptic strength of the synapse \((w_0)\).
The idea is that the synaptic strengths (weight \(w\)) are learnable and control the strength of influence (and direction: excitatory being positive weight, inhibitory being negative weight) of one neuron to another.

When the signal is carried to the cell body by the dendrite, the cell body add-up all interaction \((w_i \cdot x_i)\). If the sum is above a certain threshold, the neuron can fire, sending a spike along its axon. The firing rate of the neuron is modeled by an activation function \(f\).

02Feed-forward pass

• \(n\) inputs \(x\)
• 3 neurons in a single hidden layer \(h\)
• 2 outputs \(y\)
• \(W_1\) as a set of weights from \(x\) to \(h\)
• \(W_2\) as a set of weights from \(h\) to \(y\)

Since there is only one hidden layer, there will be only 2 steps in the feed-forward cycle

Step 1: Finding values of \(\bar h\) from input \(\bar x\)

We denote \(W_{ij}\) the weight that connects the input to the hidden neuron \(j\). So the weight that connects input \(2\) to the hidden neuron \(3\) is denoted as \(W_{23}\).

To make sure the values do not explode or increase too much in size, we use an activation function \(\phi\).

Some examples of activation functions

The hyperbolic tangent: to ensure the output is between -1 and 1

The sigmoid: to ensure the output is between 0 and 1

The disadvantage of the sigmoid as an activation function for hidden layers

Let's denote the sigmoid function \(f(x) = \frac{1}{1+e^{-x}}\).
And its derivative \(f'(x) = f(x)(1-f(x))\).

The sigmoid here forces the model to "lose" information from the data. If we plot the derivative and think about the possible max value of the derivative of the sigmoid, the output is squeezed by at least one quarter at each layer during back-propagation, this can become a huge loss of information in deeper neural network.
Sigmoid being between 0 and 1, we can see that the max value of the derivative is 0.25.

In practice, we avoid using the Sigmoid in DNN as activation functions for hidden units.

The Rectified Linear Unit (ReLU): to ensure negative values to be 0 and positive values remain the same

Advantage

• Faster during training
• Good for deep neural networks since the max of the derivative is 1, so no squeezing effect of the error during back-propagation.

Disadvantage

If the learning rate is too high, ReLu units become fragile during the training phase and can die.
large gradient flowing through a ReLU neuron could cause the weights to update in such a way that the neuron will never activate on any data point again. If this happens, then the gradient flowing through the unit will forever be zero from that point on (by Andrej Karpathy here)

In short

They all allow the network to represent nonlinear relationships between its inputs and outputs (crucial because most real world data is nonlinear). But using them is tricky since they contribute to the vanishing gradient problem.

In practice

Use the ReLU, be careful with the learning rates, and possibly monitor the fraction of "dead" units
in a network. If this concerns you, give Leaky ReLU or Maxout a try. Never use sigmoid. Try tanh, but expect it to work worse than ReLU/Maxout.

Step 2: Finding values of \(\bar y\) from the calculated \(\bar h\)

Mathematically the idea is the same as for finding \(\bar h\) in step 1

Once \(\bar y\) is found, adding an activation function is optional. In some problems, we can use the softmax function (ie. multiclass classification). The softmax will allow the values to be between 0 and 1 and the sum of the values will be 1 (good for probabilities).

From the output \(\bar y\) and the ground truth, an error \(E\) is computed (i.e the difference between the predicted output and the desired output).
Since the goal is to find a set of weights that minimizes the error, a backward calculation needs to be done.

The backpropagation is done by using Stochastic Gradient Descent using the chain rule.
Let's consider the error \(E_A\) obtained from weight \(W_A\) at point \(A\) after a forward pass.

To reduce the error, we need to increase the weight (if the \(W_A\) increases the point \(A\) will be lower on the curve, thus reducing the error).

Since the gradient \(\color{red}\nabla\) (derivative or slope of the curve) at point \(A\) is negative (pointing down), we need to change the weights in its negative direction to increase the value of \(W_A\).

Another example at point \(B\):
At point \(B\), the gradient is positive, so if we update the weight in the negative direction of the gradient, it will decrease the weight \(W_B\), thus the error \(E_B\).

Update rule and backpropagation

The update of a single weight is represented as follows

• \(\alpha\) is the learning rate
• \(\frac{\partial E}{\partial W}\) is the partial derivative of the error with respect to this particular weight (We use the partial derivative instead of the derivative to know the effect of this particular weight on the error because the error itself depends on many other variables)
• \(\Delta W_{ij}^k = \alpha(-\frac{\partial E}{\partial W})\) is the backpropagation part, it represents the amount by which the weight needs to be updated between layer \(k\) for neuron \(i\) and layer \(k+1\) at neuron \(j\)