Convolutional Neural Networks (CNNs)

01Structure of a CNN

Convolutional layer

The convolutional layer act as a feature extractor in order to find spatial information/pattern from an input image.
The CNN is made of several filters or kernels \(K\) stacked together, and the values inside these kernels are the weights. Each kernel are responsible for extracting a specific kind of feature (i.e High pass kernel will extract edges).
The kernels are convolved with the input image and slide over it to produce multiple filtered versions of the image. So 3 kernels will produce 3 filtered images.

The output of a convolutional layer is a set of feature maps also called activation maps which are filtered versions of the input image.
Since we have multiple kernels, they will be stacked together so the dimensionality on the C axis (Depth) increases (here from 1 to 3).

Activation Function

I have described the role of activation function in this note on neural networks and here is an interpretation of activation functions when it comes to images.
Image pixels range from 0-255, but in practice, the input image to a CNN has pixels ranging from 0-1 because neural networks work better with scaled values. The process of scaling 0-255 to 0-1 is called Normalization. Then the normalized image passes through a CNN.

After the convolution operation, the values of pixels fall in a different range, we may even have negative values.

To account for this change in pixel value, we apply an activation function that transforms each pixel value (i.e 0 when negative, remains the same if not: ReLu).

Max pooling Layer

After a convolutional layer + activation comes a pooling layer.
The most common type of pooling layer is a max-pooling layer. The max-pooling operation breaks an image into smaller patches. A max-pooling layer is defined by a patch size and a stride.
The patch then moves by some stride across the entire image and reduces the image (H x W) dimension. For a patch of size 2x2 and a stride of 2, this window will perfectly cover the image.

Usually, choose a patch size and a stride size that are the same to make sure we are covering the entire image, otherwise, we could have overlap if the stride is too small or we can miss some pixels if too large.
Here is an example of the max pooling operation:

Let's zoom in on a part of an image (left) and choose a patch size of 2x2 (right) for the max pooling operation

Now the goal is to fill the first value (upper-left) of the patch by applying the max pooling operation.
Since the patch is 2x2 in size, the pooling operation is applied on the first 2x2 window of the image.

Since Max-pooling has been chosen, the operation is to take the maximum of the four values \(\color{red}max(20, 45, 140, 58)=140\)

Then, if we chose a slide size of 2, the window will shift by 2 on the right, and we apply the same operation.

And the process continues until the image is covered

Before the pooling operation, the sample image size had a 4x4 size. After applying a max-pooling operation with a stride of 2 and a patch size of 2, the image dimension has been reduced by half on H and W axis.

So when the image goes through the CNN, the depth increase after the convolution operation since it is convolved with a stack of kernels to produce a stack of the feature map, then the height and width decrease after the max-pooling operation.

Why is this useful?

• Since the image increase in depth, reducing the dimension helps in having a scalable network
• Makes a network resistant to small pixel value changes in an input image
• It increases in the field of view for later layers. Since the H and W are reduced, a later kernel will be able to see a larger version of the image at once.

Here is a drawing to illustrate what I mean by the last point

The smaller the scene is relative to my eye, the wider my field of view will be

Fully-connected Layer

At the end of a convolutional neural network, is a fully-connected layer (sometimes more than one). Fully-connected means that every output that's produced at the end of the last pooling layer is input to each node in this fully-connected layer. For example, for a final pooling layer that produces a stack of outputs that are 20x20x10 (H, W, C), the fully-connected layer will see 20x20x10 = 4000 inputs.

The role of the last fully-connected layer is to produce a list of class scores (in the case of image classification). So, the last fully-connected layer will have as many nodes as there are classes.
Again, as explained here we can apply an optional activation function at the end, for example, softmax for image classification.

02CNN model interpretability

Deep learning models are known for being a black box. But this doesn't mean we cannot interpret their results. Some technics exist to understand for example why a model has predicted a particular class.

Occlusion Experiments

Which area of the image is most important in classifying this image?

Occlusion means to block out or mask part of an image or object. For example, if you are looking at a person but their face is behind a book; this person's face is hidden (occluded). Occlusion can be used in feature visualization by blocking out selective parts of an image and seeing how a network responds.

The process for an occlusion experiment is as follows:

• Mask part of an image before feeding it into a trained CNN,
• Draw a heatmap of class scores for each masked image
• Slide the masked area to a different spot and repeat steps 1 and 2

The result should be a heatmap that shows the predicted class of an image as a function of which part of an image was occluded. The reasoning is that if the class score for a partially occluded image is different than the true class, then the occluded area was likely very important!

Saliency Maps

Which pixels are most important in classifying this image?

Salience can be thought of as the importance of something.

Saliency maps aim to show these important pictures by computing the gradient of the class score with respect to the image pixels. A gradient is a measure of change, and so, the gradient of the class score with respect to the image pixels is a measure of how much a class score for an image changes if a pixel changes a little bit. So it identifies the most important pixel that allows classifying a certain image in a certain class

Guided Backpropagation

If we change this pixel value slightly, how will the output of a particular neuron or layer in the network change?

Similar to the process for constructing a saliency map, we can compute the gradients for mid-level neurons in a network with respect to the input pixels.

If the expected output change a lot, then the pixel that experienced a change is important to that particular layer.

This is very similar to the backpropagation steps for measuring the error between an input and output and propagating it back through a network. Guided backpropagation tells us exactly which parts of the image patches, that we've looked at, activate a specific neuron/layer.