This is part two of my series on how neural nets work. If you’re here, you should have a solid understanding of how forward propagation works. If not, check out my article on it here. Let’s quickly go over everything as a refresher. Our network is a digit classifier for greyscale images with dimensions of 28 by 28. It consists of 4 layers of nodes, which you can think of as compound functions. Connecting nodes to each other are numbers that we call weights. The first layer of our network does not consist of nodes but of our input data, which is a one-dimensional vector of length 784(28 x 28). For each node in the second layer, a dot product of the previous layer and the weights that connect them is computed, the output of which is fed into a ReLU activation function. Do the same for the third layer, except that the input is not our image, but the second layer’s output. For the fourth layer, swap out the ReLU for a Softmax function. The output of our network is a vector of probabilities where each value represents a class(digits 0-9). We are now ready to begin backpropagation but, before we can do that, there are two more topics we have to discuss: batches, and one-hot encoding.

Instead of forward propagating a single image at a time, we’ll pass in many images concurrently in what is called a batch. One we have gone through forward propagation for each piece of data, we can compute the gradients using the average of the errors. This will cause our network to converge faster and have a lower chance of overfitting.

This is a method of converting a scalar to a vector. We often use this encoding as a way to simplify our equations and eventual code. Here’s an example:

Before we can begin to improve our network, let's first figure out how to quantify the word “better”. Your intuition may say being “better” means that the accuracy has increased, but as it turns out there is a more useful statistic. Remember, our Softmax output is not a single digit representing the network's guess, but is a vector of probabilities for each class. You can think of this vector as the confidence that a given class is the correct label. Now, imagine that the input for our network is an image of a 4. If our probability for a 4 is 11%, it could still be our network's guess, as every other probability could be 10% or 9%. This is because the “guess” is simply the plurality of our vector. While we may still get the correct output from our network, it is likely that a small change in our input data will cause this to not be the case. Thus, we want to define our error function in such a way that the error is low when our network has a high confidence for the correct class, and high when it has a low confidence. The first implementation you might think of is 1-X, where X is the correct class’ probability.

However, this simple error function has some drawbacks. We might want our function to punish lower confidences much more than higher confidences. An example of this would be (1-X)^2, called Mean Squared Error. This is very common in regression problems however, for classification problems, we prefer to use another function: Categorical Cross Entropy, or log-loss. I will go over what the function actually is when some of the variables have been explained.

The “learning” in machine learning is the process of tweaking our weights such that our error function will decrease. Essentially, we want to find the minimum of our error function. Let's imagine for a minute that instead of a complex neural net, our function is a simple parabola. If you have taken calculus, you might have some ideas about how to find this minimum. Perhaps we would first find every zero of the function’s first derivative, and then use the second derivative to determine whether those points are minima. While this works very well for our simple parabola, finding every zero of the first derivative is not very practical for complex functions like our neural net. Instead, it’s common to use a strategy called Gradient Descent.

Before we can begin our derivations, we need to figure out how to take derivatives of complex functions. If you have taken any level of calculus, you should hopefully be familiar with the chain rule. If not, the rule is simply a means to compute the derivative of a compound function. It looks like this:

In our context, we use the chain rule to get the derivative of a series of functions. The chain rule will look something like this:

Let's go over the notation we will use to describe our equations before we go any further.

We should also formally define the functions that we use during the derivations.

Intuitively, it makes sense to begin our derivations with our first layer of weights. Unfortunately, as we propagate through our layers, this becomes incredibly difficult. For example, a single weight may only affect a single node in the subsequent layer, but that node affects every node in the layer after that. As you can see, our derivation will quickly spiral out of feasibility. Instead, we will begin from the end of our network and move towards the front. This is where the name backpropagation comes from, as we are propagating our gradients backwards.

Let’s begin at the very end of our network with our error function. Remember, we are using Categorical Cross Entropy, which has the property that its derivative when combined with the Softmax becomes very simple. This essentially means that during backpropagation, we are effectively combining these two functions into one and computing only a single derivative, bypassing the chain rule altogether.

OK, we can compute the derivative of the last layer of weights with respect to our error function. Let’s move on to the next layer of weights. Because we have the derivative of the linear combination of the last layer with respect to the error function, we need to find just three gradients:

The process for finding the third layer of gradients is just about exactly the same as the second layer, so I won’t include it in this post, however you are encouraged to derive it yourself.

Now that we have the derivatives for each of our weights with respect to the error function, we can enjoy the fruits of our labor and update them as we do in Gradient Descent. This is very simple, and will look like this:

The alpha in each of these equations is our learning rate, yet another hyperparameter. The learning rate determines the size of our gradient descent steps. A typical value for this would be 0.1.

That's it! You now (hopefully) understand both forward and backward propagation. All that's left to do is repeat this process using new images in our input batch each time. As you perform more iterations, your network will slowly become more and more accurate.