<jupyter_start><jupyter_text>All those moments will be los(s)t in time, like tears in rain.> I am completely operational, and all my circuits are functioning perfectly.- toc: true - badges: true- comments: true- categories: [jupyter]- image: images/posts/2020-12-10-Tears-In-Rain/Tears-In-Rain.jpg<jupyter_code>#hide
!pip install -Uqq fastbook
import fastbook
fastbook.setup_book()
#hide
from fastai.vision.all import *
from fastbook import *

matplotlib.rc('image', cmap='Greys')<jupyter_output><empty_output><jupyter_text>Under the Hood: Training a Digit Classifier This one is the big one. Now we have made it past the introduction of the course it is time to get under the hood and start implementing some of the functionality for ourselves. I am going to be taking quite thorough notes as I want to make sure I understand everything before I move onto Chapter 5. So far all the heavy lifting has been done for us, the fastai library is a nice high level API written on top of PyTorch and abstracted in such a way that the "magic" / "obfuscation" is a little hard to determine what is actually happening. So in this chapter we will be making a hand written digit classifier, a simple classifier that can determine whether or not a 28px * 28px image of a hand written digit is either a **3** or a **7**. We will try to figure out a good baseline from which we can assess our model and then proceed to write out each element of the model in simple python, before seeing it all wrapped up in the nice and tidy fastai API.  Pixels: The Foundations of Computer Vision Here we are going to take a look at what we are actually dealing with when it comes to our hand written digits. Thankfully there is a great training set put together by <NAME> called the [MNIST](https://en.wikipedia.org/wiki/MNIST_database) or Modified National Institute of Standards and Technology database. It contains thousands of individual hand written digits that have been collated as 28px * 28px grayscale images. This is the data we will be using to build our classifier.   The team at fastai have made it simple to download and unzip the data we are going to use for this lesson. Instead of having to manually go to the [fastai datasets](https://course.fast.ai/datasets) documentation page, download the tar file, unzip it in a location accessible to your notebooks they have a handy [utility](https://github.com/fastai/fastai/blob/715c027b0ad8586feda09e29fb2b483dfe30c910/fastai/data/external.py) that will do all of that in single line of code:<jupyter_code>path = untar_data(URLs.MNIST_SAMPLE)
#hide
Path.BASE_PATH = path<jupyter_output><empty_output><jupyter_text>The have also added the UNIX [ls()](https://en.wikipedia.org/wiki/Ls) function to Python [path](https://docs.python.org/3/library/pathlib.html) to give a handy way to see what is in our directory. Here you can see we have our training and validation directory along with a label.csv file.<jupyter_code>path.ls()<jupyter_output><empty_output><jupyter_text>Inside each of the training and validation directories we have a folder which contains all of our **3's** and **7's** respectively<jupyter_code>(path/'train').ls()<jupyter_output><empty_output><jupyter_text>Now we can create a list of the file paths and store them in variables<jupyter_code>threes = (path/'train'/'3').ls().sorted()
sevens = (path/'train'/'7').ls().sorted()
threes<jupyter_output><empty_output><jupyter_text>Let's take a look at what we are working with. By indexing into the above **threes** list we can retrive the first path in the list. Using [PIL](https://pillow.readthedocs.io/en/stable/) the Python Imaging Library, we can display the data at that path.<jupyter_code>im3_path = threes[1]
im3 = Image.open(im3_path)
im3<jupyter_output><empty_output><jupyter_text>At this point it may not be entirely intuative what image information is. It is just an array of values for 0 to 255 for an 8-bit grayscale image like we have above. By casting into a Numpy array and taking a slice we can se what that might look like.<jupyter_code>array(im3)[4:10,4:10]<jupyter_output><empty_output><jupyter_text>As we have seen earlier in the course the PyTorch tensors have very similar functionality to Numpy arrays, but have the added benefit of being pushed to the GPU, this is a optimization and can be used in replacement of standard Numpy arrays. As a huge fan of Numpy my natural propensity is to use them all the time. But I think I need to reconsider . . .<jupyter_code>tensor(im3)[4:10,4:10]<jupyter_output><empty_output><jupyter_text>By loading the tensor into a Pandas Dataframe we are able to see what the pixel values look like by using the .background_gradients method<jupyter_code>im3_t = tensor(im3)
df = pd.DataFrame(im3_t[4:15,4:22])
df.style.set_properties(**{'font-size':'6pt'}).background_gradient('Greys')<jupyter_output><empty_output><jupyter_text>![](2020-12-10-Tears-In-Rain/Datatable.jpg)  First Try: Pixel Similarity Before we start building our model we should consider if there are feasible alternatives to our problem. Working with Deep Learning is like weilding a very large hammer and every problem can appear to us as a nail. But for small self contained problem that do not require scale, simple alternatives are preferable. Here we are exploring an alternative to determine a baseline. What results can we achieve with a naive approach, is this sufficient to solve our problem and can they be improved by the use of a deep learning model. This is an important step, making sure that you are making headway on your problem is more important than having a shiny new model.  Problem Statement Can we create a simple baseline application that can classify an unseen 28px * 28px image of a **3** or a **7**. Theoretically, this is a simpler task than differentiating between a **3** and an **8** for example because both have similar curves at the top of the number and the bottom which might be difficult to disambiguate. If we try making a `mean` image from all the data in the **threes** folder and another for the **sevens** and compare the `pixel similarity` of an unseen image with this `mean` image we might be able to classify it with some manner of accuracy   Process Let's create two more arrays that store a tensor of each image in our **threes** and **sevens** folders.<jupyter_code>seven_tensors = [tensor(Image.open(o)) for o in sevens]
three_tensors = [tensor(Image.open(o)) for o in threes]
len(three_tensors),len(seven_tensors)<jupyter_output><empty_output><jupyter_text>In a similar way to the method above to cast our image to an array or tensor, we can use the fastai `show_image` method to take a tensor and display it as an image, this will be useful for debugging<jupyter_code>show_image(three_tensors[1]);<jupyter_output><empty_output><jupyter_text>Our next operation is to normalize the values of the images between 0 and 1, first we stack the images on top of one another using `torch.stack`, convert the integer values in the tensor stack to a float to ensure that values aren't rounded after our division operation, and then we divide the image by 255.  By looking at the shape of the torch tensor stack we can see that we have 6131 image tensors that have 28 rows and 28 columns<jupyter_code>stacked_sevens = torch.stack(seven_tensors).float()/255
stacked_threes = torch.stack(three_tensors).float()/255
stacked_threes.shape<jupyter_output><empty_output><jupyter_text>The length of the tensor.shape is the same as the tensor rank or number of dimensions, we can see that below by using the `ndim` method. This is a fast way of checking that your tensors have the correct rank before moving forward building your model or baseline<jupyter_code>len(stacked_threes.shape)
stacked_threes.ndim<jupyter_output><empty_output><jupyter_text>Now that we have all of our image tensors stack on top of one another and normalized, we can use the `mean` method to find the average. By passing in the argument 0, we are telling the operation to take the `mean` across the first axis. In this example this is the `mean` of the 6131 images.<jupyter_code>mean3 = stacked_threes.mean(0)
show_image(mean3);
mean7 = stacked_sevens.mean(0)
show_image(mean7);<jupyter_output><empty_output><jupyter_text>Now that we have our `mean` images, we can compare one of the images against them to see what the `pixel similarity` is and determine if the image is either a **3** or a **7**<jupyter_code>a_3 = stacked_threes[1]
show_image(a_3);<jupyter_output><empty_output><jupyter_text>L1 and L2 norm - Take the mean of the absolute value of differences (absolute value is the function that replaces negative values with positive values). This is called the mean absolute difference or L1 norm- Take the mean of the square of differences (which makes everything positive) and then take the square root (which undoes the squaring). This is called the root mean squared error (RMSE) or L2 norm. To determine the `pixel similarity` of our test image and our mean image we are going to use the `mean absolte difference` or the `L1 Norm` and the `root mean squared error` or the `L2 Norm`.  If we were simply going to subtract one for the other we could end up with negative values, which once averaged would negate positive values and give us inconcluse information. By comparing the test image againt the mean of the **3's** and the mean of the **7's** we can see that error is lower when comparing to the mean of the **3's** giving us a classification that this image is of a **3**<jupyter_code>dist_3_abs = (a_3 - mean3).abs().mean()
dist_3_sqr = ((a_3 - mean3)**2).mean().sqrt()
dist_3_abs,dist_3_sqr
dist_7_abs = (a_3 - mean7).abs().mean()
dist_7_sqr = ((a_3 - mean7)**2).mean().sqrt()
dist_7_abs,dist_7_sqr<jupyter_output><empty_output><jupyter_text>Here we can see by using the `l1_loss` and the `mse_loss` methods we are getting the same answers our implementations above<jupyter_code>F.l1_loss(a_3.float(),mean7), F.mse_loss(a_3,mean7).sqrt()<jupyter_output><empty_output><jupyter_text>Computing Metrics Using Broadcasting At this point we could simply use a loop and iterate through each of the images in the validation set, calculate the `L1` loss for each image and determine whether or not our baseline application determines the number to be a **3** or a **7**, check the prediction against and determine an accuracy. This would take a very long time and wouldn't make use of the GPU accelleration needed for Deep Learning. Here we are going to look at the secret sauce that makes Python (an interpreted language) powerful enough to be used in Deep Learning applications. What is that secret sauce you ask? `Broadcasting` So lets start by stacking the images from our validation directories in the same way we did with our training images. We will normalize them and check the shape of the tensor stack<jupyter_code>valid_3_tens = torch.stack([tensor(Image.open(o)) 
                            for o in (path/'valid'/'3').ls()])
valid_3_tens = valid_3_tens.float()/255
valid_7_tens = torch.stack([tensor(Image.open(o)) 
                            for o in (path/'valid'/'7').ls()])
valid_7_tens = valid_7_tens.float()/255
valid_3_tens.shape,valid_7_tens.shape<jupyter_output><empty_output><jupyter_text>Here we are going to write our `L1 Norm` in the form of a function, here taking the mean across the rows and columns of the absolute difference of the tow images.<jupyter_code>def mnist_distance(a,b): return (a-b).abs().mean((-1,-2))
mnist_distance(a_3, mean3)<jupyter_output><empty_output><jupyter_text>But by using `Broadcasting`, we can instead use our entire tensor stack and compare it against the `mean` image of the **3** all at once. What is happening under the hood is that PyTorch is making a "virtual" copy of the mean image tensor for every image tensor in the validation stack so it can determine the `pixel similarity` for all of them at once. What it returns is a tensor of all of the results.<jupyter_code>valid_3_dist = mnist_distance(valid_3_tens, mean3)
valid_3_dist, valid_3_dist.shape<jupyter_output><empty_output><jupyter_text>Now lets create a simple definition to determine if the prediction is a **3** if the result is `False` then we assume that the image is classified as a **7** If the `mnist_distance` measured againt the `mean` **3** is lower than the `mean` **7** then the function will return `True`<jupyter_code>def is_3(x): return mnist_distance(x,mean3) < mnist_distance(x,mean7)<jupyter_output><empty_output><jupyter_text>Here by passing in our test image we can see that it returns `True`, but by casting it to a float we can get a value instead<jupyter_code>is_3(a_3), is_3(a_3).float()<jupyter_output><empty_output><jupyter_text>Taking advantage of `Broadcasting` we can pass the entire tensor stack to the function and we get back an array of predictions<jupyter_code>is_3(valid_3_tens)<jupyter_output><empty_output><jupyter_text>Lets check the accuracy of our classification application. We are expecting every image tensor in the `valid_3_tens` tensor to return true and all the image tensors in the `valid_7_tens` tensor to return false. We can convert them to a floating point value and take the `mean` to determine the accuracy<jupyter_code>accuracy_3s =      is_3(valid_3_tens).float() .mean()
accuracy_7s = (1 - is_3(valid_7_tens).float()).mean()

accuracy_3s,accuracy_7s,(accuracy_3s+accuracy_7s)/2<jupyter_output><empty_output><jupyter_text>Wow! It looks like our naive `pixel similarity` application gives us a **95%** accuracy on this task!  We have only used PyTorch for its power tensor operations in this task and proven that with a simple baseline we can determine a highly accurate classifier. Let's see if making a Deep Learning model we can improve the accuracy even further!  Stochastic Gradient Descent (SGD) Up until this point we have a classifier but it really does follow the description by <NAME>  **Suppose we arrange for some automatic means of testing the effectiveness of any current weight assignment in terms of actual performance and provide a mechanism for altering the weight assignment so as to maximize the performance. We need not go into the details of such a procedure to see that it could be made entirely automatic and to see that a machine so programmed would "learn" from its experience.** To turn our function into a machine learning classifier we will need: - Initialize the weights.- For each image, use these weights to predict whether it appears to be a 3 or a 7.- Based on these predictions, calculate how good the model is (its loss).- Calculate the gradient, which measures for each weight, how changing that weight would change the loss- Step (that is, change) all the weights based on that calculation.- Go back to the step 2, and repeat the process.- Iterate until you decide to stop the training process (for instance, because the model is good enough or you don't want to wait any longer).<jupyter_code>gv('''
init->predict->loss->gradient->step->stop
step->predict[label=repeat]
''')<jupyter_output><empty_output><jupyter_text>![](2020-12-10-Tears-In-Rain/SGD.jpg) To understand `SGD` a little better lets start with a simpler example using this quadratic function:<jupyter_code>def f(x): return x**2<jupyter_output><empty_output><jupyter_text>Let's plot what that function looks like:<jupyter_code>plot_function(f, 'x', 'x**2')<jupyter_output>/opt/conda/envs/fastai/lib/python3.8/site-packages/fastbook/__init__.py:73: UserWarning: Not providing a value for linspace's steps is deprecated and will throw a runtime error in a future release. This warning will appear only once per process. (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729096996/work/aten/src/ATen/native/RangeFactories.cpp:23.)
  x = torch.linspace(min,max)<jupyter_text>The sequence of steps we described earlier starts by picking some random value for a parameter, and calculating the value of the loss:<jupyter_code>plot_function(f, 'x', 'x**2')
plt.scatter(-1.5, f(-1.5), color='red');<jupyter_output><empty_output><jupyter_text>Now we look to see what would happen if we increased or decreased our parameter by a little bit—the adjustment. This is simply the slope at a particular point: ![](2020-12-10-Tears-In-Rain/Slope.jpg) We can change our weight by a little in the direction of the slope, calculate our loss and adjustment again, and repeat this a few times. Eventually, we will get to the lowest point on our curve: ![](2020-12-10-Tears-In-Rain/Gradient.jpg) This basic idea goes all the way back to Isaac Newton, who pointed out that we can optimize arbitrary functions in this way  Calculating Gradients *This is a lot of text from this portion of the book, its **really** important and I getting it right is paramount* "The one magic step is the bit where we calculate the gradients. As we mentioned, we use calculus as a performance optimization; it allows us to more quickly calculate whether our loss will go up or down when we adjust our parameters up or down. In other words, the gradients will tell us how much we have to change each weight to make our model better." Thankfully PyTorch is a very powerful auto-differential library. It utilises the [Chain Rule](https://medium.com/machine-learning-and-math/deep-learning-and-chain-rule-of-calculus-80896a1e91f9) to [calculate the derivative](https://pytorch.org/docs/stable/notes/autograd.html) of our functions. First, let's pick a tensor value which we want gradients at:<jupyter_code>xt = tensor(3.).requires_grad_()<jupyter_output><empty_output><jupyter_text>Notice the special method requires_grad_? That's the magical incantation we use to tell PyTorch that we want to calculate gradients with respect to that variable at that value. It is essentially tagging the variable, so PyTorch will remember to keep track of how to compute gradients of the other, direct calculations on it that you will ask for. This API might throw you off if you're coming from math or physics. In those contexts the "gradient" of a function is just another function (i.e., its derivative), so you might expect gradient-related APIs to give you a new function. **But in deep learning, "gradients" usually means the value of a function's derivative at a particular argument value**. The PyTorch API also puts the focus on the argument, not the function you're actually computing the gradients of. It may feel backwards at first, but it's just a different perspective. Now we calculate our function with that value. Notice how PyTorch prints not just the value calculated, but also a note that it has a gradient function it'll be using to calculate our gradients when needed:<jupyter_code>yt = f(xt)
yt<jupyter_output><empty_output><jupyter_text>Calculating the derivative value of our function at this input tensor is simple, we just call the `backward` method.<jupyter_code>yt.backward()<jupyter_output><empty_output><jupyter_text>The "backward" here refers to backpropagation, which is the name given to the process of calculating the derivative of each layer. We can now view the gradients by checking the grad attribute of our tensor:<jupyter_code>xt.grad<jupyter_output><empty_output><jupyter_text>If you remember your high school calculus rules, the derivative of x***2 is 2*x, and we have x=3, so the gradients should be 2*3=6, which is what PyTorch calculated for us!Now we'll repeat the preceding steps, but with a vector argument for our function:<jupyter_code>xt = tensor([3.,4.,10.]).requires_grad_()
xt<jupyter_output><empty_output><jupyter_text>And we'll add sum to our function so it can take a vector (i.e., a rank-1 tensor), and return a scalar (i.e., a rank-0 tensor):<jupyter_code>def f(x): return (x**2).sum()

yt = f(xt)
yt<jupyter_output><empty_output><jupyter_text>Our gradients are 2*xt, as we'd expect!<jupyter_code>yt.backward()
xt.grad<jupyter_output><empty_output><jupyter_text>The gradients only tell us the slope of our function, they don't actually tell us exactly how far to adjust the parameters. But it gives us some idea of how far; if the slope is very large, then that may suggest that we have more adjustments to do, whereas if the slope is very small, that may suggest that we are close to the optimal value.  Stepping With a Learning Rate Because our gradient only shows the slope, or the direction, in which we need to change our parameters. We will need to figure out how much we move in this direction. This is where we introduce the learning rate. The learning rate is often a number between 0.001 and 0.1, although it could be anything.  Once you've picked a learning rate, you can adjust your parameters using this simple function:<jupyter_code>w -= gradient(w) * lr<jupyter_output><empty_output><jupyter_text>If you pick a learning rate that's too low, it can mean having to do a lot of steps ![](2020-12-10-Tears-In-Rain/Small_Step.jpg) But picking a learning rate that's too high is even worse—it can actually result in the loss getting worse ![](2020-12-10-Tears-In-Rain/Large_Step.jpg) If the learning rate is too high, it may also "bounce" around, rather than actually diverging ![](2020-12-10-Tears-In-Rain/Bounce.jpg)  An End-to-End SGD Example Let's work through a practical example from end-to-end. First we will make a float tensor that represents our time variable. Each index is a time in seconds<jupyter_code>time = torch.arange(0,20).float(); time<jupyter_output><empty_output><jupyter_text>Here we create a quadratic function, a function of the form a*(time***2)+(b*time)+c,  and add some noiose to simulate realworld measurments<jupyter_code>speed = torch.randn(20)*3 + 0.75*(time-9.5)**2 + 1
plt.scatter(time,speed);<jupyter_output><empty_output><jupyter_text>We want to distinguish clearly between the function's input (the time when we are measuring the coaster's speed) and its parameters (the values that define which quadratic we're trying). So, let's collect the parameters in one argument and thus separate the input, t, and the parameters, params, in the function's signature:<jupyter_code>def f(t, params):
    a,b,c = params
    return a*(t**2) + (b*t) + c<jupyter_output><empty_output><jupyter_text>We need to determine which loss function we would like to use. For continuous data, it's common to use mean squared error:<jupyter_code>def mse(preds, targets): return ((preds-targets)**2).mean().sqrt()<jupyter_output><empty_output><jupyter_text>Step 1: Initialize the parameters First thing we want to do is create a tensor for our parameters and to tell PyTorch that this tensor `requires_grad_()`.<jupyter_code>params = torch.randn(3).requires_grad_()
#hide
orig_params = params.clone()<jupyter_output><empty_output><jupyter_text>Step 2: Calculate the predictions Lets pass our time tensor and our params into our function<jupyter_code>preds = f(time, params)<jupyter_output><empty_output><jupyter_text>Here we are creating a small matplotlib function to show a comparison between our predictions and our observations<jupyter_code>def show_preds(preds, ax=None):
    if ax is None: ax=plt.subplots()[1]
    ax.scatter(time, speed)
    ax.scatter(time, to_np(preds), color='red')
    ax.set_ylim(-300,100)
show_preds(preds)<jupyter_output><empty_output><jupyter_text>Step 3: Calculate the loss Now we can use our `L2` or `RMSE` function to determine our loss<jupyter_code>loss = mse(preds, speed)
loss<jupyter_output><empty_output><jupyter_text>Step 4: Calculate the gradients By performing `Back Propogation` with our `backward()` method on the loss, we can calculate the slope of our gradient and which direction the we need to move in to improve our loss<jupyter_code>loss.backward()
params.grad<jupyter_output><empty_output><jupyter_text>Here we use a small learning rate and multiply that to our parameter's gradient to make sure we move towards the optimal solution over each iteration<jupyter_code>params.grad * 1e-5
params<jupyter_output><empty_output><jupyter_text>Step 5: Step the weights.<jupyter_code>lr = 1e-5
params.data -= lr * params.grad.data
params.grad = None<jupyter_output><empty_output><jupyter_text>**Understanding this bit depends on remembering recent history. To calculate the gradients we call backward on the loss. But this loss was itself calculated by mse, which in turn took preds as an input, which was calculated using f taking as an input params, which was the object on which we originally called required_grads_—which is the original call that now allows us to call backward on loss. This chain of function calls represents the mathematical composition of functions, which enables PyTorch to use calculus's chain rule under the hood to calculate these gradients.**<jupyter_code>preds = f(time,params)
mse(preds, speed)
show_preds(preds)<jupyter_output><empty_output><jupyter_text>Here we simply wrap our previous steps into a function - Make a prediction- Calculate the loss- Perform back propogation on the loss, see above for details- Apply the learning rate - Zero our gradients- Return our predictions<jupyter_code>def apply_step(params, prn=True):
    preds = f(time, params)
    loss = mse(preds, speed)
    loss.backward()
    params.data -= lr * params.grad.data
    params.grad = None
    if prn: print(loss.item())
    return preds<jupyter_output><empty_output><jupyter_text>Step 6: Repeat the process  Now we simply call our `apply_step` function a number of times, as we can see our loss improves each iteration<jupyter_code>for i in range(10): apply_step(params)
#hide
params = orig_params.detach().requires_grad_()
_,axs = plt.subplots(1,4,figsize=(12,3))
for ax in axs: show_preds(apply_step(params, False), ax)
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Step 7: stop Either we can stop after a certain accuracy or simply after a number of iterations  Summarizing Gradient Descent<jupyter_code>gv('''
init->predict->loss->gradient->step->stop
step->predict[label=repeat]
''')<jupyter_output><empty_output><jupyter_text>![](2020-12-10-Tears-In-Rain/SGD.jpg) - Initialize the weights.- For each image, use these weights to predict whether it appears to be a 3 or a 7.- Based on these predictions, calculate how good the model is (its loss).- Calculate the gradient, which measures for each weight, how changing that weight would change the loss- Step (that is, change) all the weights based on that calculation.- Go back to the step 2, and repeat the process.- Iterate until you decide to stop the training process (for instance, because the model is good enough or you don't want to wait any longer). To summarize, at the beginning, the weights of our model can be random (training from scratch) or come from a pretrained model (transfer learning). In the first case, the output we will get from our inputs won't have anything to do with what we want, and even in the second case, it's very likely the pretrained model won't be very good at the specific task we are targeting. So the model will need to learn better weights.We begin by comparing the outputs the model gives us with our targets (we have labeled data, so we know what result the model should give) using a loss function, which returns a number that we want to make as low as possible by improving our weights. To do this, we take a few data items (such as images) from the training set and feed them to our model. We compare the corresponding targets using our loss function, and the score we get tells us how wrong our predictions were. We then change the weights a little bit to make it slightly better.To find how to change the weights to make the loss a bit better, we use calculus to calculate the gradients. (Actually, we let PyTorch do it for us!) Let's consider an analogy. Imagine you are lost in the mountains with your car parked at the lowest point. To find your way back to it, you might wander in a random direction, but that probably wouldn't help much. Since you know your vehicle is at the lowest point, you would be better off going downhill. By always taking a step in the direction of the steepest downward slope, you should eventually arrive at your destination. We use the magnitude of the gradient (i.e., the steepness of the slope) to tell us how big a step to take; specifically, we multiply the gradient by a number we choose called the learning rate to decide on the step size. We then iterate until we have reached the lowest point, which will be our parking lot, then we can stop.  The MNIST Loss Function Now we have seen how the mothod works on a simple function, we can now put this into practice on our MNIST **3's** and **7's** problem. As we have our dependent variable x: Remember:- **Dependent Variable** == input variables- **Independent Varibale** == output variables Our dependent variable in this is example are the images themselves. Here we concatinate the stacked image tensors of the **3's** and the **7's**. Having the image as Matrix is irrelevant we can use the Pytorch `view` method to reshape every tensor to rank 1,<jupyter_code>train_x = torch.cat([stacked_threes, stacked_sevens]).view(-1, 28*28)<jupyter_output><empty_output><jupyter_text>We also need a label for each of the images, here we can simply create a tensor by combining an array of 1's of length of the number of **3's** and an array of 0's the length of the number of **7's**. We use the PyTorch function `unsqueeze` to transpose the tensor from a vector with 123396 elements into a tensor with 12396 rows and a single column<jupyter_code>train_y = tensor([1]*len(threes) + [0]*len(sevens)).unsqueeze(1)
train_x.shape,train_y.shape<jupyter_output><empty_output><jupyter_text>Now we need to create a dataset, a dataset needs to be able to be indexable, and at that index we expect a tuple (data, label). Here we are using the python `zip` function to take the `train_x` and `train_y` varibles and combine them into a tuple at each index as described<jupyter_code>dset = list(zip(train_x,train_y))
x,y = dset[0]
x.shape,y<jupyter_output><empty_output><jupyter_text>We need to do the same operations above for our validation set as well- Concatinate the images and reshape- Create labels and unsqueeze- Zip data and label into dataset<jupyter_code>valid_x = torch.cat([valid_3_tens, valid_7_tens]).view(-1, 28*28)
valid_y = tensor([1]*len(valid_3_tens) + [0]*len(valid_7_tens)).unsqueeze(1)
valid_dset = list(zip(valid_x,valid_y))<jupyter_output><empty_output><jupyter_text>Here is a simple function that will initialize our parameters with random values. PyTorch `randn` returns a tensor the shape and size of its argument with normalized values between 0 and 1. We can use a variance argument here to scale the random values if necessary, but not in this example. We want to be able to calculate the gradient of this tensor, so we use the `requires_grad` method<jupyter_code>def init_params(size, std=1.0): return (torch.randn(size)*std).requires_grad_()<jupyter_output><empty_output><jupyter_text>We create and initialize our weights and bias variables<jupyter_code>weights = init_params((28*28,1))
bias = init_params(1)<jupyter_output><empty_output><jupyter_text>In neural networks, the w in the equation y=w*x+b is called the weights, and the b is called the bias. Together, the weights and bias make up the parameters. We can now calculate a prediction for one image:<jupyter_code>(train_x[0]*weights.T).sum() + bias<jupyter_output><empty_output><jupyter_text>While we could use a Python for loop to calculate the prediction for each image, that would be very slow. Because Python loops don't run on the GPU, and because Python is a slow language for loops in general, we need to represent as much of the computation in a model as possible using higher-level functions.In this case, there's an extremely convenient mathematical operation that calculates w*x for every row of a matrix—it's called matrix multiplication. ![](2020-12-10-Tears-In-Rain/Matmul.gif) In Python, matrix multiplication is represented with the @ operator. Let's try it:<jupyter_code>def linear1(xb): return xb@weights + bias
preds = linear1(train_x)
preds<jupyter_output><empty_output><jupyter_text>Lets check how good our random initialization is. Here is make a determination that if a prediction is over 0 it's a **3** else its a **7** and we compare it against our labels<jupyter_code>corrects = (preds>0.0).float() == train_y
corrects<jupyter_output><empty_output><jupyter_text>As you can see, and could have predicted, a random initilization is correct roughly 50% of the time.<jupyter_code>corrects.float().mean().item()<jupyter_output><empty_output><jupyter_text>So lets change one of our parameters a little bit and see if that changes our result<jupyter_code>weights[0] *= 1.0001
preds = linear1(train_x)
((preds>0.0).float() == train_y).float().mean().item()<jupyter_output><empty_output><jupyter_text>As we can see above changing one parameter a little has absolutely no affect on the results of our network. What does this mean practically?By changing this pixel by some small amount is not sufficient in itself to change the prediction of an image from a **3** to a **7**Becaue there is no change we cannot calculate a gradient and we cannot make a step to improve our predictions. This is because of the thresholding in our determining our correctness. **In mathematical terms, accuracy is a function that is constant almost everywhere (except at the threshold, 0.5), so its derivative is nil almost everywhere (and infinity at the threshold). This then gives gradients that are 0 or infinite, which are useless for updating the model.** So how do we fix this?<jupyter_code>trgts  = tensor([1,0,1])
prds   = tensor([0.9, 0.4, 0.2])<jupyter_output><empty_output><jupyter_text>Here's a first try at a loss function that measures the distance between predictions and targets:<jupyter_code>def mnist_loss(predictions, targets):
    return torch.where(targets==1, 1-predictions, predictions).mean()<jupyter_output><empty_output><jupyter_text>**Read the Docs: It's important to learn about PyTorch functions like this, because looping over tensors in Python performs at Python speed, not C/CUDA speed! Try running help(torch.where) now to read the docs for this function, or, better still, look it up on the PyTorch documentation site.**<jupyter_code>torch.where(trgts==1, 1-prds, prds)<jupyter_output><empty_output><jupyter_text>You can see that this function returns a lower number when predictions are more accurate, when accurate predictions are more confident (higher absolute values), and when inaccurate predictions are less confident. In PyTorch, we always assume that a lower value of a loss function is better. Since we need a scalar for the final loss, mnist_loss takes the mean of the previous tensor:<jupyter_code>mnist_loss(prds,trgts)<jupyter_output><empty_output><jupyter_text>For instance, if we change our prediction for the one "false" target from 0.2 to 0.8 the loss will go down, indicating that this is a better prediction:<jupyter_code>mnist_loss(tensor([0.9, 0.4, 0.8]),trgts)<jupyter_output><empty_output><jupyter_text>One problem with mnist_loss as currently defined is that it assumes that predictions are always between 0 and 1. We need to ensure, then, that this is actually the case! As it happens, there is a function that does exactly that—let's take a look.  Sigmoid The sigmoid function always outputs a number between 0 and 1. It's defined as follows:<jupyter_code>def sigmoid(x): return 1/(1+torch.exp(-x))
plot_function(torch.sigmoid, title='Sigmoid', min=-4, max=4)<jupyter_output><empty_output><jupyter_text>As you can see, it takes any input value, positive or negative, and smooshes it onto an output value between 0 and 1. It's also a smooth curve that only goes up, which makes it easier for SGD to find meaningful gradients.Let's update mnist_loss to first apply sigmoid to the inputs:<jupyter_code>def mnist_loss(predictions, targets):
    predictions = predictions.sigmoid()
    return torch.where(targets==1, 1-predictions, predictions).mean()<jupyter_output><empty_output><jupyter_text>SGD and Mini-Batches In the context of SGD, "Minibatch" means that the gradient is calculated across the entire batch before updating weights. If you are not using a "minibatch", every training example in a "batch" updates the learning algorithm's parameters independently.<jupyter_code>coll = range(15)
dl = DataLoader(coll, batch_size=5, shuffle=True)
list(dl)
ds = L(enumerate(string.ascii_lowercase))
ds
dl = DataLoader(ds, batch_size=6, shuffle=True)
list(dl)<jupyter_output><empty_output><jupyter_text>Putting It All Together First, let's re-initialize our parameters:<jupyter_code>weights = init_params((28*28,1))
bias = init_params(1)<jupyter_output><empty_output><jupyter_text>A DataLoader can be created from a Dataset and we can set our bacth size here:<jupyter_code>dl = DataLoader(dset, batch_size=256)
xb,yb = first(dl)
xb.shape,yb.shape<jupyter_output><empty_output><jupyter_text>We'll do the same for the validation set:<jupyter_code>valid_dl = DataLoader(valid_dset, batch_size=256)<jupyter_output><empty_output><jupyter_text>Let's create a mini-batch of size 4 for testing:<jupyter_code>batch = train_x[:4]
batch.shape
preds = linear1(batch)
preds
loss = mnist_loss(preds, train_y[:4])
loss<jupyter_output><empty_output><jupyter_text>Now we can calculate the gradients:<jupyter_code>loss.backward()
weights.grad.shape,weights.grad.mean(),bias.grad<jupyter_output><empty_output><jupyter_text>Let's put this into a function:<jupyter_code>def calc_grad(xb, yb, model):
    preds = model(xb)
    loss = mnist_loss(preds, yb)
    loss.backward()
calc_grad(batch, train_y[:4], linear1)
weights.grad.mean(),bias.grad<jupyter_output><empty_output><jupyter_text>But look what happens if we call it twice:<jupyter_code>calc_grad(batch, train_y[:4], linear1)
weights.grad.mean(),bias.grad<jupyter_output><empty_output><jupyter_text>The gradients have changed! The reason for this is that loss.backward actually adds the gradients of loss to any gradients that are currently stored. So, we have to set the current gradients to 0 first:<jupyter_code>weights.grad.zero_()
bias.grad.zero_();<jupyter_output><empty_output><jupyter_text>**Inplace Operations: Methods in PyTorch whose names end in an underscore modify their objects in place. For instance, bias.zero_() sets all elements of the tensor bias to 0.** Our only remaining step is to update the weights and biases based on the gradient and learning rate. When we do so, we have to tell PyTorch not to take the gradient of this step too—otherwise things will get very confusing when we try to compute the derivative at the next batch! **If we assign to the data attribute of a tensor then PyTorch will not take the gradient of that step. Here's our basic training loop for an epoch:**<jupyter_code>def train_epoch(model, lr, params):
    for xb,yb in dl:
        calc_grad(xb, yb, model)
        for p in params:
            p.data -= p.grad*lr
            p.grad.zero_()<jupyter_output><empty_output><jupyter_text>We also want to check how we're doing, by looking at the accuracy of the validation set. To decide if an output represents a 3 or a 7, we can just check whether it's greater than 0. So our accuracy for each item can be calculated (using broadcasting, so no loops!) with:<jupyter_code>(preds>0.0).float() == train_y[:4]<jupyter_output><empty_output><jupyter_text>That gives us this function to calculate our validation accuracy:<jupyter_code>def batch_accuracy(xb, yb):
    preds = xb.sigmoid()
    correct = (preds>0.5) == yb
    return correct.float().mean()
batch_accuracy(linear1(batch), train_y[:4])<jupyter_output><empty_output><jupyter_text>and then put the batches together:<jupyter_code>def validate_epoch(model):
    accs = [batch_accuracy(model(xb), yb) for xb,yb in valid_dl]
    return round(torch.stack(accs).mean().item(), 4)
validate_epoch(linear1)<jupyter_output><empty_output><jupyter_text>That's our starting point. Let's train for one epoch, and see if the accuracy improves:<jupyter_code>lr = 1.
params = weights,bias
train_epoch(linear1, lr, params)
validate_epoch(linear1)<jupyter_output><empty_output><jupyter_text>Then do a few more:<jupyter_code>for i in range(20):
    train_epoch(linear1, lr, params)
    print(validate_epoch(linear1), end=' ')<jupyter_output>0.8265 0.89 0.9183 0.9276 0.9398 0.9467 0.9506 0.9525 0.9559 0.9579 0.9598 0.9608 0.9613 0.9618 0.9633 0.9637 0.9647 0.9657 0.9672 0.9677<jupyter_text>Looking good! We're already about at the same accuracy as our "pixel similarity" approach, and we've created a general-purpose foundation we can build on. Our next step will be to create an object that will handle the SGD step for us. In PyTorch, it's called an optimizer.  Creating an Optimizer PyTorch's `nn.Linear` does the same thing as our init_params and linear together. It contains both the weights and biases in a single class. Here's how we replicate our model from the previous section:<jupyter_code>linear_model = nn.Linear(28*28,1)<jupyter_output><empty_output><jupyter_text>Every PyTorch module knows what parameters it has that can be trained; they are available through the parameters method:<jupyter_code>w,b = linear_model.parameters()
w.shape,b.shape<jupyter_output><empty_output><jupyter_text>Now, let's create an optimizer:<jupyter_code>class BasicOptim:
    def __init__(self,params,lr): self.params,self.lr = list(params),lr

    def step(self, *args, **kwargs):
        for p in self.params: p.data -= p.grad.data * self.lr

    def zero_grad(self, *args, **kwargs):
        for p in self.params: p.grad = None<jupyter_output><empty_output><jupyter_text>We can create our optimizer by passing in the model's parameters:<jupyter_code>opt = BasicOptim(linear_model.parameters(), lr)
def train_epoch(model):
    for xb,yb in dl:
        calc_grad(xb, yb, model)
        opt.step()
        opt.zero_grad()
validate_epoch(linear_model)
def train_model(model, epochs):
    for i in range(epochs):
        train_epoch(model)
        print(validate_epoch(model), end=' ')
train_model(linear_model, 20)<jupyter_output>0.4932 0.7685 0.8554 0.9136 0.9346 0.9482 0.957 0.9634 0.9658 0.9678 0.9697 0.9717 0.9736 0.9746 0.9761 0.977 0.9775 0.9775 0.978 0.9785<jupyter_text>fastai provides the SGD class which, by default, does the same thing as our BasicOptim:<jupyter_code>linear_model = nn.Linear(28*28,1)
opt = SGD(linear_model.parameters(), lr)
train_model(linear_model, 20)<jupyter_output>0.4932 0.8179 0.8496 0.914 0.9346 0.9482 0.957 0.9619 0.9658 0.9673 0.9692 0.9712 0.9741 0.9751 0.9761 0.9775 0.9775 0.978 0.9785 0.979<jupyter_text>fastai also provides Learner.fit, which we can use instead of train_model. To create a Learner we first need to create a DataLoaders, by passing in our training and validation DataLoaders:<jupyter_code>dls = DataLoaders(dl, valid_dl)<jupyter_output><empty_output><jupyter_text>To create a Learner without using an application (such as cnn_learner) we need to pass in all the elements that we've created in this chapter: the DataLoaders, the model, the optimization function (which will be passed the parameters), the loss function, and optionally any metrics to print:<jupyter_code>learn = Learner(dls, nn.Linear(28*28,1), opt_func=SGD,
                loss_func=mnist_loss, metrics=batch_accuracy)
learn.fit(10, lr=lr)<jupyter_output><empty_output><jupyter_text>Adding a Nonlinearity So far we have a general procedure for optimizing the parameters of a function, and we have tried it out on a very boring function: a simple linear classifier. A linear classifier is very constrained in terms of what it can do. To make it a bit more complex (and able to handle more tasks), we need to add something nonlinear between two linear classifiers—this is what gives us a neural network.Here is the entire definition of a basic neural network:<jupyter_code>def simple_net(xb): 
    res = xb@w1 + b1
    res = res.max(tensor(0.0))
    res = res@w2 + b2
    return res<jupyter_output><empty_output><jupyter_text>That's it! All we have in simple_net is two linear classifiers with a max function between them.Here, w1 and w2 are weight tensors, and b1 and b2 are bias tensors; that is, parameters that are initially randomly initialized, just like we did in the previous section:<jupyter_code>w1 = init_params((28*28,30))
b1 = init_params(30)
w2 = init_params((30,1))
b2 = init_params(1)<jupyter_output><empty_output><jupyter_text>The key point about this is that w1 has 30 output activations (which means that w2 must have 30 input activations, so they match). That means that the first layer can construct 30 different features, each representing some different mix of pixels. You can change that 30 to anything you like, to make the model more or less complex.That little function res.max(tensor(0.0)) is called a rectified linear unit, also known as ReLU. We think we can all agree that rectified linear unit sounds pretty fancy and complicated... But actually, there's nothing more to it than res.max(tensor(0.0))—in other words, replace every negative number with a zero. This tiny function is also available in PyTorch as F.relu:<jupyter_code>plot_function(F.relu)<jupyter_output><empty_output><jupyter_text>**Mathematically, we say the composition of two linear functions is another linear function. So, we can stack as many linear classifiers as we want on top of each other, and without nonlinear functions between them, it will just be the same as one linear classifier.**<jupyter_code>simple_net = nn.Sequential(
    nn.Linear(28*28,30),
    nn.ReLU(),
    nn.Linear(30,1)
)<jupyter_output><empty_output><jupyter_text>nn.Sequential creates a module that will call each of the listed layers or functions in turn.nn.ReLU is a PyTorch module that does exactly the same thing as the F.relu function. Most functions that can appear in a model also have identical forms that are modules. Generally, it's just a case of replacing F with nn and changing the capitalization. When using nn.Sequential, PyTorch requires us to use the module version. Since modules are classes, we have to instantiate them, which is why you see nn.ReLU() in this example.Because nn.Sequential is a module, we can get its parameters, which will return a list of all the parameters of all the modules it contains. Let's try it out! As this is a deeper model, we'll use a lower learning rate and a few more epochs.<jupyter_code>learn = Learner(dls, simple_net, opt_func=SGD,
                loss_func=mnist_loss, metrics=batch_accuracy)
learn.fit(40, 0.1)
plt.plot(L(learn.recorder.values).itemgot(2));<jupyter_output><empty_output><jupyter_text>And we can view the final accuracy:<jupyter_code>learn.recorder.values[-1][2]<jupyter_output><empty_output><jupyter_text>At this point we have something that is rather magical:A function that can solve any problem to any level of accuracy (the neural network) given the correct set of parametersA way to find the best set of parameters for any function (stochastic gradient descent)  Going Deeper Here what happens when we train an 18-layer model . . .<jupyter_code>dls = ImageDataLoaders.from_folder(path)
learn = cnn_learner(dls, resnet18, pretrained=False,
                    loss_func=F.cross_entropy, metrics=accuracy)
learn.fit_one_cycle(1, 0.1)<jupyter_output><empty_output><jupyter_text>Almost 100% accuracy!  Jargon Recap - **ReLU :** Function that returns 0 for negative numbers and doesn't change positive numbers.- **Mini-batch :** A small group of inputs and labels gathered together in two arrays. A gradient descent step is updated on this batch (rather than a whole epoch).- **Forward pass :** Applying the model to some input and computing the predictions.- **Loss :** A value that represents how well (or badly) our model is doing.- **Gradient :** The derivative of the loss with respect to some parameter of the model.- **Backward pass :** Computing the gradients of the loss with respect to all model parameters.- **Gradient descent :** Taking a step in the directions opposite to the gradients to make the model parameters a little bit better.- **Learning rate :** The size of the step we take when applying SGD to update the parameters of the model.  Questionnaire **How is a grayscale image represented on a computer? How about a color image?** Images are represented by arrays with pixel values representing the content of the image. For greyscale images, a 2-dimensional array is used with the pixels representing the greyscale values, with a range of 256 integers. A value of 0 would represent white, and a value of 255 represents black, and different shades of greyscale in between. For color images, three color channels (red, green, blue) are typicall used, with a separate 256-range 2D array used for each channel. A pixel value of 0 again represents white, with 255 representing solid red, green, or blue. The three 2-D arrays form a final 3-D array (rank 3 tensor) representing the color image. **How are the files and folders in the `MNIST_SAMPLE` dataset structured? Why?** There are two subfolders, train and valid, the former contains the data for model training, the latter contains the data for validating model performance after each training step. Evaluating the model on the validation set serves two purposes: a) to report a human-interpretable metric such as accuracy (in contrast to the often abstract loss functions used for training), b) to facilitate the detection of overfitting by evaluating the model on a dataset it hasn’t been trained on (in short, an overfitting model performs increasingly well on the training set but decreasingly so on the validation set). Of course, every practicioner could generate their own train/validation-split of the data. Public datasets are usually pre-split to simplifiy comparing results between implementations/publications.Each subfolder has two subsubfolders 3 and 7 which contain the .jpg files for the respective class of images. This is a common way of organizing datasets comprised of pictures. For the full MNIST dataset there are 10 subsubfolders, one for the images for each digit. **Explain how the "pixel similarity" approach to classifying digits works.** In the “pixel similarity” approach, we generate an archetype for each class we want to identify. In our case, we want to distinguish images of 3’s from images of 7’s. We define the archetypical 3 as the pixel-wise mean value of all 3’s in the training set. Analoguously for the 7’s. You can visualize the two archetypes and see that they are in fact blurred versions of the numbers they represent.In order to tell if a previously unseen image is a 3 or a 7, we calculate its distance to the two archetypes (here: mean pixel-wise absolute difference). We say the new image is a 3 if its distance to the archetypical 3 is lower than two the archetypical 7. **What is a list comprehension? Create one now that selects odd numbers from a list and doubles them.** Lists (arrays in other programming languages) are often generated using a for-loop. A list comprehension is a Pythonic way of condensing the creation of a list using a for-loop into a single expression. List comprehensions will also often include if clauses for filtering.<jupyter_code>lst_in = range(10)
lst_out = [2*el for el in lst_in if el%2==1]
# is equivalent to:
lst_out = []
for el in lst_in:
   if el%2==1:
       lst_out.append(2*el)<jupyter_output><empty_output><jupyter_text>**What is a "rank-3 tensor"?** The rank of a tensor is the number of dimensions it has. An easy way to identify the rank is the number of indices you would need to reference a number within a tensor. A scalar can be represented as a tensor of rank 0 (no index), a vector can be represented as a tensor of rank 1 (one index, e.g., v[i]), a matrix can be represented as a tensor of rank 2 (two indices, e.g., a[i,j]), and a tensor of rank 3 is a cuboid or a “stack of matrices” (three indices, e.g., b[i,j,k]). In particular, the rank of a tensor is independent of its shape or dimensionality, e.g., a tensor of shape 2x2x2 and a tensor of shape 3x5x7 both have rank 3.Note that the term “rank” has different meanings in the context of tensors and matrices (where it refers to the number of linearly independent column vectors). **What is the difference between tensor rank and shape? How do you get the rank from the shape?** Rank is the number of axes or dimensions in a tensor; shape is the size of each axis of a tensor. **How do you get the rank from the shape?** The length of a tensor’s shape is its rank.So if we have the images of the 3 folder from the MINST_SAMPLE dataset in a tensor called stacked_threes and we find its shape like this.<jupyter_code>stacked_threes.shape<jupyter_output><empty_output><jupyter_text>torch.Size([6131, 28, 28])We just need to find its length to know its rank. This is done as follows.<jupyter_code>len(stacked_threes.shape)<jupyter_output><empty_output><jupyter_text>3You can also get a tensor’s rank directly with ndim .<jupyter_code>stacked_threes.ndim<jupyter_output><empty_output><jupyter_text>3  **What are RMSE and L1 norm?** Root mean square error (RMSE), also called the L2 norm, and mean absolute difference (MAE), also called the L1 norm, are two commonly used methods of measuring “distance”. Simple differences do not work because some difference are positive and others are negative, canceling each other out. Therefore, a function that focuses on the magnitudes of the differences is needed to properly measure distances. The simplest would be to add the absolute values of the differences, which is what MAE is. RMSE takes the mean of the square (makes everything positive) and then takes the square root (undoes squaring). **How can you apply a calculation on thousands of numbers at once, many thousands of times faster than a Python loop?** As loops are very slow in Python, it is best to represent the operations as array operations rather than looping through individual elements. If this can be done, then using NumPy or PyTorch will be thousands of times faster, as they use underlying C code which is much faster than pure Python. Even better, PyTorch allows you to run operations on GPU, which will have significant speedup if there are parallel operations that can be done. **Create a 3×3 tensor or array containing the numbers from 1 to 9. Double it. Select the bottom-right four numbers.**<jupyter_code>a = torch.Tensor(list(range(1,10))).view(3,3); print(a)<jupyter_output><empty_output><jupyter_text>tensor([[1., 2., 3.],        [4., 5., 6.],        [7., 8., 9.]])<jupyter_code>b = 2*a; 
print(b)<jupyter_output><empty_output><jupyter_text>tensor([[ 2.,  4.,  6.],                     [ 8., 10., 12.],                     [14., 16., 18.]])<jupyter_code>b[1:,1:]<jupyter_output><empty_output><jupyter_text>tensor([[10., 12.],                    [16., 18.]]) **What is broadcasting?** Scientific/numerical Python packages like NumPy and PyTorch will often implement broadcasting that often makes code easier to write. In the case of PyTorch, tensors with smaller rank are expanded to have the same size as the larger rank tensor. In this way, operations can be performed between tensors with different rank. **Are metrics generally calculated using the training set, or the validation set? Why?** Metrics are generally calculated on a validation set. As the validation set is unseen data for the model, evaluating the metrics on the validation set is better in order to determine if there is any overfitting and how well the model might generalize if given similar data. **What is SGD?** SGD, or stochastic gradient descent, is an optimization algorithm. Specifically, SGD is an algorithm that will update the parameters of a model in order to minimize a given loss function that was evaluated on the predictions and target. The key idea behind SGD (and many optimization algorithms, for that matter) is that the gradient of the loss function provides an indication of how that loss function changes in the parameter space, which we can use to determine how best to update the parameters in order to minimize the loss function. This is what SGD does. **Why does SGD use mini-batches?** We need to calculate our loss function (and our gradient) on one or more data points. We cannot calculate on the whole datasets due to compute limitations and time constraints. If we iterated through each data point, however, the gradient will be unstable and imprecise, and is not suitable for training. As a compromise, we calculate the average loss for a small subset of the dataset at a time. This subset is called a mini-batch. Using mini-batches are also more computationally efficient than single items on a GPU. **What are the seven steps in SGD for machine learning?** - Initialize the weights.- For each image, use these weights to predict whether it appears to be a 3 or a 7.- Based on these predictions, calculate how good the model is (its loss).- Calculate the gradient, which measures for each weight, how changing that weight would change the loss- Step (that is, change) all the weights based on that calculation.- Go back to the step 2, and repeat the process.- Iterate until you decide to stop the training process (for instance, because the model is good enough or you don't want to wait any longer). **How do we initialize the weights in a model?** Random weights work pretty well. **What is "loss"?** The loss function will return a value based on the given predictions and targets, where lower values correspond to better model predictions. **Why can't we always use a high learning rate?** The loss may “bounce” around (oscillate) or even diverge, as the optimizer is taking steps that are too large, and updating the parameters faster than it should be. **What is a "gradient"?** The gradients tell us how much we have to change each weight to make our model better. It is essentially a measure of how the loss function changes with changes of the weights of the model (the derivative). **Do you need to know how to calculate gradients yourself?** Manual calculation of the gradients are not required, as deep learning libraries will automatically calculate the gradients for you. This feature is known as automatic differentiation. In PyTorch, if requires_grad=True, the gradients can be returned by calling the backward method: a.backward() **Why can't we use accuracy as a loss function?** A loss function needs to change as the weights are being adjusted. Accuracy only changes if the predictions of the model change. So if there are slight changes to the model that, say, improves confidence in a prediction, but does not change the prediction, the accuracy will still not change. Therefore, the gradients will be zero everywhere except when the actual predictions change. The model therefore cannot learn from the gradients equal to zero, and the model’s weights will not update and will not train. A good loss function gives a slightly better loss when the model gives slightly better predictions. Slightly better predictions mean if the model is more confident about the correct prediction. For example, predicting 0.9 vs 0.7 for probability that a MNIST image is a 3 would be slightly better prediction. The loss function needs to reflect that. **Draw the sigmoid function. What is special about its shape?** ![](2020-12-10-Tears-In-Rain/Sigmoid.jpg) Sigmoid function is a smooth curve that squishes all values into values between 0 and 1. Most loss functions assume that the model is outputting some form of a probability or confidence level between 0 and 1 so we use a sigmoid function at the end of the model in order to do this. **What is the difference between a loss function and a metric?** The key difference is that metrics drive human understanding and losses drive automated learning. In order for loss to be useful for training, it needs to have a meaningful derivative. Many metrics, like accuracy are not like that. Metrics instead are the numbers that humans care about, that reflect the performance of the model. **What is the function to calculate new weights using a learning rate?** The optimizer step function **What does the `DataLoader` class do?** The DataLoader class can take any Python collection and turn it into an iterator over many batches. **Write pseudocode showing the basic steps taken in each epoch for SGD.**<jupyter_code>for x,y in dl:
   pred = model(x)
   loss = loss_func(pred, y)
   loss.backward()
   parameters -= parameters.grad * lr<jupyter_output><empty_output><jupyter_text>**Create a function that, if passed two arguments `[1,2,3,4]` and `'abcd'`, returns `[(1, 'a'), (2, 'b'), (3, 'c'), (4, 'd')]`. What is special about that output data structure?**<jupyter_code>def func(a,b): return list(zip(a,b))<jupyter_output><empty_output><jupyter_text>This data structure is useful for machine learning models when you need lists of tuples where each tuple would contain input data and a label. **What does `view` do in PyTorch?** It changes the shape of a Tensor without changing its contents. **What are the "bias" parameters in a neural network? Why do we need them?** Without the bias parameters, if the input is zero, the output will always be zero. Therefore, using bias parameters adds additional flexibility to the model. **What does the `@` operator do in Python?** This is the matrix multiplication operator. **What does the `backward` method do?** This method returns the current gradients. **Why do we have to zero the gradients?** PyTorch will add the gradients of a variable to any previously stored gradients. If the training loop function is called multiple times, without zeroing the gradients, the gradient of current loss would be added to the previously stored gradient value. **What information do we have to pass to `Learner`?** We need to pass in the DataLoaders, the model, the optimization function, the loss function, and optionally any metrics to print. **Show Python or pseudocode for the basic steps of a training loop.**<jupyter_code>def train_epoch(model, lr, params):
    for xb,yb in dl:
        calc_grad(xb, yb, model)
        for p in params:
            p.data -= p.grad*lr
            p.grad.zero_()

for i in range(20):
    train_epoch(model, lr, params)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07365530090101666, 'fraction_numerical': 0.017185166150040954, 'mean_word_length': 5.150745826336067, 'pattern_counts': {'":': 0, '<': 432, '<?xml version=': 0, '>': 448, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '108', 'dir': 'jupyter-structured-clean-dedup', 'id': '2748', 'license': 'Apache-2.0', 'n_tokens_mistral': 18292, 'n_tokens_neox': 17274, 'n_words': 8424, 'path': '_notebooks/2020-12-10-Tears-In-Rain.ipynb', 'repo_name': 'bailey-deep-learning/im_sorry_dave', 'size': '62274'}