<jupyter_start><jupyter_text>Automatic Differentiation with AutoGradWhen training neural networks, the most frequently used algorithm is **back propagation**. In this algorithm, parameters (model weights) are adjusted according to the **gradient** of the loss function with respect to the given parameter.To compute those gradients, PyTorch has a built-in mechanism called **AutoGrad**. It supports automatic computation of gradient for any computational graph. Tensors, Functions and Computational graphConsider the simplest one-layer neural network, with input `x`, parameters `w` and `b`, and some loss function. It can be defined in PyTorch in the following manner:<jupyter_code>import torch
x = torch.ones(5) # input tensor
y = torch.zeros(3) # expected output
w = torch.randn(5,3,requires_grad=True)
b = torch.randn(3,requires_grad=True)
z = torch.matmul(x,w)+b
loss = torch.nn.functional.binary_cross_entropy_with_logits(z,y)<jupyter_output><empty_output><jupyter_text>In this code defines the following **computational graph**:In this network, `w` and `b` are **parameters**, which we need to optimize. Thus, we need to be able to compute the gradients of loss function with respect to those variables. In orded to do that, we set the `requires_grad` property of those tensors.> **Note:** You can set the value of `requires_grad` when creating a tensor, or later by using `x.requires_grad_(True)` method.A function that we apply to tensors to construct computational graph is in fact an object of class `Function`. This object knows how to compute the function in the *forward* direction, and also how to compute it's derivative during the *backward propagation* step. A reference to the backward propagation function is stored in `grad_fn` property of a tensor. You can find more information of `Function` [in documentation](https://pytorch.org/docs/stable/autograd.htmlfunction).<jupyter_code>print(z.grad_fn,loss.grad_fn,sep='\n')<jupyter_output><AddBackward0 object at 0x7f7645cb7310>
<BinaryCrossEntropyWithLogitsBackward object at 0x7f7645cb7790><jupyter_text>Computing GradientsTo optimize weights of parameters in the neural network, we need to compute the defivatives of our loss function with respect to parameters, namely, we need $\frac{\partial loss}{\partial w}$ and $\frac{\partial loss}{\partial b}$ under some fixed values of `x` and `y`. To compute those derivatives, we call `loss.backward()`, and then retrieve the values from `w.grad` and `b.grad`:<jupyter_code>loss.backward()
print(w.grad)
print(b.grad)<jupyter_output>tensor([[0.2136, 0.3118, 0.0472],
        [0.2136, 0.3118, 0.0472],
        [0.2136, 0.3118, 0.0472],
        [0.2136, 0.3118, 0.0472],
        [0.2136, 0.3118, 0.0472]])
tensor([0.2136, 0.3118, 0.0472])<jupyter_text>**Notes:** * We can only obtain the `grad` properties for the leaf nodes of the computational graph, which have `requires_grad` property set to `True`. For all other nodes in our graph gradients will not be available.* We can only perform gradient calculations using `backward` once on a given graph, for performance reasons. If we need to do several `backward` calls on the same graph, we need to pass `retain_graph=True` to the `backward` call.  Tensor Gradients and Jacobian ProductsIn many cases, we have a scalar loss function, and we need to compute the gradient with respect to some parameters. However, there are cases when the output function is an arbitrary tensor. In this case, PyTorch allows you to compute so-called **Jacobian product**, and not the actual gradient.For a vector function $\vec{y}=f(\vec{x})$, where $\vec{x}=\langle x_1,\dots,x_n\rangle$ and $\vec{y}=\langle y_1,\dots,y_m\rangle$, a gradient of $\vec{y}$ with respect to $\vec{x}$ is given by **Jacobian matrix**:$$\begin{align}J=\left(\begin{array}{ccc}   \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{1}}{\partial x_{n}}\\   \vdots & \ddots & \vdots\\   \frac{\partial y_{m}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}}   \end{array}\right)\end{align}$$Instead of computing the Jacobian matrix itself, PyTorch allows you to compute **Jacobian Product** $v^T\cdot J$ for a given input vector $v=(v_1 \dots v_m)$. This is achieved by calling `backward` with $v$ as an argument. The size of $v$ should be the same as the size of the original tensor, with respect to which we want to compute the product:<jupyter_code>inp = torch.eye(5,requires_grad=True)
out = (inp+1).pow(2)
out.backward(torch.ones_like(inp),retain_graph=True)
print("First call\n",inp.grad)
out.backward(torch.ones_like(inp),retain_graph=True)
print("\nSecond call\n",inp.grad)
inp.grad.zero_()
out.backward(torch.ones_like(inp),retain_graph=True)
print("\nCall after zeroing gradients\n",inp.grad)<jupyter_output>First call
 tensor([[4., 2., 2., 2., 2.],
        [2., 4., 2., 2., 2.],
        [2., 2., 4., 2., 2.],
        [2., 2., 2., 4., 2.],
        [2., 2., 2., 2., 4.]])

Second call
 tensor([[8., 4., 4., 4., 4.],
        [4., 8., 4., 4., 4.],
        [4., 4., 8., 4., 4.],
        [4., 4., 4., 8., 4.],
        [4., 4., 4., 4., 8.]])

Call after zeroing gradients
 tensor([[4., 2., 2., 2., 2.],
        [2., 4., 2., 2., 2.],
        [2., 2., 4., 2., 2.],
        [2., 2., 2., 4., 2.],
        [2., 2., 2., 2., 4.]])<jupyter_text>Notice that when we call `backward` for the second time with the same argument, the value of the gradient is different. This happens because when doing `backward` propagation, PyTorch **accumulates the gradients**, i.e. the value of computed gradients is added to the `grad` property of all leaf nodes of computational graph. If you want to compute the proper gradients, you need to zero out the `grad` property before. In real-life training an *optimizer* helps us to do this.**Note:** Previously we were calling `backward()` function without parameters. This is essentially equivalent to calling `backward(torch.tensor(1.0))`, which is a useful way to compute the gradients in case of a scalar-valued function, such as loss during neural network training.  Disabling Gradient TrackingBy default, all tensors with `requires_grad=True` are tracking their computational history and support gradient computation. However, there are some cases when we do not need to do that, for example, when we have trained the model and just want to apply it to some input data, i.e. we only want to do *forward* computations through the network. We can stop tracking computations by surrounding our computation code with `with torch.no_grad()` block:<jupyter_code>z = torch.matmul(x,w)+b
print(z.requires_grad)

with torch.no_grad():
    z = torch.matmul(x,w)+b
print(z.requires_grad)<jupyter_output>True
False<jupyter_text>Another way to achieve the same result is to use the `detach()` method on the tensor:<jupyter_code>z = torch.matmul(x,w)+b
z_det = z.detach()
print(z_det.requires_grad)<jupyter_output>False<jupyter_text>All forward-pass computations on tensors that do not track gradients would be more efficient.  Example of Gradient DescentLet's use the AutoGrad functionality to minimize a simple function of two variables $f(x_1,x_2)=(x_1-3)^2+(x_2+2)^2$. We will use the `x` tensor to represent the coordinates of a point. To do the gradient descent, we start with some initial value $x^{(0)}=(0,0)$, and compute each consecutive step using:$$x^{(n+1)} = x^{(n)} - \eta\nabla f$$Here $\eta$ is so-called **learning rate** (we will call it `lr` in our code), and $\nabla f = (\frac{\partial f}{\partial x_1},\frac{\partial f}{\partial x_2})$ is the gradient of $f$.We will start by defining the initial value of `x` and the function `f`:<jupyter_code>x = torch.zeros(2,requires_grad=True)
f = lambda x : (x-torch.tensor([3,-2])).pow(2).sum()
lr = 0.1<jupyter_output><empty_output><jupyter_text>For the gradient descent, let's do 15 iterations. On each iteration, we will update the coordinate tensor `x` and print its coordinates to make sure that we are approaching the minimum:<jupyter_code>for i in range(15):
    y = f(x)
    y.backward()
    gr = x.grad
    x.data.add_(-lr*gr)
    x.grad.zero_()
    print("Step {}: x[0]={}, x[1]={}".format(i,x[0],x[1]))<jupyter_output>Step 0: x[0]=0.6000000238418579, x[1]=-0.4000000059604645
Step 1: x[0]=1.0800000429153442, x[1]=-0.7200000286102295
Step 2: x[0]=1.4639999866485596, x[1]=-0.9760000705718994
Step 3: x[0]=1.7711999416351318, x[1]=-1.1808000802993774
Step 4: x[0]=2.0169599056243896, x[1]=-1.3446400165557861
Step 5: x[0]=2.2135679721832275, x[1]=-1.4757120609283447
Step 6: x[0]=2.370854377746582, x[1]=-1.5805696249008179
Step 7: x[0]=2.4966835975646973, x[1]=-1.6644556522369385
Step 8: x[0]=2.597346782684326, x[1]=-1.7315645217895508
Step 9: x[0]=2.677877426147461, x[1]=-1.7852516174316406
Step 10: x[0]=2.7423019409179688, x[1]=-1.8282012939453125
Step 11: x[0]=2.793841600418091, x[1]=-1.8625609874725342
Step 12: x[0]=2.835073232650757, x[1]=-1.8900487422943115
Step 13: x[0]=2.868058681488037, x[1]=-1.912039041519165
Step 14: x[0]=2.894446849822998, x[1]=-1.929631233215332
{'fraction_non_alphanumeric': 0.13362116683272446, 'fraction_numerical': 0.08657146020148344, 'mean_word_length': 4.69609079445145, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '467661', 'license': 'MIT', 'n_tokens_mistral': 3453, 'n_tokens_neox': 2965, 'n_words': 1069, 'path': 'autograd.ipynb', 'repo_name': 'btholt/pytorchblitz', 'size': '9033'}