<jupyter_start><jupyter_text>Differentiation in Autograd<jupyter_code>import torch

a = torch.tensor([2., 3.], requires_grad=True)
b = torch.tensor([6., 4.], requires_grad=True)
Q = 3*a**3 - b**2
external_grad = torch.tensor([1., 1.])
Q.backward(gradient=external_grad)
# check if collected gradients are correct
print(9*a**2 == a.grad)
print(-2*b == b.grad)
a.grad<jupyter_output><empty_output><jupyter_text>he output tensor of an operation will require gradients even if only a single input tensor has requires_grad=True<jupyter_code>x = torch.rand(5, 5)
y = torch.rand(5, 5)
z = torch.rand((5, 5), requires_grad=True)

a = x + y
print(f"Does `a` require gradients? : {a.requires_grad}")
b = x + z
print(f"Does `b` require gradients?: {b.requires_grad}")<jupyter_output>Does `a` require gradients? : False
Does `b` require gradients?: True<jupyter_text>In a NN, parameters that don’t compute gradients are usually called **frozen parameters**. It is useful to “freeze” part of your model if you know in advance that you won’t need the gradients of those parameters (this offers some performance benefits by reducing autograd computations).Another common usecase where exclusion from the DAG is important is for [finetuning a pretrained network](https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html)<jupyter_code>from torch import nn, optim

model = torchvision.models.resnet18(pretrained=True)

# Freeze all the parameters in the network
for param in model.parameters():
    param.requires_grad = False
model.fc
model.fc = nn.Linear(512, 10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1141609482220836, 'fraction_numerical': 0.015595757953836557, 'mean_word_length': 6.035087719298246, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '478405', 'license': 'MIT', 'n_tokens_mistral': 547, 'n_tokens_neox': 519, 'n_words': 163, 'path': 'autograd.ipynb', 'repo_name': 'jb2020-super/pytorch-tester', 'size': '1603'}