<jupyter_start><jupyter_text>RoadMap 11 - Torch NN Layers - Loss Functions    1.   torch.nn.L1Loss                     - L1 Loss Function    2.   torch.nn.MSELoss                    - Mean Square Error Loss Function    3.   torch.nn.CrossEntropyLoss           - Cross entropy loss function    4.   torch.nn.NLLLoss                    - Negative Log Likelihood loss function    5.   torch.nn.PoissonNLLLoss             - Poisson Negative Log Likelihood loss function    6.   torch.nn.nn.KLDivLoss               - Kullback Leiber Divergence loss function    7.   torch.nn.BCELoss                    - Binary cross entropy loss function    8.   torch.BCEWithLogitsLoss             - Binary cross entropy loss with logits function    9.   torch.nn.MarginRankingLoss          - Margin ranking loss function    10.  torch.nn.HingeEmbeddingLoss         - Hinge Embedding Loss function    11.  torch.nn.MultiLabelMarginLoss       - Multi Label Margin loss function    12.  torch.nn.SmoothL1Loss               - Smooth L1 Loss function    13.  torch.nn.MultiLabelSoftMarginLoss   - Multi Label Soft Margin Loss function    14.  torch.nn.CosineEmbeddingLoss        - Cosine Embedding loss function    15.  torch.nn.MultiMarginLoss            - Multi Margin loss function    16.  torch.nn.TripletMarginLoss          - Triplet Margin Loss Function<jupyter_code>import os
import sys
import torch
import numpy as np
import torch.nn as nn
from torchvision import transforms, datasets
from PIL import Image
import cv2
import matplotlib.pyplot as plt
import torchvision

# FUNCTIONAL modules - Implementing each module as functions
import torch.nn.functional as F<jupyter_output><empty_output><jupyter_text>Extra Blog Layers1. https://medium.com/udacity-pytorch-challengers/a-brief-overview-of-loss-functions-in-pytorch-c0ddb78068f7    2. https://machinelearningmastery.com/loss-and-loss-functions-for-training-deep-learning-neural-networks/3. https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/<jupyter_code># Input
image_name = "dog.jpg"
image_pil = Image.open(image_name)

transform = transforms.Compose([transforms.ToTensor()])
image = transform(image_pil).float()
image_nchw = torch.unsqueeze(image, 0)
print(image_nchw.size())
plt.imshow(image_pil)<jupyter_output>torch.Size([1, 3, 576, 768])<jupyter_text>L1 loss Function1. nn.L1Loss - Creates a criterion that measures the mean absolute value of the element-wise difference between input x and target y            - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True            - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True            - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’    Function    $$\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quadl_n = \left| x_n - y_n \right|$$<jupyter_code>print("Module implementation")
loss = nn.L1Loss()
input_data = torch.randn(10, requires_grad=True)
target_data = torch.randn(10)
output_loss = loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("Output Loss = ", output_loss)
print("\n")

print("Functional implementation")
output_loss = F.l1_loss(input_data, target_data, reduction='elementwise_mean')
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module implementation
Input Data =  tensor([ 1.6689,  0.9617, -0.8708,  1.7215, -1.7089, -1.4635,  0.6810,  0.9599,
         0.8776, -0.3141], requires_grad=True)
Output Loss =  tensor(1.4486, grad_fn=<L1LossBackward>)


Functional implementation
Output Loss =  tensor(1.4486, grad_fn=<L1LossBackward>)<jupyter_text>Mean Square Error Loss2. nn.MSELoss - Creates a criterion that measures the mean squared error between n elements in the input x and target y.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’                    Function:$$\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quadl_n = \left( x_n - y_n \right)^2$$<jupyter_code>print("Module implementation")
loss = nn.MSELoss()
input_data = torch.randn(10, requires_grad=True)
target_data = torch.randn(10)
output_loss = loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("Output Loss = ", output_loss)
print("\n")

print("Functional implementation")
output_loss = F.mse_loss(input_data, target_data, reduction='elementwise_mean')
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module implementation
Input Data =  tensor([-0.6542, -1.0031, -0.0445, -0.2803, -0.3421, -1.1031,  0.9784,  0.1492,
         0.8774, -2.4233], requires_grad=True)
Output Loss =  tensor(1.7418, grad_fn=<MseLossBackward>)


Functional implementation
Output Loss =  tensor(1.7418, grad_fn=<MseLossBackward>)<jupyter_text>Cross Entropy Loss3. nn.CrossEntropyLoss - This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class. It is useful when training a classification problem with C classes. If provided, the optional argument weight should be a 1D Tensor assigning weight to each of the classes. This is particularly useful when you have an unbalanced training set.    - weight (Tensor, optional) – a manual rescaling weight given to each class. If given, has to be a Tensor of size C    - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True    - ignore_index (int, optional) – Specifies a target value that is ignored and does not contribute to the input gradient. When size_average is True, the loss is averaged over non-ignored targets.    - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True    - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’          Function:$$ \text{loss}(x, class) = weight[class] \left(-x[class] + \log\left(\sum_j \exp(x[j])\right)\right) $$<jupyter_code>print("Module Implementation")
loss = nn.CrossEntropyLoss()
softmax = nn.Softmax()
input_data = torch.randn(3, 4, requires_grad=True)
input_data = softmax(input_data)
target_data = torch.tensor([0, 1, 2])
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")


print("Functional implementation")
output_loss = F.cross_entropy(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module Implementation
Input Data =  tensor([[0.0514, 0.0519, 0.6677, 0.2290],
        [0.1202, 0.2288, 0.3351, 0.3158],
        [0.1132, 0.3875, 0.4544, 0.0449]], grad_fn=<SoftmaxBackward>)


Target Data =  tensor([0, 1, 2])


Output Loss =  tensor(1.4089, grad_fn=<NllLossBackward>)




Functional implementation
Output Loss =  tensor(1.4089, grad_fn=<NllLossBackward>)<jupyter_text>Negative log likelihood loss4. nn.NLLLoss - The negative log likelihood loss. It is useful to train a classification problem with C classes. If provided, the optional argument weight should be a 1D Tensor assigning weight to each of the classes. This is particularly useful when you have an unbalanced training set.    - weight (Tensor, optional) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.    - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True    - ignore_index (int, optional) – Specifies a target value that is ignored and does not contribute to the input gradient. When size_average is True, the loss is averaged over non-ignored targets.    - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True    - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’    Function:$$\begin{split}\ell(x, y) = \begin{cases}    \sum_{n=1}^N \frac{1}{\sum_{n=1}^N w_{y_n}} l_n, & \text{if}\;    \text{size_average} = \text{True},\\    \sum_{n=1}^N l_n,  & \text{if}\;    \text{size_average} = \text{False}.\end{cases}\end{split}$$<jupyter_code>print("Module Implementation")
m = nn.LogSoftmax()
loss = nn.NLLLoss()
# input is of size N x C = 3 x 5
input_data = torch.randn(3, 10, requires_grad=True)
input_data = m(input_data)
# each element in target has to have 0 <= value < C
target_data = torch.tensor([1, 0, 4])
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)


print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")


print("Functional implementation")
output_loss = F.nll_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module Implementation
Input Data =  tensor([[-1.5806, -1.6066, -3.2434, -3.1549, -2.9530, -2.7948, -1.4620, -2.3547,
         -3.3739, -3.2791],
        [-1.8911, -4.4236, -2.4025, -2.9850, -2.2202, -3.2712, -2.4602, -2.8455,
         -2.5018, -1.1268],
        [-2.0087, -4.0166, -2.9455, -2.8539, -3.3092, -2.3496, -2.0006, -1.1939,
         -3.0565, -2.1174]], grad_fn=<LogSoftmaxBackward>)


Target Data =  tensor([1, 0, 4])


Output Loss =  tensor(2.2690, grad_fn=<NllLossBackward>)




Functional implementation
Output Loss =  tensor(2.2690, grad_fn=<NllLossBackward>)<jupyter_text>Poisson Negative log likelihood loss5. nn.PoissonNLLLoss - Negative log likelihood loss with Poisson distribution of target.        - log_input (bool, optional) – if True the loss is computed as exp(input)−target∗input, if False the loss is input−target∗log(input+eps)        - full (bool, optional) – whether to compute full loss, i. e. to add the Stirling approximation term target∗log(target)−target+0.5∗log(2πtarget).        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - eps (float, optional) – Small value to avoid evaluation of log(0), when log_input == False. Default: 1e-8        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’            Function:$$\begin{align}\begin{aligned}\text{target} \sim \mathrm{Poisson}(\text{input})\\\text{loss}(\text{input}, \text{target}) = \text{input} - \text{target} * \log(\text{input})                            + \log(\text{target!})\end{aligned}\end{align}$$<jupyter_code>print("Module Implementation")
loss = nn.PoissonNLLLoss()
input_data = torch.randn(5, 2, requires_grad=True)
target_data = torch.randn(5, 2)
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)


print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.poisson_nll_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module Implementation
Input Data =  tensor([[-0.3775,  1.5565],
        [-0.2749,  0.6706],
        [ 0.4687,  1.6879],
        [ 0.6063,  0.6137],
        [-0.3487, -0.9277]], requires_grad=True)


Target Data =  tensor([[ 0.6550,  0.6313],
        [-1.0439,  0.6372],
        [-1.3411, -1.0727],
        [-0.1483,  1.3046],
        [ 1.8552, -1.4957]])


Output Loss =  tensor(1.9469, grad_fn=<MeanBackward0>)




Functional implementation
Output Loss =  tensor(1.9469, grad_fn=<MeanBackward0>)<jupyter_text>Kullback Leiber Divergence loss6. nn.KLDivLoss - KL divergence is a useful distance measure for continuous distributions and is often useful when performing direct regression over the space of (discretely sampled) continuous output distributions.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’<jupyter_code>print("Module Implementation")
loss = nn.KLDivLoss(size_average=False)
log_softmax = nn.LogSoftmax(1)
softmax = nn.Softmax(1)

batch_size = 2

input_data = torch.randn(batch_size, 3, requires_grad=True)
log_input_data = log_softmax(input_data)

target_data = torch.randn(batch_size, 3, requires_grad=True)
softmax_target_data = softmax(target_data)

output_loss = loss(log_input_data, softmax_target_data) / batch_size
output_loss.backward(retain_graph=True)

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.kl_div(log_input_data, softmax_target_data, size_average=False) / batch_size
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module Implementation
Input Data =  tensor([[ 1.0913,  0.8709, -0.3859],
        [-0.8672,  0.2453,  0.2501]], requires_grad=True)


Target Data =  tensor([[ 2.6353e-01, -2.5150e-01,  4.9396e-04],
        [ 4.4976e-01, -5.5277e-01,  5.8091e-01]], requires_grad=True)


Output Loss =  tensor(0.2249, grad_fn=<DivBackward0>)




Functional implementation
Output Loss =  tensor(0.2249, grad_fn=<DivBackward0>)<jupyter_text>Binary Cross Entripy Loss7. nn.BCELoss - Creates a criterion that measures the Binary Cross Entropy between the target and the output        - weight (Tensor, optional) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size “nbatch”.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’                   Function:   $$\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quadl_n = - w_n \left[ y_n \cdot \log x_n + (1 - y_n) \cdot \log (1 - x_n) \right]$$<jupyter_code>print("Module Implementation")
loss = nn.BCELoss()
sigmoid = nn.Sigmoid()

input_data = torch.randn(5, requires_grad=True)
sigmoid_input_data = sigmoid(input_data) 

target_data = torch.empty(5).random_(2) # Binary Target

output_loss = loss(sigmoid_input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.binary_cross_entropy(sigmoid_input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module Implementation
Input Data =  tensor([-1.4087, -1.0918,  0.0857,  0.7139,  0.2992], requires_grad=True)


Target Data =  tensor([0., 1., 1., 0., 1.])


Output Loss =  tensor(0.7837, grad_fn=<BinaryCrossEntropyBackward>)




Functional implementation
Output Loss =  tensor(0.7837, grad_fn=<BinaryCrossEntropyBackward>)<jupyter_text>Binary Cross Entropy with Logits Loss8. nn.BCEWithLogitsLoss - This loss combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.        - weight (Tensor, optional) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size “nbatch”.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’        - pos_weight – a weight of positive examples. Must be a vector with length equal to the number of classes.    Function    $$\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quadl_n = - w_n \left[ t_n \cdot \log \sigma(x_n)+ (1 - t_n) \cdot \log (1 - \sigma(x_n)) \right]$$<jupyter_code>print("Module implementation")
loss = nn.BCEWithLogitsLoss()
sigmoid = nn.Sigmoid()

input_data = torch.randn(5, requires_grad=True)
sigmoid_input_data = sigmoid(input_data) 

target_data = torch.empty(5).random_(2) # Binary Target

output_loss = loss(sigmoid_input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.binary_cross_entropy_with_logits(sigmoid_input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module implementation
Input Data =  tensor([-0.2327, -1.6082, -0.4721, -1.4554,  0.7109], requires_grad=True)


Target Data =  tensor([0., 0., 0., 0., 0.])


Output Loss =  tensor(0.8996, grad_fn=<BinaryCrossEntropyWithLogitsBackward>)




Functional implementation
Output Loss =  tensor(0.8996, grad_fn=<BinaryCrossEntropyWithLogitsBackward>)<jupyter_text>Margin Ranking Loss9. nn.MarginRankingLoss - Creates a criterion that measures the loss given inputs x1, x2, two 1D mini-batch Tensor's, and a label 1D mini-batch tensor 'y with values (1 or -1). If y == 1 then it assumed the first input should be ranked higher (have a larger value) than the second input, and vice-versa for y == -1.        - margin (float, optional) – Has a default value of 0.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’    Function:$$\text{loss}(x, y) = \max(0, -y * (x1 - x2) + \text{margin})$$<jupyter_code>print("Module implementation")
loss = nn.MarginRankingLoss()
input_data1 = torch.randn(3, requires_grad=True)
input_data2 = torch.randn(3, requires_grad=True)
target_data = torch.tensor([-1., 1., 1.])
output_loss = loss(input_data1, input_data2, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data1 = ", input_data1)
print("\n")
print("Input Data2 = ", input_data2)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.margin_ranking_loss(input_data1, input_data2, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module implementation
Input Data1 =  tensor([-0.0064, -0.0055, -0.2032], requires_grad=True)


Input Data2 =  tensor([-0.1289, -0.2205,  0.9442], requires_grad=True)


Target Data =  tensor([-1.,  1.,  1.])


Output Loss =  tensor(0.4233, grad_fn=<MeanBackward0>)




Functional implementation
Output Loss =  tensor(0.4233, grad_fn=<MeanBackward0>)<jupyter_text>Hinge Embedding Loss10. nn.HingeEmbeddingLoss - Measures the loss given an input tensor x and a labels tensor y containing values (1 or -1). This is usually used for measuring whether two inputs are similar or dissimilar, e.g. using the L1 pairwise distance as x, and is typically used for learning nonlinear embeddings or semi-supervised learning.        - margin (float, optional) – Has a default value of 1.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’                    Function:$$\begin{split}l_n = \begin{cases}    x_n, & \text{if}\; y_n = 1,\\    \max \{0, \Delta - x_n\}, & \text{if}\; y_n = -1,\end{cases}\end{split}$$<jupyter_code>print("Module implementation")
loss = nn.HingeEmbeddingLoss()
input_data = torch.randn(3, requires_grad=True)
target_data = torch.tensor([-1., 1., 1.])
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.hinge_embedding_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module implementation
Input Data =  tensor([ 0.0038, -1.3820,  0.4059], requires_grad=True)


Target Data =  tensor([-1.,  1.,  1.])


Output Loss =  tensor(0.0067, grad_fn=<MeanBackward0>)




Functional implementation
Output Loss =  tensor(0.0067, grad_fn=<MeanBackward0>)<jupyter_text>Multi Label Margin Loss11. nn.MultiLabelMarginLoss - Creates a criterion that optimizes a multi-class multi-classification hinge loss (margin-based loss) between input x (a 2D mini-batch Tensor) and output y (which is a 2D Tensor of target class indices). For each sample in the mini-batch        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’    Function:    $$\text{loss}(x, y) = \sum_{ij}\frac{\max(0, 1 - (x[y[j]] - x[i]))}{\text{x.size}(0)}$$<jupyter_code>print("Module implementation")
loss = nn.MultiLabelMarginLoss()
softmax = nn.Softmax()

input_data = torch.randn(3, requires_grad=True)
input_data = softmax(input_data)

target_data = torch.tensor([0, 1, 2])
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.multilabel_margin_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module implementation
Input Data =  tensor([0.4973, 0.4560, 0.0467], grad_fn=<SoftmaxBackward>)


Target Data =  tensor([0, 1, 2])


Output Loss =  tensor(0., grad_fn=<MultilabelMarginLossBackward>)




Functional implementation
Output Loss =  tensor(0., grad_fn=<MultilabelMarginLossBackward>)<jupyter_text>Smooth L1 Loss12. nn.SmoothL1Loss - Creates a criterion that uses a squared term if the absolute element-wise error falls below 1 and an L1 term otherwise. It is less sensitive to outliers than the MSELoss and in some cases prevents exploding gradients (e.g. see “Fast R-CNN” paper by <NAME>).         - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’    Function:$$\text{loss}(x, y) = \frac{1}{n} \sum_{i} z_{i}$$Where, $$z_{i}\text{ is }$$ \begin{split}z_{i} =\begin{cases}0.5 (x_i - y_i)^2, & \text{if } |x_i - y_i| < 1 \\|x_i - y_i| - 0.5, & \text{otherwise }\end{cases}\end{split}<jupyter_code>print("Module implementation")
loss = nn.SmoothL1Loss()
input_data = torch.randn(3, requires_grad=True)
target_data = torch.randn(3, requires_grad=False)
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.smooth_l1_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module implementation
Input Data =  tensor([ 0.4350,  0.6409, -0.5428], requires_grad=True)


Target Data =  tensor([-0.2078,  1.8307,  1.2295])


Output Loss =  tensor(0.7229, grad_fn=<SmoothL1LossBackward>)




Functional implementation
Output Loss =  tensor(0.7229, grad_fn=<SmoothL1LossBackward>)<jupyter_text>Multi Label Soft Margin loss13. nn.MultiLabelSoftMarginLoss - Creates a criterion that optimizes a multi-label one-versus-all loss based on max-entropy, between input x and target y of size (N, C)        - weight (Tensor, optional) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’    Function:$$loss(x, y) = - \sum_i y[i] * \log((1 + \exp(-x[i]))^{-1})                 + (1-y[i]) * \log\left(\frac{\exp(-x[i])}{(1 + \exp(-x[i]))}\right)$$<jupyter_code>print("Module implementation")
loss = nn.MultiLabelSoftMarginLoss()
input_data = torch.randn((1, 3), requires_grad=True)
target_data = torch.randn((1, 3), requires_grad=True)
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph = True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.multilabel_soft_margin_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output><empty_output><jupyter_text>Cosine Embedding Loss14. nn.CosineEmbeddingLoss - Creates a criterion that measures the loss given input tensors x1, x2 and a Tensor label y with values 1 or -1. This is used for measuring whether two inputs are similar or dissimilar, using the cosine distance, and is typically used for learning nonlinear embeddings or semi-supervised learning.        - margin (float, optional) – Should be a number from -1 to 1, 0 to 0.5 is suggested. If margin is missing, the default value is 0.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’    Function:$$\begin{split}\text{loss}(x, y) =\begin{cases}1 - \cos(x_1, x_2), & \text{if } y == 1 \\\max(0, \cos(x_1, x_2) - \text{margin}), & \text{if } y == -1\end{cases}\end{split}$$<jupyter_code>print("Module Implementation")
loss = nn.CosineEmbeddingLoss()

input_data1 = torch.randn(3, 3, requires_grad=True)
input_data2 = torch.randn(3, 3, requires_grad=True)
target_data = torch.tensor([-1., 1., 1.])
output_loss = loss(input_data1, input_data2, target_data)
output_loss.backward(retain_graph = True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data1 = ", input_data1)
print("\n")
print("Input Data2 = ", input_data2)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.cosine_embedding_loss(input_data1, input_data2, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module Implementation
Input Data1 =  tensor([[-0.7491, -0.3227,  1.2011],
        [-0.1215, -0.1149, -0.5620],
        [ 1.2559,  0.6572, -1.7037]], requires_grad=True)


Input Data2 =  tensor([[ 0.9953, -1.0401,  1.2301],
        [-0.6587, -0.4275, -1.5131],
        [ 1.1152, -1.9418, -1.4029]], requires_grad=True)


Target Data =  tensor([-1.,  1.,  1.])


Output Loss =  tensor(0.3263, grad_fn=<MeanBackward0>)




Functional implementation
Output Loss =  tensor(0.3263, grad_fn=<MeanBackward0>)<jupyter_text>Multi margin loss15. nn.MultiMarginLoss - Creates a criterion that optimizes a multi-class classification hinge loss (margin-based loss) between input x (a 2D mini-batch Tensor) and output y (which is a 1D tensor of target class indices, 0≤y≤x.size(1))        - p (int, optional) – Has a default value of 1. 1 and 2 are the only supported values        - margin (float, optional) – Has a default value of 1.        - weight (Tensor, optional) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’    Function:    $$\text{loss}(x, y) = \frac{\sum_i \max(0, \text{margin} - x[y] + x[i]))^p}{\text{x.size}(0)}$$<jupyter_code>print("Model Implementation")
loss = nn.MultiMarginLoss()
input_data = torch.randn(3, requires_grad=True)
target_data = torch.tensor([1, 2, 3], requires_grad=False)
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph = True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.multi_margin_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Model Implementation
Input Data =  tensor([-0.3182, -0.4905, -0.2097], requires_grad=True)


Target Data =  tensor([1, 2, 3])


Output Loss =  tensor(0.8177, grad_fn=<MultiMarginLossBackward>)




Functional implementation
Output Loss =  tensor(0.8177, grad_fn=<MultiMarginLossBackward>)<jupyter_text>Triplet Margin Loss16. nn.TripletMarginLoss - Creates a criterion that measures the triplet loss given an input tensors x1, x2, x3 and a margin with a value greater than 0. This is used for measuring a relative similarity between samples.        - margin (float, optional) – Default: 1.        - p (int, optional) – The norm degree for pairwise distance. Default: 2.        - swap (float, optional) – The distance swap is described in detail in the paper Learning shallow convolutional feature descriptors with triplet losses by <NAME>, <NAME> et al. Default: False.        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’    Function:    $$L(a, p, n) = \max \{d(a_i, p_i) - d(a_i, n_i) + {\rm margin}, 0\}\text{where,}d(x_i, y_i) = \left\lVert {\bf x}_i - {\bf y}_i \right\rVert_p$$<jupyter_code>print("Module implementation")
triplet_loss = nn.TripletMarginLoss(margin=1.0, p=2)
input_data1 = torch.randn(100, 128, requires_grad=True)
input_data2 = torch.randn(100, 128, requires_grad=True)
input_data3 = torch.randn(100, 128, requires_grad=True)
output = triplet_loss(input_data1, input_data2, input_data3)
output.backward(retain_graph = True)

print("Output Loss = ", output_loss)
print("\n")

print("Functional implementation")
output_loss = F.triplet_margin_loss(input_data1, input_data2, input_data3, margin=1.0, p=2)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")<jupyter_output>Module implementation
Output Loss =  tensor(0.8177, grad_fn=<MultiMarginLossBackward>)


Functional implementation
Output Loss =  tensor(1.0206, grad_fn=<MeanBackward0>)
{'fraction_non_alphanumeric': 0.10434286197564276, 'fraction_numerical': 0.025625845737483086, 'mean_word_length': 4.497094374709437, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 93, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '77144', 'license': 'Apache-2.0', 'n_tokens_mistral': 14925, 'n_tokens_neox': 13904, 'n_words': 5348, 'path': 'K) RoadMap 11 - Torch NN 5 - Loss Functions.ipynb', 'repo_name': 'Charlottecuc/Pytorch_Tutorial', 'size': '47308'}