<jupyter_start><jupyter_text>STAT 453: Deep Learning (Spring 2020)  Instructor: <NAME> (<EMAIL>)  Course website: http://pages.stat.wisc.edu/~sraschka/teaching/stat453-ss2020/  GitHub repository: https://github.com/rasbt/stat453-deep-learning-ss20<jupyter_code>%load_ext watermark
%watermark -a '<NAME>' -v -p torch<jupyter_output><empty_output><jupyter_text>Understanding Onehot Encoding and Cross Entropy in PyTorch<jupyter_code>import torch<jupyter_output><empty_output><jupyter_text>Onehot Encoding<jupyter_code>def to_onehot(y, num_classes):
    y_onehot = torch.zeros(y.size(0), num_classes)
    y_onehot.scatter_(1, y.view(-1, 1).long(), 1).float()
    return y_onehot

y = torch.tensor([0, 1, 2, 2])

y_enc = to_onehot(y, 3)

print('one-hot encoding:\n', y_enc)<jupyter_output><empty_output><jupyter_text>Softmax Suppose we have some net inputs Z, where each row is one training example:<jupyter_code>Z = torch.tensor( [[-0.3,  -0.5, -0.5],
                   [-0.4,  -0.1, -0.5],
                   [-0.3,  -0.94, -0.5],
                   [-0.99, -0.88, -0.5]])

Z<jupyter_output><empty_output><jupyter_text>Next, we convert them to "probabilities" via softmax:$$P(y=j \mid z^{(i)}) = \sigma_{\text{softmax}}(z^{(i)}) = \frac{e^{z^{(i)}}}{\sum_{j=0}^{k} e^{z_{k}^{(i)}}}.$$<jupyter_code>def softmax(z):
    return (torch.exp(z.t()) / torch.sum(torch.exp(z), dim=1)).t()

smax = softmax(Z)
print('softmax:\n', smax)<jupyter_output><empty_output><jupyter_text>The probabilties can then be converted back to class labels based on the largest probability in each row:<jupyter_code>def to_classlabel(z):
    return torch.argmax(z, dim=1)

print('predicted class labels: ', to_classlabel(smax))
print('true class labels: ', to_classlabel(y_enc))<jupyter_output><empty_output><jupyter_text>Cross Entropy Next, we compute the cross entropy for each training example: $$\mathcal{L}(\mathbf{W}; \mathbf{b}) = \frac{1}{n} \sum_{i=1}^{n} H(T_i, O_i),$$ $$H(T_i, O_i) = -\sum_m T_i \cdot log(O_i).$$<jupyter_code>def cross_entropy(softmax, y_target):
    return - torch.sum(torch.log(softmax) * (y_target), dim=1)

xent = cross_entropy(smax, y_enc)
print('Cross Entropy:', xent)<jupyter_output><empty_output><jupyter_text>In PyTorch<jupyter_code>import torch.nn.functional as F<jupyter_output><empty_output><jupyter_text>Note that `nll_loss` takes log(softmax) as input:<jupyter_code>F.nll_loss(torch.log(smax), y, reduction='none')<jupyter_output><empty_output><jupyter_text>Note that `cross_entropy` takes logits as input:<jupyter_code>F.cross_entropy(Z, y, reduction='none')<jupyter_output><empty_output><jupyter_text>Defaults By default, nll_loss & cross_entropy are already returning the average over training examples, which is useful for stability during optimization.<jupyter_code>F.cross_entropy(Z, y)
torch.mean(cross_entropy(smax, y_enc))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.163769123783032, 'fraction_numerical': 0.021557719054242003, 'mean_word_length': 6.051470588235294, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '144739', 'license': 'MIT', 'n_tokens_mistral': 1149, 'n_tokens_neox': 1118, 'n_words': 208, 'path': 'L07-logistic/code/cross-entropy-pytorch.ipynb', 'repo_name': 'VanessaYan/stat453-deep-learning-ss20', 'size': '2908'}