<jupyter_start><jupyter_text>Converting Outputs to Probabilities Using the Logistic FunctionWe don't always have to create a probability distrbution over the outputs by using a softmax function.A logistic function with normalization can also give us a good probability distribution from the output.The probability distribution that can be obtained from the normalized logistic function (also called the sigmoid function) is less extreme than that of the softmax.So, it might not work as well as a loss function, but oftentimes yields better thresholdable probabilities.Here we compute the cross-entropy loss using the normalized sigmoid function instead of the softmax.<jupyter_code>import torch
import torch.nn.functional as F

data = torch.Tensor([[6, 2, 1.9]])

weights = torch.Tensor([[1, 0],[0, 1],[0, 1]])

c = torch.mm(data, weights)
print("c1 and c2: " + str(c))

result = F.sigmoid(torch.autograd.Variable(c))
print("sigmoid: " + str(result))

# Normalize it

result = result / torch.sum(result)
print("normalized sigmoid: " + str(result))<jupyter_output>c1 and c2: 
 6.0000  3.9000
[torch.FloatTensor of size 1x2]

sigmoid: Variable containing:
 0.9975  0.9802
[torch.FloatTensor of size 1x2]

normalized sigmoid: Variable containing:
 0.5044  0.4956
[torch.FloatTensor of size 1x2]<jupyter_text>Convince yourself that the output of the 'softmax' function is the same as 'e to the power of c normalized'.<jupyter_code>result = torch.log(result)
print("log(sigmoid): " + str(result))

# The correct category
target = torch.LongTensor(1)
target[0] = 0 

loss = F.nll_loss(result, torch.autograd.Variable(target))
print("Loss: "+str(loss.data[0]))<jupyter_output>log(sigmoid): Variable containing:
-0.6844 -0.7020
[torch.FloatTensor of size 1x2]

Loss: 0.6844037175178528<jupyter_text>Compare the loss obtained using the sigmoid function with the loss from the softmax function.<jupyter_code>loss = F.cross_entropy(torch.autograd.Variable(c), torch.autograd.Variable(target))
print("Loss: "+str(loss.data[0]))<jupyter_output>Loss: 0.11551953107118607
{'fraction_non_alphanumeric': 0.10257656781720953, 'fraction_numerical': 0.049586776859504134, 'mean_word_length': 5.792079207920792, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '375771', 'license': 'MIT', 'n_tokens_mistral': 736, 'n_tokens_neox': 670, 'n_words': 216, 'path': 'exercise_480.ipynb', 'repo_name': 'ahurriyetoglu/DeepLearningBasicsTutorial', 'size': '2057'}