<jupyter_start><jupyter_text>203 ActivationView more, visit my tutorial page: https://morvanzhou.github.io/tutorials/My Youtube Channel: https://www.youtube.com/user/MorvanZhouDependencies:* torch: 0.5.0a0 (0.4.1+)* matplotlib<jupyter_code>import torch
import torch.nn.functional as F
from torch.autograd import Variable
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Firstly generate some fake data<jupyter_code>x = torch.linspace(-5, 5, 200)  # x data (tensor), shape=(200, 1) # torch.Tensor
x = Variable(x) #torch.Tensor
x_np = x.data.numpy()   # numpy array for plotting # numpy.ndarray<jupyter_output><empty_output><jupyter_text>Following are popular activation functions<jupyter_code># # deprecated, used in torch: 0.1.11
# y_relu = F.relu(x).data.numpy()
# y_sigmoid = F.sigmoid(x).data.numpy()
# y_tanh = F.tanh(x).data.numpy()
# y_softplus = F.softplus(x).data.numpy()
# # y_softmax = F.softmax(x)

y_relu = torch.relu(x).data.numpy()
y_sigmoid = torch.sigmoid(x).data.numpy()
y_tanh = torch.tanh(x).data.numpy()
y_softplus = F.softplus(x).data.numpy() #there's no softplus in torch
y_softmax = torch.softmax(x, dim=0).data.numpy()

# softmax is a special kind of activation function, it is about probability
# and will make the sum as 1.
import numpy as np
np.sum(y_softmax)<jupyter_output><empty_output><jupyter_text>Plot to visualize these activation function<jupyter_code>%matplotlib inline
plt.figure(1, figsize=(16, 6))
plt.subplot(231)
plt.plot(x_np, y_relu, c='red', label='relu')
plt.ylim((-1, 5))
plt.legend(loc='best')

plt.subplot(232)
plt.plot(x_np, y_sigmoid, c='red', label='sigmoid')
plt.ylim((-0.2, 1.2))
plt.legend(loc='best')

plt.subplot(233)
plt.plot(x_np, y_tanh, c='red', label='tanh')
plt.ylim((-1.2, 1.2))
plt.legend(loc='best')

plt.subplot(234)
plt.plot(x_np, y_softplus, c='red', label='softplus')
plt.ylim((-0.2, 6))
plt.legend(loc='best')

plt.subplot(235)
plt.plot(x_np, y_softmax, c='red', label='softmax')
plt.ylim((-0.05, 0.1))
plt.legend(loc='best')

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17159476490547745, 'fraction_numerical': 0.030053320407174018, 'mean_word_length': 6.141868512110727, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '505587', 'license': 'MIT', 'n_tokens_mistral': 918, 'n_tokens_neox': 846, 'n_words': 160, 'path': 'tutorial-contents-notebooks/203_activation.ipynb', 'repo_name': 'keineahnung2345/MorvanZhou-PyTorch-Tutorial-With-Detailed-Note', 'size': '2063'}