<jupyter_start><jupyter_text>Tensorflow and keras<jupyter_code>import keras
from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.optimizers import SGD
import numpy as np

def sample_threeclass(n, ratio=0.8):
    np.random.seed(42)
    y_0 = np.random.randint(2, size=(n, 1))
    switch = (np.random.random(size=(n, 1)) <= ratio)
    y_1 = ~y_0 & switch
    y_2 = ~y_0 & ~switch
    y = np.concatenate([y_0, y_1, y_2], axis=1)
    
    X = y_0 + (np.random.normal(size=n) / 5)[np.newaxis].T
    return (X, y)


X_train, y_train = sample_threeclass(1000)
X_test, y_test = sample_threeclass(100)<jupyter_output><empty_output><jupyter_text>LinearThe simplest activation function is linear. A linear activation is simply a linear transform on the data. The input signals from the tensors that sink into the node are simply added up, and the result of that addition is packaged and sent downstream through the output tensor. A linear activator allows a node to learn linear relationships within the data.<jupyter_code>clf = Sequential()
clf.add(Dense(3, activation='linear', input_shape=(1,), name='hidden'))
clf.add(Dense(3, activation='softmax', name='out'))
clf.compile(loss='categorical_crossentropy', optimizer=SGD(), metrics=['accuracy'])

clf.fit(X_train, y_train, epochs=10, batch_size=16)<jupyter_output>Epoch 1/10
63/63 [==============================] - 0s 639us/step - loss: 0.7187 - accuracy: 0.7580
Epoch 2/10
63/63 [==============================] - 0s 517us/step - loss: 0.6279 - accuracy: 0.8240
Epoch 3/10
63/63 [==============================] - 0s 858us/step - loss: 0.5607 - accuracy: 0.8590
Epoch 4/10
63/63 [==============================] - 0s 721us/step - loss: 0.5103 - accuracy: 0.8770
Epoch 5/10
63/63 [==============================] - 0s 580us/step - loss: 0.4718 - accuracy: 0.8820
Epoch 6/10
63/63 [==============================] - 0s 595us/step - loss: 0.4423 - accuracy: 0.8860
Epoch 7/10
63/63 [==============================] - 0s 731us/step - loss: 0.4195 - accuracy: 0.8900
Epoch 8/10
63/63 [==============================] - 0s 595us/step - loss: 0.4016 - accuracy: 0.8910
Epoch 9/10
63/63 [==============================] - 0s 568us/step - loss: 0.3874 - accuracy: 0.8920
Epoch 10/10
63/63 [==============================] - 0s 907us/step - loss: 0.3759 - accuracy: 0.8920[...]<jupyter_text>Logistic (sigmoid)Non-linear activation functions come in many shapes. The most general class of non-linear activation function is so-called sigmoid functions, which are distinguishable by having an S-shaped value curve.The most well-known sigmoid function is the logistic function:<jupyter_code>def logistic_func(x): return np.e**x/(np.e**x + 1)

import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(np.arange(-5, 5, 0.2), [logistic_func(x) for x in np.arange(-5, 5, 0.2)])
plt.axis('off')
clf = Sequential()
clf.add(Dense(3, activation='sigmoid', input_shape=(1,), name='hidden'))
clf.add(Dense(3, activation='softmax', name='out'))
clf.compile(loss='categorical_crossentropy', optimizer=SGD(), metrics=['accuracy'])

clf.fit(X_train, y_train, epochs=20, batch_size=16, verbose=0)
fig = plt.figure(figsize=(12, 8))
plt.plot(range(len(clf.history.history['accuracy'])), clf.history.history['accuracy'], linewidth=4)
import seaborn as sns; sns.despine()
plt.title("Sigmoid Activation Accuracy Per Epoch", fontsize=20)
pass<jupyter_output><empty_output><jupyter_text>TanhAn alternative to the logistic function is arctan or "tanh". This curve has different properties:<jupyter_code>plt.plot(np.arange(-5, 5, 0.2), [np.tanh(x) for x in np.arange(-5, 5, 0.2)])
plt.axis('off')

clf = Sequential()
clf.add(Dense(3, activation='tanh', input_shape=(1,), name='hidden'))
clf.add(Dense(3, activation='softmax', name='out'))
clf.compile(loss='categorical_crossentropy', optimizer=SGD(), metrics=['accuracy'])

clf.fit(X_train, y_train, epochs=10, batch_size=16, verbose=0)
fig = plt.figure(figsize=(12, 8))
plt.plot(range(len(clf.history.history['accuracy'])), clf.history.history['accuracy'], linewidth=4)
import seaborn as sns; sns.despine()
plt.title("Tanh Activation Accuracy Per Epoch", fontsize=20)
pass<jupyter_output><empty_output><jupyter_text>Rectified linear (ReLU)The rectified linear function is a piecewise function which staples together a flat and a linear activation function. It is non-linear, but very much not sigmoid<jupyter_code>def relu(x):
    return 0 if x <= 0 else x

plt.plot(np.arange(-5, 5, 0.2), [relu(x) for x in np.arange(-5, 5, 0.2)])
plt.axis('off')
pass

clf = Sequential()
clf.add(Dense(3, activation='relu', input_shape=(1,), name='hidden'))
clf.add(Dense(3, activation='softmax', name='out'))
clf.compile(loss='categorical_crossentropy', optimizer=SGD(), metrics=['accuracy'])

clf.fit(X_train, y_train, epochs=10, batch_size=16, verbose=0)
fig = plt.figure(figsize=(12, 8))
plt.plot(range(len(clf.history.history['accuracy'])), clf.history.history['accuracy'], linewidth=4)
import seaborn as sns; sns.despine()
plt.title("ReLU Activation Accuracy Per Epoch", fontsize=20)
pass<jupyter_output><empty_output><jupyter_text>Leaky ReLULeaky ReLU is an attempt at rectifying the dying ReLU problem by adapting the function shape. The shape is the same, but instead of using 0 on the negative part of the curve, the Leaky ReLU uses a very small positive value:<jupyter_code>def leaky_relu(x):
    return 0.01 if x <= 0 else x

plt.plot(np.arange(-5, 5, 0.2), [relu(x) for x in np.arange(-5, 5, 0.2)])
plt.axis('off')
pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.19422903516681694, 'fraction_numerical': 0.058611361587015326, 'mean_word_length': 5.7061668681983075, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '282323', 'license': 'MIT', 'n_tokens_mistral': 2225, 'n_tokens_neox': 1969, 'n_words': 515, 'path': 'CNN/Activation_Functions.ipynb', 'repo_name': 'KarinkiManikanta/Deep-Learning', 'size': '5545'}