<jupyter_start><jupyter_text>Activation functions used in neural networks--- ***Author: <NAME>***This notebook accompanies the article: Deep Dive into Math Behind Deep Networks. The visualizations created in this book show the course of various popular activation functions.  Imports<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Auxiliary functions<jupyter_code># the auxiliary function forming the diagram
def make_plot(x, f, df, name):
    plt.figure()
    plt.figure(figsize=(12,6))
    plt.title(name, fontsize=20, fontweight='bold')
    plt.xlabel('z', fontsize=15)
    plt.ylabel('Activation function value', fontsize=15)
    sns.set_style("whitegrid")
    plt.plot(x, f, label="f (z)")
    plt.plot(x, df, label="f '(z)")
    plt.legend(loc=4, prop={'size': 15}, frameon=True,shadow=True, facecolor="white", edgecolor="black")
    plt.savefig(name + ".png")
    plt.show()<jupyter_output><empty_output><jupyter_text>Sigmoid $${\displaystyle f(x)={\frac {1}{1+e^{-x}}}}\hspace{1cm}{\displaystyle f'(x) = f(x)\cdot(1-f(x))}$$<jupyter_code>z = np.arange(-10, 10, 0.01)
f = 1/(1 + np.exp(-z))
df = f*(1 - f)
make_plot(z, f, df, "Sigmoid")<jupyter_output><empty_output><jupyter_text>Tanh $${\displaystyle f(x) = \operatorname {tgh} x={\frac {\sinh x}{\cosh x}}={\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}}\hspace{1cm}{\displaystyle f'(x) = 1 - f(x)^{2}}$$<jupyter_code>z = np.arange(-10, 10, 0.01)
f = np.tanh(z)
df = 1 - f*f
make_plot(z, f, df, "tanh")<jupyter_output><empty_output><jupyter_text>ReLU $${\displaystyle f(x) = \begin{cases}0 & \text{ if } x<0 \\ x & \text{ if } x\geq x \end{cases}}\hspace{1cm}{\displaystyle f(x) = \begin{cases}0 & \text{ if } x<0 \\ 1 & \text{ if } x\geq x \end{cases}}$$<jupyter_code>z = np.arange(-10, 10, 0.01)
f = z * (z > 0)
df = 1. * (z > 0)
make_plot(z, f, df, "ReLU")<jupyter_output><empty_output><jupyter_text>Leaky ReLU $${\displaystyle f(x) = \begin{cases}0.01 \cdot x & \text{ if } x<0 \\ x & \text{ if } x\geq x \end{cases}}\hspace{1cm}{\displaystyle f(x) = \begin{cases}0.01 & \text{ if } x<0 \\ 1 & \text{ if } x\geq x \end{cases}}$$<jupyter_code>z = np.arange(-10, 10, 0.01)
f = np.where(z > 0, z, z * 0.01)
df = np.where(z > 0, 1, 0.01)
make_plot(z, f, df, "Leaky_ReLU")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.20717299578059073, 'fraction_numerical': 0.03333333333333333, 'mean_word_length': 4.797066014669927, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '397753', 'license': 'MIT', 'n_tokens_mistral': 1018, 'n_tokens_neox': 951, 'n_words': 216, 'path': '01_mysteries_of_neural_networks/01_mathematics_of_nn/Activation functions used in neural networks.ipynb', 'repo_name': 'angliu-bu/ILearnDeepLearning.py', 'size': '2377'}