<jupyter_start><jupyter_text>Comparison between activation functions---Author : <NAME>  Group : Escaper's study  Date: 2019-07-24   In 2017, Google reported new activation function, **Swish**. ![Swish](https://miro.medium.com/max/788/1*6tUAKb3_1uGybwSufey3ZA.tiff)  [Comparison of activation functions](https://medium.com/@jaiyamsharma/experiments-with-swish-activation-function-on-mnist-dataset-fc89a8c79ff7)![Swish with Beta](https://media.springernature.com/original/springer-static/image/chp%3A10.1007%2F978-3-030-19501-4_49/MediaObjects/481228_1_En_49_Fig2_HTML.png)  Model scaffoldwith keras backend of tensorflow library, analyze function returns loss and accuracy with specific dataset and activation function. analyze(dataset, activation, epochs=3, n_hidden_layers=2, n_hidden_neurons=128, optimizer='adam',             loss='sparse_categorical_crossentropy', metrics=['accuracy'], verbose=0) Input**dataset**: sample dataset provided by library  **activation**: activation function which is runnable inside the keras layer  **epoch**: number of epochs, (default)= 3  **n_hidden_layer**: number of hidden Dense layers, (default)= 2  **n_hidden_neurons**: number of hidden neurons per layers, (default)=128  **optimizer** : (default)='adam'  **loss**: (default)='sparse_categorical_crossentropy'  **metrics**: (default )=['accuracy']  **verbos**: Epoch learning print option, (default)=0 (disabled) Output**loss**: model loss evaluated with testset  **acc**: model accuracy evaluated with testset<jupyter_code>import tensorflow as tf

def analyze(dataset, activation, epochs=3, n_hidden_layers=2, 
            n_hidden_neurons=128, optimizer='adam', 
            loss='sparse_categorical_crossentropy', 
            metrics=['accuracy'], verbose=0):
  (x_train, y_train), (x_test, y_test) = dataset.load_data()
  x_train = tf.keras.utils.normalize(x_train, axis=1)
  x_test = tf.keras.utils.normalize(x_test, axis=1)
  
  input_shape = x_train[0].shape
  
  model = tf.keras.models.Sequential()
  model.add(tf.keras.layers.Flatten(input_shape=input_shape))
  for _ in range(n_hidden_layers):
    model.add(tf.keras.layers.Dense(n_hidden_neurons, activation=activation))
  model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))

  model.compile(optimizer=optimizer, loss=loss, metrics=metrics)
  model.fit(x_train, y_train, epochs=epochs, verbose=verbose)
  loss, acc = model.evaluate(x_test, y_test, verbose=verbose)
  return loss, acc<jupyter_output><empty_output><jupyter_text>Summarize n-times running of analyze functionsummarize total loss and acc as pd.DataFrame from n-times running of analyze() summarize(ntimes, dataset, activation, **kwargs) Input**ntimes**: same models will be analyzed n-times by setting this value, (default)=5  **dataset**: sample dataset provided by library  **activation**: activation function which is runnable inside the keras layer  **kwargs**: keyword arguments anything inside analyze function will be effective Output**df**: pd.DataFrame, columns=['loss, acc'], index=range(ntimes).<jupyter_code>import pandas as pd

def summarize(ntimes, dataset, activation, **kwargs):
  df = pd.DataFrame(columns=['loss', 'acc'])
  for i in range(ntimes):
    loss, acc = analyze(dataset, activation, **kwargs)
    df.loc[i,'loss'] = loss
    df.loc[i,'acc'] = acc
  return df<jupyter_output><empty_output><jupyter_text>Total summary of activation functions. (with plot option) main(activation_functions, dataset, plot=False, **kwargs) Input**activation_functions**: dict of activtion_functions  **dataset**: sample dataset provided by library  **plot**: bool, (default)=True. if False, seaborn stripplot of loss, acc is disabled  **kwargs**: keyword arguments anything inside analyze function will be effective Output**df**: dict, key=['loss', 'acc']/, value=pd.DataFrame(columns=['..activation_function_names..'']) summary_plot(summary) Input**summary**: summary output from main() Output**None**: showing inline loss and acc plots of summary<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

def main(activation_functions, dataset, ntimes=5, plot=True, **kwargs):
  summary = {}
  print(f"""Analysis start
  Activation functions: {activation_functions.keys()}
  Dataset:{dataset.__name__}""")
  for i, (name, activation) in enumerate(activation_functions.items(), start=1):
    summary[name] = summarize(ntimes, dataset, activation, **kwargs)
    print(f'[{i}/{len(activation_functions)}][{name} analysis completed]')
  if plot:
    summary_plot(summary)
  return summary

def summary_plot(summary):
  df_loss = pd.DataFrame()
  df_acc = pd.DataFrame()
  for name, df in summary.items():
    df_loss[name] = df['loss']
    df_acc[name] = df['acc']
  plt.figure(1)
  plt.title(f'loss')
  sns.stripplot(data=df_loss, alpha=0.7)
  plt.figure(2)
  plt.title('acc')
  sns.stripplot(data=df_acc, alpha=0.7)<jupyter_output><empty_output><jupyter_text>Usage example 1. Listup your activation functions2. Assign keras dataset<jupyter_code>activation_functions = dict(
    tanh = tf.nn.tanh,
    sigmoid = tf.nn.sigmoid,
    relu = tf.nn.relu,
    leaky_relu = tf.nn.leaky_relu,
    swish = tf.nn.swish,
)
dataset = tf.keras.datasets.mnist<jupyter_output><empty_output><jupyter_text>3. run main function with your options<jupyter_code>mnist_ntimes5_epoch3_layer1_neuron5 = main(activation_functions, dataset, 
                                    ntimes=5, epochs=3, n_hidden_layers=1, n_hidden_neurons=5)
activation_functions = dict(
    tanh = tf.nn.tanh,
    sigmoid = tf.nn.sigmoid,
    relu = tf.nn.relu,
    leaky_relu = tf.nn.leaky_relu,
    swish = tf.nn.swish,
)
dataset = tf.keras.datasets.mnist

mnist_ntimes3_epoch10_layer10_neuron100 = main(activation_functions, dataset, 
                                    ntimes=3, epochs=10, n_hidden_layers=10, n_hidden_neurons=100, verbose=1)
summary_plot(mnist_ntimes3_epoch10_layer10_neuron100)
activation_functions = dict(
    tanh = tf.nn.tanh,
    sigmoid = tf.nn.sigmoid,
    relu = tf.nn.relu,
    leaky_relu = tf.nn.leaky_relu,
    swish = tf.nn.swish,
)
dataset = tf.keras.datasets.cifar10

cifar10_ntimes3_epoch10_layer10_neuron100 = main(activation_functions, dataset, 
                                    ntimes=3, epochs=3, n_hidden_layers=10, n_hidden_neurons=100)<jupyter_output>Analysis start
  Activation functions: dict_keys(['tanh', 'sigmoid', 'relu', 'leaky_relu', 'swish'])
  Dataset:tensorflow.python.keras.api._v1.keras.datasets.cifar10
[1/5][tanh analysis completed]
[2/5][sigmoid analysis completed]
[3/5][relu analysis completed]
[4/5][leaky_relu analysis completed]
[5/5][swish analysis completed]
{'fraction_non_alphanumeric': 0.1280929390464695, 'fraction_numerical': 0.023989136994568498, 'mean_word_length': 5.30732635585157, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '170600', 'license': 'MIT', 'n_tokens_mistral': 2307, 'n_tokens_neox': 2190, 'n_words': 485, 'path': 'activation_function/escapers_about_activation_function.ipynb', 'repo_name': 'jeakwon/escapers', 'size': '6630'}