<jupyter_start><jupyter_text>Experiment AnalysisIn this notebook we will evaluate the results form the experiments executed. For each experiment, one parameter is changed and all others were kept constant as to determine the effect of one variable. **The goals of this analysis are:**1. Determine the relationship of the number of parameters in the neural network and the number of timesteps in the dataset2. Determine what effect increasing the number patterns are w.r.t. this relationship3. Determine what effect sparsity has on the capacity of the neural networks4. Investigate which activation function lead to the highest retention of information5. What type of network is able to retain the most information  To determine whether a relationshop exists between the variable being investigated and the number of required parameters in each respective neural network, the Pearson correlation coefficient is used. The domain of this metric lies between -1 and +1 or in mathematical notation $P \in [-1, 1]$. If there exists a strong positive relationship between variables, the Pearson coefficient will approach +1 and for the negative case -1.<jupyter_code>import pandas as pd 
import numpy as np
import scipy
import sklearn
import pandas as pd
from sqlalchemy import Column, Integer, String
from sqlalchemy import create_engine, Column
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
import seaborn as sns
import matplotlib.pyplot as plt
# Before using a cluster
# Base = declarative_base()
# engine = create_engine('postgresql://masters_user:password@localhost:5432/masters_experiments')<jupyter_output><empty_output><jupyter_text>Timesteps Analysis<jupyter_code>df = pd.read_csv("timesteps.csv", delimiter=",")

df.head(5)<jupyter_output><empty_output><jupyter_text>Number of parameters  ∝ time steps  Overall<jupyter_code>from matplotlib import pyplot
def plot_by_filter(x_col, 
                   y_col, 
                   x_label='Sparsity length',
                   y_label='Number of network parameters',
                   title="Effect of sparsity on the number of parameters \n in a neural network with activation ", 
                   hue="network_type", 
                   filter_col="activation_function", 
                   filter_val="tanh",
                   legend_title="NN TYPE",
                  df=None):
    sns.set_style("whitegrid")
    a4_dims = (6, 2.25)
    fig, ax = pyplot.subplots(figsize=a4_dims)
    ax.set(xlabel=x_label, 
       ylabel=y_label )
    if filter_val is not None:
        ax = sns.pointplot(ax=ax, x=x_col, y=y_col, hue=hue, 
                          marker='o',  markersize=5, ci=None,
                          data = df[df[filter_col] == filter_val])
       
        ax.axes.set_title(title + filter_val,
                          fontsize=12, y=1.05)
        ax.legend(title=filter_val.upper(), loc='center right', bbox_to_anchor=(1.37, 0.5), ncol=1)
    else:
        ax = sns.pointplot(ax=ax, x=x_col, y=y_col, hue=hue, 
                          marker='o',  markersize=5, ci=None,
                          data = df)
        ax.axes.set_title(title, fontsize=12, y=1.05)
        ax.legend(title=legend_title, loc='center right', bbox_to_anchor=(1.37, 0.5), ncol=1)

    # plt.legend()
filter_col = "network_type"
    
plot_by_filter(x_col="timesteps", 
               y_col="num_network_parameters",
               x_label='Timesteps',
               y_label='Number of network parameters',
               title="Effect of timesteps on the number of parameters " + 
                       "\n in a neural network over all activation functions", 
               hue="network_type",
              filter_col=filter_col, filter_val=None, df=df)    


    

filter_col = "network_type"
for filter_val in df[filter_col].unique():
    df_temp = df[df[filter_col] == filter_val]
    df_temp = df_temp.groupby(["timesteps", "network_type"]).agg({"num_network_parameters": "mean"}).to_records()
    df_temp = pd.DataFrame.from_records(df_temp)
    df_temp["timesteps"] = df_temp["timesteps"].astype(float)
    df_temp["num_network_parameters"] = df_temp["num_network_parameters"].astype(float)
    
    print("Pearson Correlation Between Timesteps and Number of Network Parameters for", filter_val, df_temp["timesteps"].corr(df_temp["num_network_parameters"]), type="spearman")<jupyter_output><empty_output><jupyter_text>Ratio of required parameters for increase in time steps<jupyter_code>df_temp = df.groupby(["timesteps", "network_type"]).agg({"num_network_parameters": "mean"}).to_records()
df_temp = pd.DataFrame.from_records(df_temp)
df_temp.pivot(index="timesteps", columns="network_type", values="num_network_parameters").head(11)<jupyter_output><empty_output><jupyter_text>Discussion of resultsFrom the Pearson coefficient, it is seems as if increasing the number of timesteps increases the number of required parameters for the Elman and GRU RNNs, while decreasing this requirement for the LSTM. However, upon inspecting the graph and values in the table, it is more apparent that this small correlation is due to variablity in the experiment during training. Thus it is safe to assume that there is no correlation between the number of network parameters required and the number of time steps if sparsity, number of patterns and output nodes are fixed for the **average case**.  Effect of timesteps on networks with specific activation functions<jupyter_code>filter_col = "activation_function"
for filter_val in df[filter_col].unique():
    df_temp = df[(df[filter_col] == filter_val)]
    df_temp = df_temp.groupby(["timesteps"]).agg({"num_network_parameters": "mean"}).to_records()
    df_temp = pd.DataFrame.from_records(df_temp)
    df_temp["timesteps"] = df_temp["timesteps"].astype(float)
    df_temp["num_network_parameters"] = df_temp["num_network_parameters"].astype(float)
    
    print("Pearson Correlation Between Timesteps and Number of Network Parameters for", filter_val, df_temp["timesteps"].corr(df_temp["num_network_parameters"]))
df_temp = df.groupby(["timesteps", "activation_function"]).agg({"num_network_parameters": "mean"}).to_records()
df_temp = pd.DataFrame.from_records(df_temp)
df_temp.pivot(index="timesteps", columns="activation_function", values="num_network_parameters").head(11)<jupyter_output><empty_output><jupyter_text>Discussion of activation functions  ∝ time stepsThe correlation coefficient between the required network parameters required and the increase in time steps for respective activation functions indicate that for **most activation functions**, increasing time steps will not have an effect on the required parameters of the network. Interestingly enough, this is not the case for the **selu** and **softplus**. For networks using these activation functions, the amount of memory loss is effected by the increase in timesteps.The **softmax** and **linear** activation functions seem to cope the best with the increase in timesteps and the **relu** activation function has the highest variance. The high variance of the **relu** function lends itself to be usefull in avoiding local optima.<jupyter_code>filter_col = "activation_function"
for filter_val in df[filter_col].unique():
    for filter_val_1 in df["network_type"].unique():
        df_temp = df[df["network_type"] == filter_val_1]
        df_temp = df[df[filter_col] == filter_val]
        df_temp = df_temp.groupby(["timesteps"]).agg({"num_network_parameters": "mean"}).to_records()
        df_temp = pd.DataFrame.from_records(df_temp)
        df_temp["timesteps"] = df_temp["timesteps"].astype(float)
        df_temp["num_network_parameters"] = df_temp["num_network_parameters"].astype(float)

        print("Pearson Correlation Between Timesteps and Number of Network Parameters for", filter_val_1 + " "+ filter_val, df_temp["timesteps"].corr(df_temp["num_network_parameters"]))
filter_col = "activation_function"
for filter_val in df[filter_col].unique():
    plot_by_filter(x_col="timesteps", 
                   y_col="num_network_parameters",
                   x_label='Timesteps',
                   y_label='Number of network parameters',
                   title="Effect of timesteps on the number of parameters " + 
                           "\n in a neural network with activation " +str(filter_val), 
                   hue="network_type",
                  filter_col=filter_col, filter_val=filter_val, df=df)<jupyter_output><empty_output><jupyter_text>Comparing the correlation between the type of neural network and activation function it is clear that assumptions made about activation functions hold for all recurrent neural networks.  Effect of time steps on training time<jupyter_code>filter_col = "network_type"
    
plot_by_filter(x_col="timesteps", 
               y_col="epocs",
               x_label='Timesteps',
               y_label='Number of EPOCS required to train network',
               title="Effect of timesteps on training time ", 
               hue="network_type",
              filter_col=filter_col, filter_val=None, df=df)<jupyter_output><empty_output><jupyter_text>Effect of time steps on training time<jupyter_code>filter_col = "activation_function"
for filter_val in df[filter_col].unique():
    plot_by_filter(x_col="timesteps", 
                   y_col="epocs",
                   x_label='Timesteps',
                   y_label='Number of EPOCS required to train network',
                   title="Effect of timesteps on training time " +
                           "\n for a neural network with activation " +str(filter_val), 
                   hue="network_type",
                  filter_col=filter_col, filter_val=filter_val, df=df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09158974358974359, 'fraction_numerical': 0.005435897435897436, 'mean_word_length': 3.7869415807560136, 'pattern_counts': {'":': 5, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '9316', 'license': 'MIT', 'n_tokens_mistral': 2832, 'n_tokens_neox': 2712, 'n_words': 823, 'path': 'Notebooks/ExperimentAnalysisTimesteps.ipynb', 'repo_name': 'thesneakysneak/memory_capacity_retention_rnns', 'size': '9750'}