<jupyter_start><jupyter_text>Kaggle Titanic survival - TensorFlow api-based neural net.Taking https://github.com/MichaelAllen1966/1804_python_healthcare/blob/master/titanic/20b_tensorflow_api.ipynb and adapting the code to run in batch mode by making use of nbparameterise library.This workbook builds a neural network to predict survival on the titanic using the common framework TensorFlow (using keras).It can be run in standalone based on the parameter values defined in the first code cell.Alternatively, the notebook can be run in a batch mode by setting up the parameter values in "batch_running_ipynb.ipynb". That notebook will access the parameters in the first cell of this notebook and alter their value before running this notebook and saving it (with a unique name).  Parameters that can be changed in a batch runThe first code cell needs to contain all of the parameters that can be changed in a batch run.<jupyter_code># define the network structure
number_hidden_layers =  1
number_nodes = 1
dropout = 0.5
wide_and_deep = 1
single_final_hidden_layer = 0

# define the network training mode
learning_rate = 0.01
include_class_weights = 1
calculation_batch_size = 32
max_epoch = 3

number_kfold = 5
output_folder = "output/"<jupyter_output><empty_output><jupyter_text>Use the parameter values to create a string to be used as the model name<jupyter_code># replace "." with a "-""
learning_rate_str =  str(learning_rate).replace("0.", "0-")
dropout_str = str(dropout).replace("0.", "0-")

model_name = (f"model_{number_hidden_layers}_{number_nodes}_{dropout_str}_"
              f"{wide_and_deep}_{single_final_hidden_layer}_{learning_rate_str}"
              f"_{calculation_batch_size}_{number_kfold}_{max_epoch}")
# Turn warnings off to keep notebook tidy
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Load modules<jupyter_code>import numpy as np
import pandas as pd

# sklearn for pre-processing
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import StratifiedKFold

# TensorFlow api model
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import backend as K
from tensorflow.keras.losses import binary_crossentropy

import os<jupyter_output><empty_output><jupyter_text>Create output folder<jupyter_code># if folder and subfolder don't exist, create them
if not os.path.exists(f"{output_folder}"):
    os.makedirs(f"{output_folder}")<jupyter_output><empty_output><jupyter_text>Download data if not previously downloaded<jupyter_code>download_required = True

if download_required:
    
    # Download processed data:
    address = 'https://raw.githubusercontent.com/MichaelAllen1966/' + \
                '1804_python_healthcare/master/titanic/data/processed_data.csv'
    
    data = pd.read_csv(address)

    # Create a data subfolder if one does not already exist
    import os
    data_directory ='./data/'
    if not os.path.exists(data_directory):
        os.makedirs(data_directory)

    # Save data
    data.to_csv(data_directory + 'processed_data.csv', index=False)<jupyter_output><empty_output><jupyter_text>Define function to calculate accuracy measurements<jupyter_code>def calculate_accuracy(observed, predicted):
    
    """
    Calculates a range of accuracy scores from observed and predicted classes.
    
    Takes two list or NumPy arrays (observed class values, and predicted class 
    values), and returns a dictionary of results.
    
     1) observed positive rate: proportion of observed cases that are +ve
     2) Predicted positive rate: proportion of predicted cases that are +ve
     3) observed negative rate: proportion of observed cases that are -ve
     4) Predicted negative rate: proportion of predicted cases that are -ve  
     5) accuracy: proportion of predicted results that are correct    
     6) precision: proportion of predicted +ve that are correct
     7) recall: proportion of true +ve correctly identified
     8) f1: harmonic mean of precision and recall
     9) sensitivity: Same as recall
    10) specificity: Proportion of true -ve identified:        
    11) positive likelihood: increased probability of true +ve if test +ve
    12) negative likelihood: reduced probability of true +ve if test -ve
    13) false positive rate: proportion of false +ves in true -ve patients
    14) false negative rate: proportion of false -ves in true +ve patients
    15) true positive rate: Same as recall
    16) true negative rate
    17) positive predictive value: chance of true +ve if test +ve
    18) negative predictive value: chance of true -ve if test -ve
    
    """
    
    # Converts list to NumPy arrays
    if type(observed) == list:
        observed = np.array(observed)
    if type(predicted) == list:
        predicted = np.array(predicted)
    
    # Calculate accuracy scores
    observed_positives = observed == 1
    observed_negatives = observed == 0
    predicted_positives = predicted == 1
    predicted_negatives = predicted == 0
    
    true_positives = (predicted_positives == 1) & (observed_positives == 1)
    
    false_positives = (predicted_positives == 1) & (observed_positives == 0)
    
    true_negatives = (predicted_negatives == 1) & (observed_negatives == 1)
    
    accuracy = np.mean(predicted == observed)
    
    precision = (np.sum(true_positives) /
                 (np.sum(true_positives) + np.sum(false_positives)))
        
    recall = np.sum(true_positives) / np.sum(observed_positives)
    
    sensitivity = recall
    
    f1 = 2 * ((precision * recall) / (precision + recall))
    
    specificity = np.sum(true_negatives) / np.sum(observed_negatives)
    
    positive_likelihood = sensitivity / (1 - specificity)
    
    negative_likelihood = (1 - sensitivity) / specificity
    
    false_positive_rate = 1 - specificity
    
    false_negative_rate = 1 - sensitivity
    
    true_positive_rate = sensitivity
    
    true_negative_rate = specificity
    
    positive_predictive_value = (np.sum(true_positives) / 
                                 np.sum(observed_positives))
    
    negative_predictive_value = (np.sum(true_negatives) / 
                                  np.sum(observed_positives))
    
    # Create dictionary for results, and add results
    results = dict()
    
    results['observed_positive_rate'] = np.mean(observed_positives)
    results['observed_negative_rate'] = np.mean(observed_negatives)
    results['predicted_positive_rate'] = np.mean(predicted_positives)
    results['predicted_negative_rate'] = np.mean(predicted_negatives)
    results['accuracy'] = accuracy
    results['precision'] = precision
    results['recall'] = recall
    results['f1'] = f1
    results['sensitivity'] = sensitivity
    results['specificity'] = specificity
    results['positive_likelihood'] = positive_likelihood
    results['negative_likelihood'] = negative_likelihood
    results['false_positive_rate'] = false_positive_rate
    results['false_negative_rate'] = false_negative_rate
    results['true_positive_rate'] = true_positive_rate
    results['true_negative_rate'] = true_negative_rate
    results['positive_predictive_value'] = positive_predictive_value
    results['negative_predictive_value'] = negative_predictive_value
    
    return results<jupyter_output><empty_output><jupyter_text>Define function to scale dataIn neural networks it is common to to scale input data 0-1 rather than use standardisation (subtracting mean and dividing by standard deviation) of each feature).<jupyter_code>def scale_data(X_train, X_test):
    """Scale data 0-1 based on min and max in training set"""
    
    # Initialise a new scaling object for normalising input data
    sc = MinMaxScaler()

    # Set up the scaler just on the training set
    sc.fit(X_train)

    # Apply the scaler to the training and test sets
    train_sc = sc.transform(X_train)
    test_sc = sc.transform(X_test)
    
    return train_sc, test_sc<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code>data = pd.read_csv('data/processed_data.csv')
# Make all data 'float' type
data = data.astype(float)
data.drop('PassengerId', inplace=True, axis=1)
X = data.drop('Survived',axis=1) # X = all 'data' except the 'survived' column
y = data['Survived'] # y = 'survived' column from 'data'
# Convert to NumPy as required for k-fold splits
X_np = X.values
y_np = y.values<jupyter_output><empty_output><jupyter_text>Take a look at the data<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>Set up neural netHere we use the api-based method to set up a TensorFlow neural network. This method allows usto more flexibly define the inputs for each layer (it would not be possible to dynamically change the wide and deep or single final layer using the Sequential method).We will put construction of the neural net into a separate function. We need a function for each option of number of hidden layers as this aspect is not possible to dynamically change using the api-based method.The inputs are connected to up to three hidden layers. Each layer having the number of nodes that is a multiple of the number of inputs (up to x 10), before being connected to an output node that give a probability of being a category. It is an option for the input values to feed directly into the output node (along with the final hidden layer) - known as wide and deep. Another option is to have an additional layer prior to the output layer that contains a single node.It also contains some useful additions (batch normalisation and dropout) as described below.The layers of the network are:1) An input layer (which needs to be defined)2) A fully-connected (dense) layer. This is defined by the number of inputs (the number of input features) and the number of outputs. The number of outputs is a multuiple of the number of inputs (up to 10 x). The output of the layer uses ReLU (rectified linear unit) activation. ReLU activation is most common for the inner layers of a neural network. Negative input values are set to zero. Positive input values are left unchanged.3) A batch normalisation layer. This is not usually used for small models, but can increase the speed of training for larger models. It is added here as an example of how to include it (in large models all dense layers would be followed by a batch normalisation layer). The layer definition includes the number of inputs to normalise. 4) A dropout layer. This layer randomly sets outputs from the preceding layer to zero during training (a different set of outputs is zeroed for each training iteration). This helps prevent over-fitting of the model to the training data. Typically between 0.1 and 0.3 outputs are set to zero (p=0.1 means 10% of outputs are set to zero).The above three layers are then repeated up to 3 times.5) The option of a single node final layer6) A final fully connected linear layer of one nodes (more nodes could be used for more classes, in which case use softmax activation and categorical_crossentropy in the loss function). The output of the net is the probability of surviving (usually a probability of >= 0.5 will be classes as ‘survived’). Option to feed the input values into this output layer.<jupyter_code>def make_net_1_hidden_layer(number_features, number_nodes = 5, 
                            dropout_rate = 0.25, wide_and_deep = 0, 
                            single_final_hidden_layer = 0, 
                            learning_rate = 0.003):
    
    #create a network with 1 hidden layer
    
    # Clear Tensorflow
    K.clear_session()
    
    # Define layers
    inputs = layers.Input(shape=number_features)
    
    dense_1 = layers.Dense(number_features * number_nodes, 
                           activation = 'relu')(inputs)
    norm_1 = layers.BatchNormalization()(dense_1)    
    dropout_1 = layers.Dropout(dropout_rate)(norm_1)
        
    if wide_and_deep == 1:
        # Create a layer that concatenates last dense layer and the input layer
        concat = layers.Concatenate()([inputs, dropout_1])
        
        if single_final_hidden_layer == 0:
            # wide deep goes to output layer
            outputs = layers.Dense(1, activation = 'sigmoid')(concat)

        else:
            # wide deep goes a final hidden layer with 1 node
            single_node = layers.Dense(1, activation = 'sigmoid')(concat)
            # then into output layer
            outputs = layers.Dense(1, activation = 'sigmoid')(single_node)
            
    else:
        if single_final_hidden_layer == 0:
            # not include a final hidden layer with 1 node
            outputs = layers.Dense(1, activation = 'sigmoid')(dropout_1)
            
        else:
            # include a final hidden layer with 1 node
            single_node = layers.Dense(1, activation = 'sigmoid')(dropout_1)
            # then into output layer
            outputs = layers.Dense(1, activation = 'sigmoid')(single_node)
            
    net = Model(inputs, outputs)
    
    # Compiling model
    opt = Adam(lr = learning_rate)
    
    net.compile(loss = 'binary_crossentropy',
                optimizer = opt,
                metrics = ['accuracy'])
    return net
def make_net_2_hidden_layers(number_features, number_nodes = 5, 
                             dropout_rate = 0.25, wide_and_deep = 0, 
                             single_final_hidden_layer = 0, 
                             learning_rate = 0.003):
    
    #create a network with 2 hidden layers
    
    # Clear Tensorflow
    K.clear_session()
    
    # Define layers
    inputs = layers.Input(shape=number_features)
    
    dense_1 = layers.Dense(number_features * number_nodes,
                           activation = 'relu')(inputs)
    norm_1 = layers.BatchNormalization()(dense_1)    
    dropout_1 = layers.Dropout(dropout_rate)(norm_1)
    
    dense_2 = layers.Dense(number_features * number_nodes, 
                           activation = 'relu')(dropout_1)
    norm_2 = layers.BatchNormalization()(dense_2)
    dropout_2 = layers.Dropout(dropout_rate)(norm_2)
    
    if wide_and_deep == 1:
        # Create a layer that concatenates last dense layer and the input layer
        concat = layers.Concatenate()([inputs, dropout_2])
        
        if single_final_hidden_layer == 0:
            # wide deep goes to output layer
            outputs = layers.Dense(1, activation = 'sigmoid')(concat)

        else:
            # wide deep goes a final hidden layer with 1 node
            single_node = layers.Dense(1, activation = 'sigmoid')(concat)
            # then into output layer
            outputs = layers.Dense(1, activation = 'sigmoid')(single_node)
            
    else:
        if single_final_hidden_layer == 0:
            # not include a final hidden layer with 1 node
            outputs = layers.Dense(1, activation = 'sigmoid')(dropout_2)
            
        else:
            # include a final hidden layer with 1 node
            single_node = layers.Dense(1, activation = 'sigmoid')(dropout_2)
            # then into output layer
            outputs = layers.Dense(1, activation = 'sigmoid')(single_node)
   
    net = Model(inputs, outputs)
    
    # Compiling model
    opt = Adam(lr = learning_rate)
    
    net.compile(loss = 'binary_crossentropy',
                optimizer = opt,
                metrics = ['accuracy'])
    return net
def make_net_3_hidden_layers(number_features, number_nodes = 5, 
                             dropout_rate = 0.25, wide_and_deep = 0, 
                             single_final_hidden_layer = 0, 
                             learning_rate = 0.003):
    
    #create a network with 3 hidden layers
    
    # Clear Tensorflow
    K.clear_session()
    
    # Define layers
    inputs = layers.Input(shape=number_features)
    
    dense_1 = layers.Dense(number_features * number_nodes, 
                           activation = 'relu')(inputs)
    norm_1 = layers.BatchNormalization()(dense_1)    
    dropout_1 = layers.Dropout(dropout_rate)(norm_1)
    
    dense_2 = layers.Dense(number_features * number_nodes, 
                           activation = 'relu')(dropout_1)
    norm_2 = layers.BatchNormalization()(dense_2)
    dropout_2 = layers.Dropout(dropout_rate)(norm_2)
    
    dense_3 = layers.Dense(number_features * number_nodes, 
                           activation = 'relu')(dropout_2)
    norm_3 = layers.BatchNormalization()(dense_3)
    dropout_3 = layers.Dropout(dropout_rate)(norm_3)
    
    if wide_and_deep == 1:
        # Create a layer that concatenates last dense layer and the input layer
        concat = layers.Concatenate()([inputs, dropout_3])
        
        if single_final_hidden_layer == 0:
            # wide deep goes to output layer
            outputs = layers.Dense(1, activation = 'sigmoid')(concat)

        else:
            # wide deep goes a final hidden layer with 1 node
            single_node = layers.Dense(1, activation = 'sigmoid')(concat)
            # then into output layer
            outputs = layers.Dense(1, activation = 'sigmoid')(single_node)
            
    else:
        if single_final_hidden_layer == 0:
            # not include a final hidden layer with 1 node
            outputs = layers.Dense(1, activation = 'sigmoid')(dropout_3)
            
        else:
            # include a final hidden layer with 1 node
            single_node = layers.Dense(1, activation = 'sigmoid')(dropout_3)
            # then into output layer
            outputs = layers.Dense(1, activation = 'sigmoid')(single_node)
        
    net = Model(inputs, outputs)
    
    # Compiling model
    opt = Adam(lr = learning_rate)
    
    net.compile(loss = 'binary_crossentropy',
                optimizer = opt,
                metrics = ['accuracy'])
    return net
def make_net(number_hidden_layers, number_features, number_nodes, dropout_rate, 
             wide_and_deep, single_final_hidden_layer, learning_rate):

    if number_hidden_layers == 1:
        model = make_net_1_hidden_layer(number_features, number_nodes, 
                                        dropout_rate, wide_and_deep, 
                                        single_final_hidden_layer,
                                        learning_rate)
    elif number_hidden_layers == 2:
        model = make_net_2_hidden_layers(number_features, number_nodes, 
                                         dropout_rate, wide_and_deep, 
                                         single_final_hidden_layer, 
                                         learning_rate)
    elif number_hidden_layers == 3:
        model = make_net_3_hidden_layers(number_features, number_nodes, 
                                         dropout_rate, wide_and_deep, 
                                         single_final_hidden_layer, 
                                         learning_rate)
    else:
        print(f"User requested {number_hidden_layers} hidden layers, this "
              f"option is not currently possible, a network will be created "
              f"with 3 hidden layers.")
        model = make_net_3(number_features, number_nodes, dropout_rate, 
                           wide_and_deep, single_final_hidden_layer, 
                           learning_rate)
    return(model)<jupyter_output><empty_output><jupyter_text>Run the model with k-fold validation<jupyter_code># Set up lists to hold results
training_acc_results = []
test_acc_results = []

# Set up splits
skf = StratifiedKFold(n_splits = 5)
skf.get_n_splits(X, y)

# Loop through the k-fold splits
k_counter = 0

for train_index, test_index in skf.split(X_np, y_np):
    k_counter +=1
    print('K_fold {}'.format(k_counter))
    
    # Get X and Y train/test
    X_train, X_test = X_np[train_index], X_np[test_index]
    y_train, y_test = y_np[train_index], y_np[test_index]
    
    # Scale X data
    X_train_sc, X_test_sc = scale_data(X_train, X_test)
    
    # Define network
    
    # Define network
    number_features = X_train_sc.shape[1]
    model = make_net(number_hidden_layers, number_features, number_nodes, 
                     dropout, wide_and_deep, single_final_hidden_layer, 
                     learning_rate)
     
    model.summary()
    
    # Train model
    history = model.fit(X_train_sc,
                        y_train,
                        epochs = max_epoch,
                        batch_size = calculation_batch_size,
                        validation_data = (X_test_sc, y_test),
                        verbose = 0)
            
    ### Test model (print results for each k-fold iteration)
    probability = model.predict(X_train_sc)
    y_pred_train = probability >= 0.5
    y_pred_train = y_pred_train.flatten()
    accuracy_train = np.mean(y_pred_train == y_train)
    training_acc_results.append(accuracy_train)

    probability = model.predict(X_test_sc)
    y_pred_test = probability >= 0.5
    y_pred_test = y_pred_test.flatten()
    accuracy_test = np.mean(y_pred_test == y_test)
    test_acc_results.append(accuracy_test)<jupyter_output><empty_output><jupyter_text>Create and save as png file a visualisation of the neural network structureThis needs a pip or conda install pydot and graphviz (included in the tensorflow_batch environment)<jupyter_code>model_path = f"{output_folder}/{model_name}"

keras.utils.plot_model(model, f"{model_path}.png", show_shapes=True)<jupyter_output><empty_output><jupyter_text>Show training and test results<jupyter_code># Show individual accuracies on training data
training_acc_results
# Show individual accuracies on test data
test_acc_results
# Get mean results
mean_training = np.mean(training_acc_results)
mean_test = np.mean(test_acc_results)

# Display each to three decimal places
print (f"mean accuracy (across all {number_kfold} k-folds) for training set: {mean_training}")
print (f"mean accuracy (across all {number_kfold} k-folds) for test set: {mean_test}")<jupyter_output><empty_output><jupyter_text>Plot results: Box PlotBox plots show median (orange line), the second and third quartiles (the box), the range (excluding outliers), and any outliers as 'whisker' points. Outliers, by convention, are conisiered to be any points outside of the quartiles +/- 1.5 times the interquartile range. The limit for outliers may be changed using the optional `whis` argument in the boxplot.Medians tend to be an easy reliable guide to the centre of a distribution (i.e. look at the medians to see whether a fit is improving or not, but also look at the box plot to see how much variability there is).Test sets tend to be more variable in their accuracy measures. Can you think why?<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

# Set up X data 
x_for_box = [training_acc_results, test_acc_results]

# Set up X labels
labels = ['Training', 'Test'] 

# Set up figure
fig = plt.figure(figsize=(5,5))

# Add subplot (can be used to define multiple plots in same figure)
ax1 = fig.add_subplot(111)

# Define Box Plot (`widths` is optional)
ax1.boxplot(x_for_box, 
            widths=0.7,
            whis=100)

# Set X and Y labels
ax1.set_xticklabels(labels)
ax1.set_ylabel('Accuracy')

# Show plot
plt.show()<jupyter_output><empty_output><jupyter_text>Using TensorFlow's training historyTensorFlow can track the history of training, enabling us to examine performance against training and test sets over time. Here we will use the same model as above, but without k-fold validation and with history tracking.`history` is a dictionary containing data collected during training. Let's take a look at the keys in this dictionary (these are the metrics monitored during training):<jupyter_code>history_dict = history.history
history_dict.keys()<jupyter_output><empty_output><jupyter_text>Plot training history:<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

acc_values = history_dict['accuracy']
val_acc_values = history_dict['val_accuracy']
epochs = range(1, len(acc_values) + 1)

plt.plot(epochs, acc_values, 'bo', label='Training acc')
plt.plot(epochs, val_acc_values, 'b', label='Test accuracy')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.show()<jupyter_output><empty_output><jupyter_text>Getting model weightsHere we show how weights for a layer can be extracted from the model if required.<jupyter_code>hidden1 = model.layers[1]
hidden1.name
weights, biases = hidden1.get_weights() # Biases are not used in this model
weights.shape
weights<jupyter_output><empty_output><jupyter_text>Evaluating modelIf a test set is not used to evaluate in training, or if there is an independnent test set, evaluation may be quickly performed using the `evaluate` method.<jupyter_code>model.evaluate(X_test_sc, y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07458851866720193, 'fraction_numerical': 0.010638297872340425, 'mean_word_length': 3.093837304847987, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 76, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '95093', 'license': 'MIT', 'n_tokens_mistral': 7317, 'n_tokens_neox': 6997, 'n_words': 2284, 'path': 'tensorflow_api.ipynb', 'repo_name': 'KerryPearn/batch_run_ipynb_nbparameterise', 'size': '24910'}