<jupyter_start><jupyter_text>A large portion of this code is taken from <NAME>'s: Hands-On machine learning with SciKit-Learn, Keras and Tensorflow (2nd edition). I have put comments based on information in the book as well as information I found elsewhere. Especially notes on <NAME>'s Disciplined approach to Hyper-parameter tuning paper. Chapter 10<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import os
np.random.seed(42)  # This is done here so we can have consistent results each time we run
                    # any ML code that uses random values.
                    # The Classifier's fit method would not converge for many values of the random seed.
                    # It did converge for random seed 42.
                    # Maybe this needs to be considered as a hyper-parameter?
                    # There are two schools of thought on random seed as a hyperparameter:
                    #   - If you're selecting a seed that gives a better output, then this is overfitting.
                    #     Don't do it - instead keep using the same seed for all your work.
                    #     The seed is only to make sure you can re-create your results.
                    #   - Since the seed can cause model parameters to be very different, use many seeds
                    #     and then use model averaging (bagging, Bayesian methods) to find your final result.

from tensorflow import keras  # Keras needs to be imported after the random seed is set
fashion_mnist = keras.datasets.fashion_mnist
(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()
X_valid, X_train = X_train_full[:5000], X_train_full[5000:]
y_valid, y_train = y_train_full[:5000], y_train_full[5000:]
class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']<jupyter_output><empty_output><jupyter_text>Build a Classification MLP using the Sequential API  One way to create an MLP<jupyter_code>model = keras.models.Sequential()    # Creates a stack of layers connected sequentially
model.add(keras.layers.Flatten(input_shape=[28, 28])) # Converts each input image into a 1-D array.
                                                      # Alternatively, use keras.layers.InputLayer with input_shape.
                                                      # It is best to specify the input_shape when creating the model.
                                                      # Without it, you will not be able to do certain things:
                                                      #   - print the model
                                                      #   - save the model
model.add(keras.layers.Dense(300, activation='relu')) # Each dense layer manages:
                                                      #   - Input weights for each neuron
                                                      #   - Bias for each neuron
                                                      # Specifying activation = 'relu' is the same as
                                                      #            activation = keras.activations.relu
                                                      # You can set:
                                                      #   kernel_initializer='random_uniform' to init uniform weights.
                                                      #   bias_initializer=keras.initializers.Constant(0.5)
                                                      #     to initialize constant bias values.
                                                      #     Other initializers: https://keras.io/initializers/
model.add(keras.layers.Dense(100, activation='relu'))
model.add(keras.layers.Dense(10, activation='softmax'))  # For binary classification, we would use "sigmoid".<jupyter_output><empty_output><jupyter_text>Alternative way to create an MLP<jupyter_code># from tensorflow.keras.models import Sequential
# from tensorflow.keras.layers import Flatten, Dense
# model = Sequential([
#     Flatten(input_shape=[28, 28]),
#     Dense(300, activation='relu'),
#     Dense(100, activation='relu'),
#     Dense(10, activation='softmax')
# ])<jupyter_output><empty_output><jupyter_text>Back to the original program<jupyter_code>model.summary()  # In Output Shape, None means the batch size can be anything.
                 # The Param # for each layer includes the bias value for each neuron.
                 # Be careful - with too many Total params, you may overfit,
                 #   especially when you do not have a lot of training data.
model.layers
model.layers[1]
model.layers[1].name
weights, biases = model.layers[1].get_weights()  # You can see that the dense layer:
                                                 #   - initialized the weights randomly
                                                 #   - set the bias values all to 0
weights
biases
model.compile(loss = 'sparse_categorical_crossentropy',  # Same as loss = keras.losses.sparse_categorical_crossentropy
                                                         #   For each instance we have a target class index from 0-9.
                                                         #   This is why we use sparse_categorical_crossentropy.
                                                         #   If we had multiple target values (ex. one-hot encoded),
                                                         #   we would use "categorical_crossentropy".
                                                         #   For binary classification, 
                                                         #     we would use "binary_crossentropy".
                                                         # Sparse labels -> on-hot encoding: 
                                                         #   Use keras.utils.to_categorical()
                                                         # One-hot encoded -> Sparse labels:
                                                         #   Use np.argmax(labels_array, axis=1)
              optimizer=keras.optimizers.SGD(lr=0.001),          # We could have used optimizer='sgd' here, but that would
                                                         #   mean we want to use the default learning rate of 0.01.
              metrics=['accuracy'])                      # Same as metrics = keras.metrics.sparse_categorical_accuracy
                                                         # We want to measure accuracy since this is a classifier.
history = model.fit(X_train, y_train, epochs=30,         # By default, epochs = 1, so we have to set it.
                    validation_data=(X_valid, y_valid))  # Instead of passing in validation data, we can say
                                                         #   use the last 10% of the training dataset
                                                         #   (before shuffling). Use validation_split=0.1
                                                         # If the training set was skewed with some classes
                                                         #   represented more than others, we can use
                                                         #   class_weight={0: 0.25, 1: 0.75} Pay more attention to
                                                         #   class index 1 since it is under-represented by a third.
                                                         #   To find this sampling:
                                                         #     from sklearn.utils import class_weight
                                                         #     class_weights = class_weight.compute_class_weight(
                                                         #                                     'balanced',
                                                         #                                     np.unique(y_train),
                                                         #                                     y_train)
                                                         # If some of your samples are better (maybe labeled
                                                         #   by experts), compared to others, pass a list of
                                                         #   sample weights (same length as the # of instances).
                                                         #   You can also provide sample weights to the validation
                                                         #   set by adding them to the validation_data as the third
                                                         #   value in the tuple.
history.params
history.epoch
history.history
pd.DataFrame(history.history).plot(figsize=(12, 10))  # Ideally, we would like to see decreasing loss values
                                                      # and increasing accuracy.
                                                      # Since the ML libraries use random numbers, it can result
                                                      # in different output.
                                                      # We can tell if the model has converged based on the error curves.
                                                      # The validation error is computed at the end of each epoch,
                                                      # and the training error is computed during each epoch.
                                                      # This is why the training curve should be shifted half an epoch
                                                      # to the left.
                                                      # If you want to continue training, call the fit() method again.
                                                      # It will continue where it left off.
                                                      # Tuning:
                                                      #   Tune the learning rate first.
                                                      #   Then try another optimizer.
                                                      #   Tune each hyperparameter. For each hyperparameter, retune
                                                      #     the learning rate.
                                                      #   Then try tuning the model parameters:
                                                      #     - Number of layers
                                                      #     - Neurons per layer
                                                      #     - Activation functions per layer
plt.grid(True)
plt.gca().set_ylim(0, 1)
plt.show()
loss_value, metrics = model.evaluate(X_test, y_test)
loss_value, metrics
X_new, y_new = X_test[:3], y_test[:3]
y_proba = model.predict(X_new)
y_proba.round(2)
y_pred = model.predict_classes(X_new)
y_pred, y_new
np.array(class_names)[y_pred]<jupyter_output><empty_output><jupyter_text>Build a Regression MLP using the Sequential API<jupyter_code>from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

housing = fetch_california_housing()

X_train_full, X_test, y_train_full, y_test = train_test_split(
                            housing.data, housing.target)
X_train, X_valid, y_train, y_valid = train_test_split(
                            X_train_full, y_train_full)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_valid = scaler.transform(X_valid)
X_test  = scaler.transform(X_test)
model = keras.models.Sequential([
    keras.layers.Dense(30, activation='relu', input_shape=X_train.shape[1:]),
    keras.layers.Dense(1)
])
model.compile(loss='mean_squared_error', optimizer='sgd')
history = model.fit(X_train, y_train, epochs=20, 
                    validation_data=(X_valid, y_valid))
mse_test = model.evaluate(X_test, y_test)
X_new = X_test[:3]
y_pred = model.predict(X_new)
print(f'y_pred: {y_pred.flatten().round(2)}\ny_new:  {y_new.flatten().round(2)}')<jupyter_output>y_pred: [0.72 1.66 3.63]
y_new:  [9 2 1]<jupyter_text>Building complex models with the functional APIA Wide and Deep Neural Network consists of:  - A deep section that connects the inputs to the outputs through layers.    This allows the network to learn complex rules.  - A wide section that connects the inputs directly to the outputs.    This allows the network to learn simple rules.<jupyter_code>input_  = keras.layers.Input(shape=X_train.shape[1:])
hidden1 = keras.layers.Dense(30, activation='relu')(input_)  # We call each layer like a function passing it the
                                                             # needed layer below it. This is why it's called
                                                             # the functional API.
hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)
concat  = keras.layers.Concatenate()([input_, hidden2])      # Input layer directly connecting to the output layer
                                                             # makes this a wide network.
                                                             # Having more than one hidden layer makes it deep.
output  = keras.layers.Dense(1)(concat)
model = keras.Model(inputs=[input_], outputs=[output])

# After this, the model is trained, evaluated, and used to predict as before. So we don't show that here.<jupyter_output><empty_output><jupyter_text>What if you wanted some features to go through the wide path,and some through the deep path? Let's do that now:<jupyter_code>input_A = keras.layers.Input(shape=[5], name='wide_input')    # 5 features through the wide path.
input_B = keras.layers.Input(shape=[6], name='deep_input')    # 6 features through the deep path.
                                                              # You should name the most important layers,
                                                              # especially when the model gets as complicated as this.
hidden1 = keras.layers.Dense(30, activation='relu')(input_B)
hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)
concat  = keras.layers.concatenate([input_A, hidden2])
output  = keras.layers.Dense(1, name='output')(concat)
model = keras.Model(inputs=[input_A, input_B], outputs=[output])
model.compile(loss='mse', optimizer=keras.optimizers.SGD(lr=1e-3))

X_train_A, X_train_B = X_train[:, :5], X_train[:, 2:]  # Data must have 8 columns. 
                                                       # This is why when we say 2: columns 2,3,4,5,6,7 are specified.
                                                       # This is correct since we need 6 columns for input b.
X_valid_A, X_valid_B = X_valid[:, :5], X_valid[:, 2:]
X_test_A,  X_test_B  = X_test[:, :5],  X_test[:, 2:]
X_new_A,   X_new_B   = X_test_A[:3],   X_test_B[:3]

history = model.fit((X_train_A, X_train_B), y_train, epochs=20,
                    validation_data=((X_valid_A, X_valid_B), y_valid))
mse_test = model.evaluate((X_test_A, X_test_B), y_test)
y_pred = model.predict((X_new_A, X_new_B))
print(f'y_pred: {y_pred.flatten().round(2)}\ny_new:  {y_new.flatten().round(2)}')<jupyter_output>y_pred: [0.71 1.8  2.83]
y_new:  [9 2 1]<jupyter_text>Sometimes, you may need multiple outputs:  - ex. You need to locate and classify the main picture in an image.        The location will be xy coords, and a bounding rectangle.        The classification will be another output.        So now we have x coord, y coord, width, height, classification as outputs  - Multiple independent tasks are needed on the same data.    You will get better results training a single network because the network    can learn features across tasks.  - As a regularization technique, you may want to add auxiliary output to one of the inner layers    to see if it is doing it's job.<jupyter_code>input_A = keras.layers.Input(shape=[5], name='wide_input')    # We will send 5 features through the wide path.
input_B = keras.layers.Input(shape=[6], name='deep_input')    # We will send 6 features through the deep path.
                                                              # You should name the most important layers,
                                                              # especially when the model gets as complicated as this.
hidden1 = keras.layers.Dense(30, activation='relu')(input_B)
hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)
concat  = keras.layers.concatenate([input_A, hidden2])
output  = keras.layers.Dense(1, name='output')(concat)
# The above code is the same as earlier.

aux_output = keras.layers.Dense(1, name='aux_output')(hidden2)
model = keras.Model(inputs=[input_A, input_B], outputs=[output, aux_output])

model.compile(loss=['mse', 'mse'], # Each output function needs it's own loss. 1st loss fn corresponds to 1st output?
              loss_weights=[0.9, 0.1], # We can give weights to each of the losses. Else Keras just adds the losses.
              optimizer='sgd')

history = model.fit(
                (X_train_A, X_train_B),
                (y_train, y_train),    # Both outputs need the labels (y-values). Since they should predict the same
                                       # thing, they're the same labels.
                epochs = 20,
                validation_data=([X_valid_A, X_valid_B], 
                                 [y_valid, y_valid]))  # Again, we need labels for both outputs here.

total_loss, main_loss, aux_loss = model.evaluate(  # Keras returns the total loss + the individual losses.
                [X_test_A, X_test_B],
                [y_test, y_test])

y_pred_main, y_pred_aux = model.predict([X_new_A, X_new_B])  # We will get predictions for each output.
y_new_A, y_new_B = y_test[:3], y_test[:3]
print(f'y_pred_main: {y_pred_main.flatten().round(2)}\ny_new_A:     {y_new_A.flatten().round(2)}')
print(f'y_pred_aux:  {y_pred_aux.flatten().round(2)}\ny_new_B:     {y_new_B.flatten().round(2)}')<jupyter_output>y_pred_main: [0.55 1.32 3.61]
y_new_A:     [0.48 0.46 5.  ]
y_pred_aux:  [0.65 1.65 2.75]
y_new_B:     [0.48 0.46 5.  ]<jupyter_text>Using the subclassing API to build Dynamic Models Both the Sequential an Functional APIs are declarative. Advantages of this are:  - model can be easily saved, cloned, shared  - model structure can be displayed and analyzed  - framework can infer shapes and check types (for error checking)  - easy to debug  Some models involve:  - loops  - varying shapes  - conditional branchingFor these models, you can use the Subclassing API<jupyter_code>class WideAndDeepModel(keras.Model):
    def __init__(self, units=30, activation='relu', **kwargs):
        super().__init__(kwargs) # handles standard args - ex. name
        self.hidden1 = keras.layers.Dense(units, activation=activation)  # Create the layers in the constructor
        self.hidden2 = keras.layers.Dense(units, activation=activation)
        self.main_output = keras.layers.Dense(1)
        self.aux_output  = keras.layers.Dense(1)
        
    def call(self, inputs):
        input_A, input_B = inputs
        hidden1 = self.hidden1(input_B)  # Use the layers as you want in the call() function.
                                         # In this call() function, you can use 
                                         #   - for loops
                                         #   - if statements
                                         #   - low-level tensorflow operations
                                         # This is very useful for researchers experimenting with new ideas.
        hidden2 = self.hidden2(hidden1)
        concat  = keras.layers.concatenate([input_A, hidden2])
        main_output = self.main_output(concat)
        aux_output = self.aux_output(hidden2)

# This WideAndDeepModel provides the same functionality as the functional API.
# You can compile it, evaluate it, and make predictions as was done earlier.
# Cost:
#   - Model architecture is hidden within the call() method, so Keras cannot save/clone it.
#   - summary() returns just a list of layers but not how they're connected to each other.
#   - Keras cannot check types/shapes ahead of time.
# Any Keras Model can be used just like a layer.
# You can easily combine them to build complex architectures.<jupyter_output><empty_output><jupyter_text>Saving and Restoring a Model<jupyter_code>model_dir = './data'
model_filename = os.path.join('.', model_dir, 'my_keras_model.h5')

model.save(model_filename)       # Saves the model in the HDF5 format, including:
                                 #   - the model architecture
                                 #   - each layer's parameters
                                 #   - connection weights and biases
                                 #   - the optimizer, it's parameters, and it's state.
                                 # This will not work when using model subclassing. For subclassing,
                                 #   you can use save_weights() and load_weights(), but you need to
                                 #   save and restore everything else yourself.
model = keras.models.load_model(model_filename)
model<jupyter_output><empty_output><jupyter_text>Using Callbacks<jupyter_code># Training can take hours sometimes, and your computer may crash.
# You should save your model at intervals while it is training using callbacks.
# Let's build the model using the functional API and use callbacks in fit().

input_  = keras.layers.Input(shape=X_train.shape[1:])
hidden1 = keras.layers.Dense(30, activation='relu')(input_)  # We call each layer like a function passing it the
                                                             # needed layer below it. This is why it's called
                                                             # the functional API.
hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)
concat  = keras.layers.Concatenate()([input_, hidden2])
output  = keras.layers.Dense(1)(concat)
model = keras.Model(inputs=[input_], outputs=[output])

checkpoint_cb = keras.callbacks.ModelCheckpoint(
                    model_filename,
                    save_best_only=True)       # If you specify validation set, save_best_only=True saves only if the
                                               # current model is better on the validation set than the earlier models.

model.compile(loss=['mse'], loss_weights=[0.9], optimizer='sgd')

history = model.fit(X_train, y_train, epochs=10, 
                    callbacks=[checkpoint_cb], # Saves model at the end of each epoch
                    validation_data=(X_valid, y_valid))
model = keras.models.load_model(model_filename) # Load the best model that was saved during the fit()
# Combine early stopping and checkpoint callbacks.
early_stopping_cb = keras.callbacks.EarlyStopping(patience=10, # Stops if the model is not getting better on the
                                                               # validation set for the last 10 epochs
                                                               # (as given by patience).
                                                  restore_best_weights=True) # After training is done, roll back
                                                                             # to the best weights for the model.
history = model.fit(X_train, y_train, epochs=100,  # The number of epochs can be large since we're stopping early.
                    validation_data=(X_valid, y_valid),
                    callbacks=[checkpoint_cb, early_stopping_cb]) # Saves model at the end of each epoch (checkpoints)
                                                                  # and interrupts training if there is no progress.
# Create your own callback
class PrintValTrainRatioCallback(keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs):
        print('val/train: {:.2f}'.format(logs['val_loss'] / logs['loss']))
# Callbacks during fit():
#   on_train_begin(), on_train_end(), on_epoch_begin(), on_epoch_end(), on_batch_begin(), and on_batch_end().
# Callbacks during evaluate():
#   on_test_begin(), on_test_end(), on_test_batch_begin(), or on_test_batch_end()
# Callbacks during predict():
#   on_predict_begin(), on_predict_end(), on_predict_batch_begin(), or on_predict_batch_end()<jupyter_output><empty_output><jupyter_text>Using TensorBoard for Visualization<jupyter_code># TensorBoard allows you to:
#   - View learning curves during training
#   - Compare learning curves between multiple runs
#   - Visualize the computation graph
#   - Analyze training statistics
#   - View images generated by your model
#   - View complex multi-dimensional data projected down to 3-D, and automatically clustered for you
# You must output the data to event files (special binary log files).
# Point the TensorBoard server to the root of the log directories.
# Configure your code to write to a different subdirectory each time it runs.

import os 
root_logdir = os.path.join(os.curdir, "my_logs") 

def get_run_logdir (): 
    import time 
    run_id = time.strftime("run_%Y_%m_%d%H_%M_%S") 
    return os.path.join(root_logdir , run_id)

run_logdir = get_run_logdir() # e.g., './my_logs/run_2019_06_07-15_15_22' 

tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)
history = model.fit(X_train, y_train, epochs = 30, 
                    validation_data=(X_valid, y_valid), 
                    callbacks=[tensorboard_cb]) 

# Then start the tensorboard server:
#   tensorboard --logdir=./my_logs --port=6006
# Open browser at localhost:6006
# To use tensorboard directly within Jupyter notebook:
#   %load_ext tensorboard 
#   %tensorboard logdir=./my_logs port=6006 
#
# Tensorflow also offers a lower-level API in the tf.summary package.
# It allows you to log extra data, such as embeddings.<jupyter_output>Train on 11610 samples, validate on 3870 samples
Epoch 1/30
11610/11610 [==============================] - 1s 48us/sample - loss: 0.2780 - val_loss: 0.3235
Epoch 2/30
11610/11610 [==============================] - 1s 47us/sample - loss: 0.2782 - val_loss: 0.3443
Epoch 3/30
11610/11610 [==============================] - 1s 46us/sample - loss: 0.3689 - val_loss: 0.3487
Epoch 4/30
11610/11610 [==============================] - 1s 47us/sample - loss: 0.2821 - val_loss: 0.3481
Epoch 5/30
11610/11610 [==============================] - 1s 47us/sample - loss: 0.4302 - val_loss: 0.3170
Epoch 6/30
11610/11610 [==============================] - 1s 48us/sample - loss: 0.2777 - val_loss: 0.3996
Epoch 7/30
11610/11610 [==============================] - 1s 46us/sample - loss: 0.6877 - val_loss: 0.3391
Epoch 8/30
11610/11610 [==============================] - 1s 46us/sample - loss: 0.4386 - val_loss: 0.3061
Epoch 9/30
11610/11610 [==============================] - 1s 47us/sample - loss: 0.2707 - val_l[...]
{'fraction_non_alphanumeric': 0.08786002089240412, 'fraction_numerical': 0.01984778391284883, 'mean_word_length': 1.7702563042579578, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 200, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '336520', 'license': 'MIT', 'n_tokens_mistral': 7108, 'n_tokens_neox': 6644, 'n_words': 2242, 'path': 'keras_api.ipynb', 'repo_name': 'samirgadkari/ds', 'size': '26818'}