<jupyter_start><jupyter_text>T81-558: Applications of Deep Neural Networks**Module 13: Advanced/Other Topics*** Instructor: [<NAME>](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).  Module 13 Video Material* Part 13.1: Flask and Deep Learning Web Services [[Video]](https://www.youtube.com/watch?v=H73m9XvKHug&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_01_flask.ipynb)* **Part 13.2: Interrupting and Continuing Training**  [[Video]](https://www.youtube.com/watch?v=8ygCyvRZ074&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_02_cloud.ipynb)* Part 13.3: Using a Keras Deep Neural Network with a Web Application  [[Video]](https://www.youtube.com/watch?v=OBbw0e-UroI&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_03_web.ipynb)* Part 13.4: When to Retrain Your Neural Network [[Video]](https://www.youtube.com/watch?v=K2Tjdx_1v9g&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_04_retrain.ipynb)* Part 13.5: AI at the Edge: Using Keras on a Mobile Device  [[Video]]() [[Notebook]](t81_558_class_13_05_edge.ipynb)  Google CoLab InstructionsThe following code ensures that Google CoLab is running the correct version of TensorFlow.<jupyter_code>try:
    from google.colab import drive
    drive.mount('/content/drive', force_remount=True)
    COLAB = True
    print("Note: using Google CoLab")
    %tensorflow_version 2.x
except:
    print("Note: not using Google CoLab")
    COLAB = False<jupyter_output><empty_output><jupyter_text>Part 13.2: Interrupting and Continuing TrainingIn this part we will see that we can stop and continue training later. This continuation is accomplished by using checkpoints. We begin by importing needed packages.<jupyter_code>import os
import re
import sys
import time
from typing import Any, List, Tuple, Union
import tensorflow as tf
import tensorflow.keras
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras import regularizers
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Flatten
from tensorflow.keras.layers import Conv2D, MaxPooling2D
from tensorflow.keras import backend as K
from tensorflow.keras.callbacks import ModelCheckpoint<jupyter_output><empty_output><jupyter_text>Next, we create several utility functions. First, one to generated an output directory that has a unique name. This technique allows us to organize multiple runs of our experiment. The Logger class is provided to route output to a log file that is stored in the output directory.<jupyter_code>def generate_output_dir(outdir, run_desc):
    prev_run_dirs = []
    if os.path.isdir(outdir):
        prev_run_dirs = [x for x in os.listdir(outdir) if os.path.isdir(os.path.join(outdir, x))]
    prev_run_ids = [re.match(r'^\d+', x) for x in prev_run_dirs]
    prev_run_ids = [int(x.group()) for x in prev_run_ids if x is not None]
    cur_run_id = max(prev_run_ids, default=-1) + 1
    run_dir = os.path.join(outdir, f'{cur_run_id:05d}-{run_desc}')
    assert not os.path.exists(run_dir)
    os.makedirs(run_dir)
    return run_dir

# From StyleGAN2
class Logger(object):
    """Redirect stderr to stdout, optionally print stdout to a file, and optionally force flushing on both stdout and the file."""

    def __init__(self, file_name: str = None, file_mode: str = "w", should_flush: bool = True):
        self.file = None

        if file_name is not None:
            self.file = open(file_name, file_mode)

        self.should_flush = should_flush
        self.stdout = sys.stdout
        self.stderr = sys.stderr

        sys.stdout = self
        sys.stderr = self

    def __enter__(self) -> "Logger":
        return self

    def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
        self.close()

    def write(self, text: str) -> None:
        """Write text to stdout (and a file) and optionally flush."""
        if len(text) == 0: # workaround for a bug in VSCode debugger: sys.stdout.write(''); sys.stdout.flush() => crash
            return

        if self.file is not None:
            self.file.write(text)

        self.stdout.write(text)

        if self.should_flush:
            self.flush()

    def flush(self) -> None:
        """Flush written text to both stdout and a file, if open."""
        if self.file is not None:
            self.file.flush()

        self.stdout.flush()

    def close(self) -> None:
        """Flush, close possible files, and remove stdout/stderr mirroring."""
        self.flush()

        # if using multiple loggers, prevent closing in wrong order
        if sys.stdout is self:
            sys.stdout = self.stdout
        if sys.stderr is self:
            sys.stderr = self.stderr

        if self.file is not None:
            self.file.close()<jupyter_output><empty_output><jupyter_text>We define the basic training parameters and where we wish to write the output to.<jupyter_code>outdir = "./data/"
run_desc = "test-train"
batch_size = 128
num_classes = 10
epochs = 12

run_dir = generate_output_dir(outdir, run_desc)
print(f"Results saved to: {run_dir}")<jupyter_output>Results saved to: ./data/00000-test-train<jupyter_text>We now obtain the data, build the model and begin training. The program will write checkpoint files to track the training progress of the neural network.<jupyter_code>def obtain_data():
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    print("Shape of x_train: {}".format(x_train.shape))
    print("Shape of y_train: {}".format(y_train.shape))
    print()
    print("Shape of x_test: {}".format(x_test.shape))
    print("Shape of y_test: {}".format(y_test.shape))

    # input image dimensions
    img_rows, img_cols = 28, 28
    if K.image_data_format() == 'channels_first':
        x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
        x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
        input_shape = (1, img_rows, img_cols)
    else:
        x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
        x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
        input_shape = (img_rows, img_cols, 1)
    x_train = x_train.astype('float32')
    x_test = x_test.astype('float32')
    x_train /= 255
    x_test /= 255
    print('x_train shape:', x_train.shape)
    print("Training samples: {}".format(x_train.shape[0]))
    print("Test samples: {}".format(x_test.shape[0]))
    # convert class vectors to binary class matrices
    y_train = tensorflow.keras.utils.to_categorical(y_train, num_classes)
    y_test = tensorflow.keras.utils.to_categorical(y_test, num_classes)
    
    return input_shape, x_train, y_train, x_test, y_test

def build_model(input_shape, num_classes):
    model = Sequential()
    model.add(Conv2D(32, kernel_size=(3, 3),
                     activation='relu',
                     input_shape=input_shape))
    model.add(Conv2D(64, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(num_classes, activation='softmax'))
    model.compile(loss='categorical_crossentropy', optimizer='adam',
                  metrics=['accuracy'])
    return model

def load_model(path):
    model = load_model(path)
    epoch = int(path.split('.')[1])
    return epoch, model

def train_model(model):
    start_time = time.time()

    ckpt_filename = os.path.join(run_dir, 'weights.{epoch:02d}-{val_loss:.2f}.hdf5')
    ckpt_callback = ModelCheckpoint(ckpt_filename, monitor='val_loss',verbose=1)
    callbacks = [ckpt_callback]

    model.fit(x_train, y_train,
              batch_size=batch_size,
              epochs=epochs,
              verbose=2, callbacks=callbacks,
              validation_data=(x_test, y_test))
    score = model.evaluate(x_test, y_test, verbose=0, callbacks=callbacks)
    print('Test loss: {}'.format(score[0]))
    print('Test accuracy: {}'.format(score[1]))

    elapsed_time = time.time() - start_time
    print("Elapsed time: {}".format(hms_string(elapsed_time)))<jupyter_output><empty_output><jupyter_text>We now begin training, using the Logger class to write the output to a log file in the output directory.<jupyter_code>with Logger(os.path.join(run_dir, 'log.txt')):
    input_shape, x_train, y_train, x_test, y_test = obtain_data()
    model = build_model(input_shape, num_classes)
    train_model(model)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11524459613196815, 'fraction_numerical': 0.021160409556313993, 'mean_word_length': 3.8811771238201, 'pattern_counts': {'":': 1, '<': 25, '<?xml version=': 0, '>': 31, 'https://': 7, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '322766', 'license': 'Apache-2.0', 'n_tokens_mistral': 3094, 'n_tokens_neox': 2933, 'n_words': 735, 'path': 't81_558_class_13_02_checkpoint.ipynb', 'repo_name': 'Maelstrom-Portal/t81_558_deep_learning', 'size': '8795'}