<jupyter_start><jupyter_text>Classifying Images with pre-built TF Container on Vertex AIThis notebook demonstrates how to implement different image models on MNIST using the [tf.keras API](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras). Learning Objectives1. Understand how to build a Dense Neural Network (DNN) for image classification.2. Understand how to use dropout (DNN) for image classification.3. Understand how to use Convolutional Neural Networks (CNN).4. Know how to deploy and use an image classifcation model using Google Cloud's [Vertex AI](https://cloud.google.com/vertex-ai/).Each learning objective will correspond to a __TODO__ in the [student lab notebook](../labs/classifying_images_with_pre-built_tf_container_on_vertex_ai.ipynb) -- try to complete that notebook first before reviewing this solution notebook.First things first. Configure the parameters below to match your own Google Cloud project details. If you don’t want to change the region, leave all settings as they are.  Otherwise, update the region as you want, leave other settings as they are and run the cell.<jupyter_code>from datetime import datetime
import os

REGION = 'us-central1'
PROJECT = !(gcloud config get-value core/project)
PROJECT = PROJECT[0]
BUCKET =  PROJECT
MODEL_TYPE = "cnn"  # "linear", "dnn", "dnn_dropout", or "cnn"

# Do not change these
os.environ["PROJECT"] = PROJECT
os.environ["BUCKET"] = BUCKET
os.environ["REGION"] = REGION
os.environ["MODEL_TYPE"] = MODEL_TYPE<jupyter_output><empty_output><jupyter_text>Building a dynamic modelThis notebook demonstrates how to implement DNN and CNN models on [MNIST](http://yann.lecun.com/exdb/mnist/) using the [tf.keras API](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras). In the previous notebook, “[Classifying_Images_with_DNN_Model.ipynb](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/deepdive2/computer_vision_fun/labs/7_Classifying_Images_with_DNN_Model.ipynb)”, you ran our code directly from the notebook. In order to run it on Vertex AI, you can also package your notebook as a python module.The boilerplate structure for this module has already been set up in the folder `mnist_models`. The module lives in the sub-folder, `trainer`, and is designated as a python package with the empty `__init__.py` (`mnist_models/trainer/__init__.py`) file. It still needs the model and a trainer to run it, so let's make them.Let's start with the trainer file first. This file parses command line arguments to feed into the model.<jupyter_code>%%writefile mnist_models/trainer/task.py
import argparse
import json
import os
import sys

from . import model


def _parse_arguments(argv):
    """Parses command-line arguments."""
    parser = argparse.ArgumentParser()
    parser.add_argument(
        '--model_type',
        help='Which model type to use',
        type=str, default='linear')
    parser.add_argument(
        '--epochs',
        help='The number of epochs to train',
        type=int, default=10)
    parser.add_argument(
        '--steps_per_epoch',
        help='The number of steps per epoch to train',
        type=int, default=100)
    parser.add_argument(
        '--job-dir',
        help='Directory where to save the given model',
        type=str, default='mnist_models/')
    return parser.parse_known_args(argv)


def main():
    """Parses command line arguments and kicks off model training."""
    args = _parse_arguments(sys.argv[1:])[0]

    # Configure path for hyperparameter tuning.
    trial_id = json.loads(
        os.environ.get('TF_CONFIG', '{}')).get('task', {}).get('trial', '')
    output_path = args.job_dir if not trial_id else args.job_dir + '/'

    model_layers = model.get_layers(args.model_type)
    image_model = model.build_model(model_layers, args.job_dir)
    model_history = model.train_and_evaluate(
        image_model, args.epochs, args.steps_per_epoch, args.job_dir)


if __name__ == '__main__':
    main()<jupyter_output>Writing mnist_models/trainer/task.py<jupyter_text>Next, let's group non-model functions into a util file to keep the model file simple. You'll copy over the `scale` and `load_dataset` functions from the previous lab.<jupyter_code>%%writefile mnist_models/trainer/util.py
import tensorflow as tf


def scale(image, label):
    """Scales images from a 0-255 int range to a 0-1 float range"""
    image = tf.cast(image, tf.float32)
    image /= 255
    image = tf.expand_dims(image, -1)
    return image, label


def load_dataset(
        data, training=True, buffer_size=5000, batch_size=100, nclasses=10):
    """Loads MNIST dataset into a tf.data.Dataset"""
    (x_train, y_train), (x_test, y_test) = data
    x = x_train if training else x_test
    y = y_train if training else y_test
    # One-hot encode the classes
    y = tf.keras.utils.to_categorical(y, nclasses)
    dataset = tf.data.Dataset.from_tensor_slices((x, y))
    dataset = dataset.map(scale).batch(batch_size)
    if training:
        dataset = dataset.shuffle(buffer_size).repeat()
    return dataset<jupyter_output>Writing mnist_models/trainer/util.py<jupyter_text>Finally, let's code the models! The tf.keras API accepts an array of layers into a model object, so you can create a dictionary of layers based on the different model types you want to use. mnist_models/trainer/model.py file has three functions: get_layers, build_model and train_and_evaluate. In get_layers function: You will build the structure of our model in get_layers with four different layers:* First, you define a linear model.* Second, you define the Keras layers for a DNN model.* Third, you define the Keras layers for a dropout model.* Lastly, you define the Keras layers for a CNN model.In the build_model function: You compile the model, specifying an optimizer to use, the loss to minimize, and metrics to report. Finally in the train_and_evaluate function, you compile your keras model by loading data into it for training.Note that these models progressively build on each other. Look at the imported tensorflow.keras.layers modules and the default values for the variables defined in get_layers for guidance.<jupyter_code>%%writefile mnist_models/trainer/model.py
import os
import shutil

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.callbacks import TensorBoard
from tensorflow.keras.layers import (
    Conv2D, Dense, Dropout, Flatten, MaxPooling2D, Softmax)

from . import util


# Image Variables
WIDTH = 28
HEIGHT = 28


def get_layers(
        model_type,
        nclasses=10,
        hidden_layer_1_neurons=400,
        hidden_layer_2_neurons=100,
        dropout_rate=0.25,
        num_filters_1=64,
        kernel_size_1=3,
        pooling_size_1=2,
        num_filters_2=32,
        kernel_size_2=3,
        pooling_size_2=2):
    """Constructs layers for a keras model based on a dict of model types."""
    model_layers = {
        'linear': [
            Flatten(),
            Dense(nclasses),
            Softmax()
        ],
        # TODO 1: Define the Keras layers for a DNN model
        'dnn': [
            Flatten(),
            Dense(hidden_layer_1_neurons, activation='relu'),
            Dense(hidden_layer_2_neurons, activation='relu'),
            Dense(nclasses),
            Softmax()
        ],
        # TODO 2: Define the Keras layers for a dropout model
        'dnn_dropout': [
            Flatten(),
            Dense(hidden_layer_1_neurons, activation='relu'),
            Dense(hidden_layer_2_neurons, activation='relu'),
            Dropout(dropout_rate),
            Dense(nclasses),
            Softmax()
        ],
        # TODO 3: Define the Keras layers for a CNN model
        'cnn': [
            Conv2D(num_filters_1, kernel_size=kernel_size_1,
                   activation='relu', input_shape=(WIDTH, HEIGHT, 1)),
            MaxPooling2D(pooling_size_1),
            Conv2D(num_filters_2, kernel_size=kernel_size_2,
                   activation='relu'),
            MaxPooling2D(pooling_size_2),
            Flatten(),
            Dense(hidden_layer_1_neurons, activation='relu'),
            Dense(hidden_layer_2_neurons, activation='relu'),
            Dropout(dropout_rate),
            Dense(nclasses),
            Softmax()
        ]
    }
    return model_layers[model_type]


def build_model(layers, output_dir):
    """Compiles keras model for image classification."""
    model = Sequential(layers)
    model.compile(optimizer='adam',
                  loss='categorical_crossentropy',
                  metrics=['accuracy'])
    return model


def train_and_evaluate(model, num_epochs, steps_per_epoch, output_dir):
    """Compiles keras model and loads data into it for training."""
    mnist = tf.keras.datasets.mnist.load_data()
    train_data = util.load_dataset(mnist)
    validation_data = util.load_dataset(mnist, training=False)

    callbacks = []
    if output_dir:
        tensorboard_callback = TensorBoard(log_dir=output_dir)
        callbacks = [tensorboard_callback]

    history = model.fit(
        train_data,
        validation_data=validation_data,
        epochs=num_epochs,
        steps_per_epoch=steps_per_epoch,
        verbose=2,
        callbacks=callbacks)

    if output_dir:
        export_path = os.path.join(output_dir, 'keras_export')
        model.save(export_path, save_format='tf')

    return history<jupyter_output>Writing mnist_models/trainer/model.py<jupyter_text>Local TrainingWith everything set up, let's run locally to test the code. Some of the previous tests have been copied over into a testing script `mnist_models/trainer/test.py` to make sure the model still passes our previous checks. On `line 13`, you can specify which model types you would like to check. `line 14` and `line 15` has the number of epochs and steps per epoch respectively.Moment of truth! Run the code below to check your models against the unit tests. If you see "OK" at the end when it's finished running, congrats! You've passed the tests!<jupyter_code>!python3 -m mnist_models.trainer.test<jupyter_output>2022-01-10 11:10:26.956391: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2022-01-10 11:10:27.032398: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
..
*** Building model for linear ***

Epoch 1/10
100/100 - 6s - loss: 1.3205 - accuracy: 0.6661 - val_loss: 0.7883 - val_accuracy: 0.8210
Epoch 2/10
100/100 - 1s - loss: 0.6711 - accuracy: 0.8396 - val_loss: 0.5605 - val_accuracy: 0.8658
Epoch 3/10
100/100 - 1s - loss: 0.5260 - accuracy: 0.8742 - val_loss: 0.4689 - val_accuracy: 0.8826
Epoch 4/10
100/100 - 2s - loss: 0.4703 - accuracy: 0.8773 - val_loss: 0.4196 - val_accuracy: 0.8934
Epoch 5/10
100/100 - 1s - loss: 0.4305 - accuracy: 0.8874 - val_loss: 0.3920 - val_accuracy: 0.8960
Epoch 6/10
100/100 - 2s - loss: 0.3896 - accuracy: 0.8963 - val_loss: 0.3653 - val_accuracy: 0.9042
Epo[...]<jupyter_text>Now you know that your models are working as expected, let's run it on Google Cloud within Vertex AI. You can run it as a python module locally first using the command line.The below cell transfers some of your variables to the command line as well as create a job directory including a timestamp.<jupyter_code>current_time = datetime.now().strftime("%Y%m%d_%H%M%S")
model_type = 'cnn'

os.environ["MODEL_TYPE"] = model_type
os.environ["JOB_DIR"] = "mnist_models/models/{}_{}/".format(
    model_type, current_time)<jupyter_output><empty_output><jupyter_text>The cell below runs the local version of the code. The epochs and steps_per_epoch flag can be changed to run for longer or shorther, as defined in your `mnist_models/trainer/task.py` file.<jupyter_code>%%bash
python3 -m mnist_models.trainer.task \
    --job-dir=$JOB_DIR \
    --epochs=5 \
    --steps_per_epoch=50 \
    --model_type=$MODEL_TYPE<jupyter_output>Epoch 1/5
50/50 - 9s - loss: 1.0487 - accuracy: 0.6644 - val_loss: 0.3334 - val_accuracy: 0.8990
Epoch 2/5
50/50 - 5s - loss: 0.3760 - accuracy: 0.8886 - val_loss: 0.2216 - val_accuracy: 0.9332
Epoch 3/5
50/50 - 4s - loss: 0.2188 - accuracy: 0.9376 - val_loss: 0.1374 - val_accuracy: 0.9590
Epoch 4/5
50/50 - 5s - loss: 0.1550 - accuracy: 0.9534 - val_loss: 0.1135 - val_accuracy: 0.9678
Epoch 5/5
50/50 - 5s - loss: 0.1457 - accuracy: 0.9554 - val_loss: 0.0957 - val_accuracy: 0.9680<jupyter_text>Training on the cloudFor this model, you will be able to use a Tensorflow pre-built container on Vertex AI, as you do not have any particular additional prerequisites. You use setuptools for this, and store the created source distribution on Cloud Storage by using “**gsutil cp**”.<jupyter_code>%%writefile mnist_models/setup.py
from setuptools import find_packages
from setuptools import setup

setup(
    name='mnist_trainer',
    version='0.1',
    packages=find_packages(),
    include_package_data=True,
    description='MNIST model training application.'
)
%%bash
cd mnist_models
python ./setup.py sdist --formats=gztar
cd ..
gsutil cp mnist_models/dist/mnist_trainer-0.1.tar.gz gs://${BUCKET}/mnist/<jupyter_output>running sdist
running egg_info
creating mnist_trainer.egg-info
writing mnist_trainer.egg-info/PKG-INFO
writing dependency_links to mnist_trainer.egg-info/dependency_links.txt
writing top-level names to mnist_trainer.egg-info/top_level.txt
writing manifest file 'mnist_trainer.egg-info/SOURCES.txt'
reading manifest file 'mnist_trainer.egg-info/SOURCES.txt'
writing manifest file 'mnist_trainer.egg-info/SOURCES.txt'
running check
creating mnist_trainer-0.1
creating mnist_trainer-0.1/mnist_trainer.egg-info
creating mnist_trainer-0.1/trainer
copying files to mnist_trainer-0.1...
copying setup.py -> mnist_trainer-0.1
copying mnist_trainer.egg-info/PKG-INFO -> mnist_trainer-0.1/mnist_trainer.egg-info
copying mnist_trainer.egg-info/SOURCES.txt -> mnist_trainer-0.1/mnist_trainer.egg-info
copying mnist_trainer.egg-info/dependency_links.txt -> mnist_trainer-0.1/mnist_trainer.egg-info
copying mnist_trainer.egg-info/top_level.txt -> mnist_trainer-0.1/mnist_trainer.egg-info
copying trainer/__init__.p[...]<jupyter_text>Then, you can start the Vertex AI Custom Job using the pre-built container. You can pass your source distribution URI using the `--python-package-uris` flag.<jupyter_code>current_time = datetime.now().strftime("%Y%m%d_%H%M%S")
model_type = 'cnn'

os.environ["MODEL_TYPE"] = model_type
os.environ["JOB_DIR"] = "gs://{}/mnist_{}_{}/".format(
    BUCKET, model_type, current_time)
os.environ["JOB_NAME"] = "mnist_{}_{}".format(
    model_type, current_time)<jupyter_output><empty_output><jupyter_text>After submitting the following job, view the status in __Vertex AI__ > __Training__ and select __Custom Jobs__ tab. Wait for the job to finish.<jupyter_code>%%bash
echo $JOB_DIR $REGION $JOB_NAME

PYTHON_PACKAGE_URIS=gs://${BUCKET}/mnist/mnist_trainer-0.1.tar.gz
MACHINE_TYPE=n1-standard-4
REPLICA_COUNT=1
PYTHON_PACKAGE_EXECUTOR_IMAGE_URI="us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-3:latest"
PYTHON_MODULE=trainer.task
    
WORKER_POOL_SPEC="machine-type=$MACHINE_TYPE,\
replica-count=$REPLICA_COUNT,\
executor-image-uri=$PYTHON_PACKAGE_EXECUTOR_IMAGE_URI,\
python-module=$PYTHON_MODULE"

gcloud ai custom-jobs create \
  --region=${REGION} \
  --display-name=$JOB_NAME \
  --python-package-uris=$PYTHON_PACKAGE_URIS \
  --worker-pool-spec=$WORKER_POOL_SPEC \
  --args="--job-dir=$JOB_DIR,--model_type=$MODEL_TYPE"
%%bash
SAVEDMODEL_DIR=${JOB_DIR}keras_export
echo $SAVEDMODEL_DIR
gsutil ls $SAVEDMODEL_DIR<jupyter_output>gs://qwiklabs-gcp-04-25783cd152f3/mnist_cnn_20220110_112058/keras_export
gs://qwiklabs-gcp-04-25783cd152f3/mnist_cnn_20220110_112058/keras_export/
gs://qwiklabs-gcp-04-25783cd152f3/mnist_cnn_20220110_112058/keras_export/saved_model.pb
gs://qwiklabs-gcp-04-25783cd152f3/mnist_cnn_20220110_112058/keras_export/assets/
gs://qwiklabs-gcp-04-25783cd152f3/mnist_cnn_20220110_112058/keras_export/variables/<jupyter_text>Deploying and predicting with modelOnce you have a model you're proud of, let's deploy it! All you need to do is to upload the created model artifact from Cloud Storage to Vertex AI as a model, create a new endpoint, and deploy the model to the endpoint. It can take 15-20 minutes to complete. Make a note of __ENDPOINT_RESOURCENAME__ for further use.<jupyter_code>%%bash
TIMESTAMP=$(date -u +%Y%m%d_%H%M%S)
MODEL_DISPLAYNAME=mnist_$TIMESTAMP
ENDPOINT_DISPLAYNAME=mnist_endpoint_$TIMESTAMP
IMAGE_URI="us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-3:latest"
SAVEDMODEL_DIR=${JOB_DIR}keras_export
echo $SAVEDMODEL_DIR

# Model
MODEL_RESOURCENAME=$(gcloud ai models upload \
    --region=$REGION \
    --display-name=$MODEL_DISPLAYNAME \
    --container-image-uri=$IMAGE_URI \
    --artifact-uri=$SAVEDMODEL_DIR \
    --format="value(model)")

echo "MODEL_DISPLAYNAME=${MODEL_DISPLAYNAME}"
echo "MODEL_RESOURCENAME=${MODEL_RESOURCENAME}"

# Endpoint
ENDPOINT_RESOURCENAME=$(gcloud ai endpoints create \
  --region=$REGION \
  --display-name=$ENDPOINT_DISPLAYNAME \
  --format="value(name)")

echo "ENDPOINT_DISPLAYNAME=${ENDPOINT_DISPLAYNAME}"
echo "ENDPOINT_RESOURCENAME=${ENDPOINT_RESOURCENAME}"

# Deployment
DEPLOYED_MODEL_DISPLAYNAME=${MODEL_DISPLAYNAME}_deployment
MACHINE_TYPE=n1-standard-2

gcloud ai endpoints deploy-model $ENDPOINT_RESOURCENAME \
  --region=$REGION \
  --model=$MODEL_RESOURCENAME \
  --display-name=$DEPLOYED_MODEL_DISPLAYNAME \
  --machine-type=$MACHINE_TYPE \
  --min-replica-count=1 \
  --max-replica-count=1 \
  --traffic-split=0=100<jupyter_output>gs://qwiklabs-gcp-04-25783cd152f3/mnist_cnn_20220110_112058/keras_export
MODEL_DISPLAYNAME=mnist_20220110_113117
MODEL_RESOURCENAME=projects/867925218804/locations/us-central1/models/5397040785868718080
ENDPOINT_DISPLAYNAME=mnist_endpoint_20220110_113117
ENDPOINT_RESOURCENAME=projects/867925218804/locations/us-central1/endpoints/2075394168124866560<jupyter_text>To predict with the model, let's take one of the example images.Write a .json file with image data to send to a Vertex AI deployed model.<jupyter_code>import json, codecs
import tensorflow as tf
import matplotlib.pyplot as plt

HEIGHT = 28
WIDTH = 28
IMGNO = 12

mnist = tf.keras.datasets.mnist.load_data()
(x_train, y_train), (x_test, y_test) = mnist
test_image = x_test[IMGNO]

# TODO 4: Prepare jsondata
jsondata = {"instances": [
    test_image.reshape(HEIGHT, WIDTH, 1).tolist()
]}
json.dump(jsondata, codecs.open("test.json", "w", encoding = "utf-8"))
plt.imshow(test_image.reshape(HEIGHT, WIDTH));
!cat test.json<jupyter_output>{"instances": [[[[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0]], [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [0], [[...]<jupyter_text>Finally, you can send it to the prediction service. The output will have a 1 in the index of the corresponding digit it is predicting. Congrats! You've completed the lab!<jupyter_code>%%bash
ENDPOINT_RESOURCENAME=#Insert ENDPOINT_RESOURCENAME from above

gcloud ai endpoints predict $ENDPOINT_RESOURCENAME \
    --region=$REGION \
    --json-request=test.json<jupyter_output>[[0, 0, 0, 0, 0, 0, 0, 0, 0, 1]]
{'fraction_non_alphanumeric': 0.12678429817605075, 'fraction_numerical': 0.047977795400475816, 'mean_word_length': 4.144569097399286, 'pattern_counts': {'":': 2, '<': 43, '<?xml version=': 0, '>': 49, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '59286', 'license': 'Apache-2.0', 'n_tokens_mistral': 7600, 'n_tokens_neox': 7028, 'n_words': 1881, 'path': 'courses/machine_learning/deepdive2/computer_vision_fun/solutions/classifying_images_with_pre-built_tf_container_on_vertex_ai.ipynb', 'repo_name': 'kalona/training-data-analyst', 'size': '20176'}