<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License. ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-keras/train-hyperparameter-tune-deploy-with-keras.png)  Training, hyperparameter tune, and deploy with Keras (Part 1) IntroductionThis tutorial shows how to train a simple deep neural network using the MNIST dataset and Keras on Azure Machine Learning. MNIST is a popular dataset consisting of 70,000 grayscale images. Each image is a handwritten digit of `28x28` pixels, representing number from 0 to 9. The goal is to create a multi-class classifier to identify the digit each image represents, and deploy it as a web service in Azure.For more information about the MNIST dataset, please visit [Yan LeCun's website](http://yann.lecun.com/exdb/mnist/). Prerequisite:* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:    * install the AML SDK    * create a workspace and its configuration file (`config.json`)* For local scoring test, you will also need to have `tensorflow` and `keras` installed in the current Jupyter kernel. Let's get started. First let's import some Python libraries.<jupyter_code>%matplotlib inline
import numpy as np
import os
import matplotlib.pyplot as plt
import azureml
from azureml.core import Workspace

# check core SDK version number
print("Azure ML SDK Version: ", azureml.core.VERSION)<jupyter_output><empty_output><jupyter_text>Initialize workspaceInitialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architectureworkspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`.<jupyter_code>ws = Workspace.from_config()
print('Workspace name: ' + ws.name, 
      'Azure region: ' + ws.location, 
      'Subscription id: ' + ws.subscription_id, 
      'Resource group: ' + ws.resource_group, sep='\n')<jupyter_output><empty_output><jupyter_text>Create an Azure ML experimentLet's create an experiment named "keras-mnist" and a folder to hold the training scripts. The script runs will be recorded under the experiment in Azure.<jupyter_code>from azureml.core import Experiment

script_folder = './keras-mnist'
os.makedirs(script_folder, exist_ok=True)

exp = Experiment(workspace=ws, name='keras-mnist')<jupyter_output><empty_output><jupyter_text>Download MNIST datasetIn order to train on the MNIST dataset we will first need to download it from Yan LeCun's web site directly and save them in a `data` folder locally.<jupyter_code>import urllib

os.makedirs('./data/mnist', exist_ok=True)

urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz', filename='./data/mnist/train-images.gz')
urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz', filename='./data/mnist/train-labels.gz')
urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz', filename='./data/mnist/test-images.gz')
urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz', filename='./data/mnist/test-labels.gz')<jupyter_output><empty_output><jupyter_text>Show some sample imagesLet's load the downloaded compressed file into numpy arrays using some utility functions included in the `utils.py` library file from the current folder. Then we use `matplotlib` to plot 30 random images from the dataset along with their labels.<jupyter_code>from utils import load_data, one_hot_encode

# note we also shrink the intensity values (X) from 0-255 to 0-1. This helps the neural network converge faster.
X_train = load_data('./data/mnist/train-images.gz', False) / 255.0
y_train = load_data('./data/mnist/train-labels.gz', True).reshape(-1)

X_test = load_data('./data/mnist/test-images.gz', False) / 255.0
y_test = load_data('./data/mnist/test-labels.gz', True).reshape(-1)

count = 0
sample_size = 30
plt.figure(figsize = (16, 6))
for i in np.random.permutation(X_train.shape[0])[:sample_size]:
    count = count + 1
    plt.subplot(1, sample_size, count)
    plt.axhline('')
    plt.axvline('')
    plt.text(x = 10, y = -10, s = y_train[i], fontsize = 18)
    plt.imshow(X_train[i].reshape(28, 28), cmap = plt.cm.Greys)
plt.show()<jupyter_output><empty_output><jupyter_text>Upload MNIST dataset to default datastore A [datastore](https://docs.microsoft.com/azure/machine-learning/service/how-to-access-data) is a place where data can be stored that is then made accessible to a Run either by means of mounting or copying the data to the compute target. A datastore can either be backed by an Azure Blob Storage or and Azure File Share (ADLS will be supported in the future). For simple data handling, each workspace provides a default datastore that can be used, in case the data is not already in Blob Storage or File Share.<jupyter_code>ds = ws.get_default_datastore()<jupyter_output><empty_output><jupyter_text>In this next step, we will upload the training and test set into the workspace's default datastore, which we will then later be mount on an `AmlCompute` cluster for training.<jupyter_code>ds.upload(src_dir='./data/mnist', target_path='mnist', overwrite=True, show_progress=True)<jupyter_output><empty_output><jupyter_text>Create or Attach existing AmlComputeYou will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecturecompute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource. If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:1. create the configuration (this step is local and only takes a second)2. create the cluster (this step will take about **20 seconds**)3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell<jupyter_code>from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException

# choose a name for your cluster
cluster_name = "gpucluster"

try:
    compute_target = ComputeTarget(workspace=ws, name=cluster_name)
    print('Found existing compute target')
except ComputeTargetException:
    print('Creating a new compute target...')
    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', 
                                                           max_nodes=4)

    # create the cluster
    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)

    # can poll for a minimum number of nodes and for a specific timeout. 
    # if no min node count is provided it uses the scale settings for the cluster
    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)

# use get_status() to get a detailed status for the current cluster. 
print(compute_target.get_status().serialize())<jupyter_output><empty_output><jupyter_text>Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named "gpu-cluster" of type `AmlCompute`.<jupyter_code>compute_targets = ws.compute_targets
for name, ct in compute_targets.items():
    print(name, ct.type, ct.provisioning_state)<jupyter_output><empty_output><jupyter_text>Copy the training files into the script folderThe Keras training script is already created for you. You can simply copy it into the script folder, together with the utility library used to load compressed data file into numpy array.<jupyter_code>import shutil

# the training logic is in the keras_mnist.py file.
shutil.copy('./keras_mnist.py', script_folder)

# the utils.py just helps loading data from the downloaded MNIST dataset into numpy arrays.
shutil.copy('./utils.py', script_folder)<jupyter_output><empty_output><jupyter_text>Construct neural network in KerasIn the training script `keras_mnist.py`, it creates a very simple DNN (deep neural network), with just 2 hidden layers. The input layer has 28 * 28 = 784 neurons, each representing a pixel in an image. The first hidden layer has 300 neurons, and the second hidden layer has 100 neurons. The output layer has 10 neurons, each representing a targeted label from 0 to 9.![DNN](nn.png)  Azure ML concepts  Please note the following three things in the code below:1. The script accepts arguments using the argparse package. In this case there is one argument `--data_folder` which specifies the file system folder in which the script can find the MNIST data```    parser = argparse.ArgumentParser()    parser.add_argument('--data_folder')```2. The script is accessing the Azure ML `Run` object by executing `run = Run.get_context()`. Further down the script is using the `run` to report the loss and accuracy at the end of each epoch via callback.```    run.log('Loss', log['loss'])    run.log('Accuracy', log['acc'])```3. When running the script on Azure ML, you can write files out to a folder `./outputs` that is relative to the root directory. This folder is specially tracked by Azure ML in the sense that any files written to that folder during script execution on the remote target will be picked up by Run History; these files (known as artifacts) will be available as part of the run history record. The next cell will print out the training code for you to inspect.<jupyter_code>with open(os.path.join(script_folder, './keras_mnist.py'), 'r') as f:
    print(f.read())<jupyter_output><empty_output><jupyter_text>Create TensorFlow estimator & add KerasNext, we construct an `azureml.train.dnn.TensorFlow` estimator object, use the `gpu-cluster` as compute target, and pass the mount-point of the datastore to the training code as a parameter.The TensorFlow estimator is providing a simple way of launching a TensorFlow training job on a compute target. It will automatically provide a docker image that has TensorFlow installed. In this case, we add `keras` package (for the Keras framework obviously), and `matplotlib` package for plotting a "Loss vs. Accuracy" chart and record it in run history.<jupyter_code>from azureml.train.dnn import TensorFlow

script_params = {
    '--data-folder': ds.path('mnist').as_mount(),
    '--batch-size': 50,
    '--first-layer-neurons': 300,
    '--second-layer-neurons': 100,
    '--learning-rate': 0.001
}

est = TensorFlow(source_directory=script_folder,
                 script_params=script_params,
                 compute_target=compute_target, 
                 pip_packages=['keras', 'matplotlib'],
                 entry_script='keras_mnist.py', 
                 use_gpu=True)<jupyter_output><empty_output><jupyter_text>And if you are curious, this is what the mounting point looks like:<jupyter_code>print(ds.path('mnist').as_mount())<jupyter_output><empty_output><jupyter_text>Submit job to runSubmit the estimator to the Azure ML experiment to kick off the execution.<jupyter_code>run = exp.submit(est)<jupyter_output><empty_output><jupyter_text>Monitor the RunAs the Run is executed, it will go through the following stages:1. Preparing: A docker image is created matching the Python environment specified by the TensorFlow estimator and it will be uploaded to the workspace's Azure Container Registry. This step will only happen once for each Python environment -- the container will then be cached for subsequent runs. Creating and uploading the image takes about **5 minutes**. While the job is preparing, logs are streamed to the run history and can be viewed to monitor the progress of the image creation.2. Scaling: If the compute needs to be scaled up (i.e. the AmlCompute cluster requires more nodes to execute the run than currently available), the cluster will attempt to scale up in order to make the required amount of nodes available. Scaling typically takes about **5 minutes**.3. Running: All scripts in the script folder are uploaded to the compute target, data stores are mounted/copied and the `entry_script` is executed. While the job is running, stdout and the `./logs` folder are streamed to the run history and can be viewed to monitor the progress of the run.4. Post-Processing: The `./outputs` folder of the run is copied over to the run historyThere are multiple ways to check the progress of a running job. We can use a Jupyter notebook widget. **Note: The widget will automatically update ever 10-15 seconds, always showing you the most up-to-date information about the run**<jupyter_code>from azureml.widgets import RunDetails
RunDetails(run).show()<jupyter_output><empty_output><jupyter_text>We can also periodically check the status of the run object, and navigate to Azure portal to monitor the run.<jupyter_code>run
run.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>In the outputs of the training script, it prints out the Keras version number. Please make a note of it.  The Run objectThe Run object provides the interface to the run history -- both to the job and to the control plane (this notebook), and both while the job is running and after it has completed. It provides a number of interesting features for instance:* `run.get_details()`: Provides a rich set of properties of the run* `run.get_metrics()`: Provides a dictionary with all the metrics that were reported for the Run* `run.get_file_names()`: List all the files that were uploaded to the run history for this Run. This will include the `outputs` and `logs` folder, azureml-logs and other logs, as well as files that were explicitly uploaded to the run using `run.upload_file()`Below are some examples -- please run through them and inspect their output.<jupyter_code>run.get_details()
run.get_metrics()
run.get_file_names()<jupyter_output><empty_output><jupyter_text>Download the saved model In the training script, the Keras model is saved into two files, `model.json` and `model.h5`, in the `outputs/models` folder on the gpu-cluster AmlCompute node. Azure ML automatically uploaded anything written in the `./outputs` folder into run history file store. Subsequently, we can use the `run` object to download the model files. They are under the the `outputs/model` folder in the run history file store, and are downloaded into a local folder named `model`.<jupyter_code># create a model folder in the current directory
os.makedirs('./model', exist_ok=True)

for f in run.get_file_names():
    if f.startswith('outputs/model'):
        output_file_path = os.path.join('./model', f.split('/')[-1])
        print('Downloading from {} to {} ...'.format(f, output_file_path))
        run.download_file(name=f, output_file_path=output_file_path)<jupyter_output><empty_output><jupyter_text>Predict on the test setLet's check the version of the local Keras. Make sure it matches with the version number printed out in the training script. Otherwise you might not be able to load the model properly.<jupyter_code>import keras
import tensorflow as tf

print("Keras version:", keras.__version__)
print("Tensorflow version:", tf.__version__)<jupyter_output><empty_output><jupyter_text>Now let's load the downloaded model.<jupyter_code>from keras.models import model_from_json

# load json and create model
json_file = open('model/model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)
# load weights into new model
loaded_model.load_weights("model/model.h5")
print("Model loaded from disk.")<jupyter_output><empty_output><jupyter_text>Feed test dataset to the persisted model to get predictions.<jupyter_code># evaluate loaded model on test data
loaded_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
y_test_ohe = one_hot_encode(y_test, 10)
y_hat = np.argmax(loaded_model.predict(X_test), axis=1)

# print the first 30 labels and predictions
print('labels:  \t', y_test[:30])
print('predictions:\t', y_hat[:30])<jupyter_output><empty_output><jupyter_text>Calculate the overall accuracy by comparing the predicted value against the test set.<jupyter_code>print("Accuracy on the test set:", np.average(y_hat == y_test))<jupyter_output><empty_output><jupyter_text>Intelligent hyperparameter tuningWe have trained the model with one set of hyperparameters, now let's how we can do hyperparameter tuning by launching multiple runs on the cluster. First let's define the parameter space using random sampling.<jupyter_code>from azureml.train.hyperdrive import RandomParameterSampling, BanditPolicy, HyperDriveConfig, PrimaryMetricGoal
from azureml.train.hyperdrive import choice, loguniform

ps = RandomParameterSampling(
    {
        '--batch-size': choice(25, 50, 100),
        '--first-layer-neurons': choice(10, 50, 200, 300, 500),
        '--second-layer-neurons': choice(10, 50, 200, 500),
        '--learning-rate': loguniform(-6, -1)
    }
)<jupyter_output><empty_output><jupyter_text>Next, we will create a new estimator without the above parameters since they will be passed in later by Hyperdrive configuration. Note we still need to keep the `data-folder` parameter since that's not a hyperparamter we will sweep.<jupyter_code>est = TensorFlow(source_directory=script_folder,
                 script_params={'--data-folder': ds.path('mnist').as_mount()},
                 compute_target=compute_target,
                 conda_packages=['keras', 'matplotlib'],
                 entry_script='keras_mnist.py', 
                 use_gpu=True)<jupyter_output><empty_output><jupyter_text>Now we will define an early termnination policy. The `BanditPolicy` basically states to check the job every 2 iterations. If the primary metric (defined later) falls outside of the top 10% range, Azure ML terminate the job. This saves us from continuing to explore hyperparameters that don't show promise of helping reach our target metric.<jupyter_code>policy = BanditPolicy(evaluation_interval=2, slack_factor=0.1)<jupyter_output><empty_output><jupyter_text>Now we are ready to configure a run configuration object, and specify the primary metric `Accuracy` that's recorded in your training runs. If you go back to visit the training script, you will notice that this value is being logged after every epoch (a full batch set). We also want to tell the service that we are looking to maximizing this value. We also set the number of samples to 20, and maximal concurrent job to 4, which is the same as the number of nodes in our computer cluster.<jupyter_code>hdc = HyperDriveConfig(estimator=est, 
                       hyperparameter_sampling=ps, 
                       policy=policy, 
                       primary_metric_name='Accuracy', 
                       primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, 
                       max_total_runs=20,
                       max_concurrent_runs=4)<jupyter_output><empty_output><jupyter_text>Finally, let's launch the hyperparameter tuning job.<jupyter_code>hdr = exp.submit(config=hdc)<jupyter_output><empty_output><jupyter_text>We can use a run history widget to show the progress. Be patient as this might take a while to complete.<jupyter_code>RunDetails(hdr).show()
hdr.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>Find and register best modelWhen all the jobs finish, we can find out the one that has the highest accuracy.<jupyter_code>best_run = hdr.get_best_run_by_primary_metric()
print(best_run.get_details()['runDefinition']['arguments'])<jupyter_output><empty_output><jupyter_text>Now let's list the model files uploaded during the run.<jupyter_code>print(best_run.get_file_names())<jupyter_output><empty_output><jupyter_text>We can then register the folder (and all files in it) as a model named `keras-dnn-mnist` under the workspace for deployment.<jupyter_code>model = best_run.register_model(model_name='keras-mlp-mnist', model_path='outputs/model')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08595835557928456, 'fraction_numerical': 0.008930738242003591, 'mean_word_length': 4.972173913043478, 'pattern_counts': {'":': 0, '<': 125, '<?xml version=': 0, '>': 125, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '31', 'dir': 'jupyter-structured-clean-dedup', 'id': '90545', 'license': 'MIT', 'n_tokens_mistral': 6051, 'n_tokens_neox': 5745, 'n_words': 2221, 'path': 'code/1.train-hyperparameter-tune-deploy-with-keras-part1.ipynb', 'repo_name': 'royboy0416/AML-DevOps-Workshop', 'size': '20603'}