<jupyter_start><jupyter_text>Part 2 - Model TrainingIn this section of the lab we will train a small fully connected neural network using the bottleneck features created in the previous part of the lab.![Transfer Learning](images/TLArch.png)We will use TensorFlow's Keras API to define the network and run training. We will run training on Azure Batch AI GPU cluster. After the model is trained, it will be registered in AML Model Registry.![AML Arch](images/amlarch.png)  Create training scriptThe script trains a simple network with one hidden layer. The input to the network is a vector of 2048 floating point numbers - the bottleneck features created in the previous step of the lab. The output layer consists of 6 units - representing six land type classes. To control overfitting the network uses a Dropout layer between the hidden layer and the output layer and L1 and L2 regularization in the output layer.The number of units in the hidden layer, L1 and L2 values, and batch size are all tuneable hyperparameters. The Dropout ratio is fixed at 0.5.Since the bottleneck feature files are small (as compared to original image datasets) they can be loaded into memory all at once. The trained model will be saved into the `./outputs` folder. This is one of the special folders in AML. The other one is the `./logs` folder. The content in these folders is automatically uploaded to the run history.The script uses AML `Run` object to track two performane measures: **training accuracy** and **validation accuracy**. The metrics are captured at the end of each epoch. Create a folder to hold the script<jupyter_code>import os
script_folder = './script'
os.makedirs(script_folder, exist_ok=True)<jupyter_output><empty_output><jupyter_text>Use Jupyter `%%writefile` magic to write the script<jupyter_code>%%writefile $script_folder/train.py

import os
import tensorflow as tf
from tensorflow.keras.applications import resnet50
from tensorflow.keras.preprocessing import image
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Dropout, Flatten, Input
from tensorflow.keras.regularizers import l1_l2

from azureml.core import Run

import numpy as np
import random
import h5py


# Create custom callback to track accuracy measures in AML Experiment
class RunCallback(tf.keras.callbacks.Callback):
    def __init__(self, run):
        self.run = run
        
    def on_epoch_end(self, batch, logs={}):
        self.run.log(name="training_acc", value=float(logs.get('acc')))
        self.run.log(name="validation_acc", value=float(logs.get('val_acc')))


# Define network
def fcn_classifier(input_shape=(2048,), units=512, classes=6,  l1=0.01, l2=0.01):
    features = Input(shape=input_shape)
    x = Dense(units, activation='relu')(features)
    x = Dropout(0.5)(x)
    y = Dense(classes, activation='softmax', kernel_regularizer=l1_l2(l1=l1, l2=l2))(x)
    model = Model(inputs=features, outputs=y)
    model.compile(optimizer='adadelta', loss='categorical_crossentropy', metrics=['accuracy'])
    return model

# Training regime
def train_evaluate(run):
   
    print("Loading bottleneck features")
    train_file_name = os.path.join(FLAGS.data_folder, FLAGS.train_file_name)
    valid_file_name = os.path.join(FLAGS.data_folder, FLAGS.valid_file_name)
    
    # Load bottleneck training features and labels
    with h5py.File(train_file_name, "r") as hfile:
        train_features = np.array(hfile.get('features'))
        train_labels = np.array(hfile.get('labels'))
        
        
    # Load bottleneck validation features and labels
    with h5py.File(valid_file_name, "r") as hfile:
        valid_features = np.array(hfile.get('features'))
        valid_labels = np.array(hfile.get('labels'))
        
    # Create a network
    model = fcn_classifier(input_shape=(2048,), units=FLAGS.units, l1=FLAGS.l1, l2=FLAGS.l2)
    
    run_callback = RunCallback(run)
    
    # Start training
    print("Starting training")
    model.fit(train_features, train_labels,
          batch_size=FLAGS.batch_size,
          epochs=FLAGS.epochs,
          shuffle=True,
          validation_data=(valid_features, valid_labels),
          callbacks=[run_callback])
          
    # Save the trained model to outp'uts which is a standard folder expected by AML
    print("Training completed.")
    os.makedirs('outputs', exist_ok=True)
    model_file = os.path.join('outputs', 'aerial_classifier.hd5')
    print("Saving model to: {0}".format(model_file))
    model.save(model_file)
    

FLAGS = tf.app.flags.FLAGS

# Default global parameters
tf.app.flags.DEFINE_integer('batch_size', 32, "Number of images per batch")
tf.app.flags.DEFINE_integer('epochs', 10, "Number of epochs to train")
tf.app.flags.DEFINE_integer('units', 512, "Number of epochs to train")
tf.app.flags.DEFINE_float('l1', 0.01, "l1 regularization")
tf.app.flags.DEFINE_float('l2', 0.01, "l2 regularization")
tf.app.flags.DEFINE_string('data_folder', './bottleneck', "Folder with bottleneck features and labels")
tf.app.flags.DEFINE_string('train_file_name', 'aerial_bottleneck_train.h5', "Training file name")
tf.app.flags.DEFINE_string('valid_file_name', 'aerial_bottleneck_valid.h5', "Validation file name")

def main(argv=None):
    # get hold of the current run
    run = Run.get_submitted_run()
    train_evaluate(run)
  

if __name__ == '__main__':
    tf.app.run()<jupyter_output><empty_output><jupyter_text>Connect to AML workspace<jupyter_code>import azureml.core
from azureml.core import Workspace

ws = Workspace.from_config()
print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\n')<jupyter_output><empty_output><jupyter_text>Create ExperimentWe will create a new experiment to manage training logs<jupyter_code>experiment_name = 'aerial-classifier-train'

from azureml.core import Experiment
exp = Experiment(workspace=ws, name=experiment_name)<jupyter_output><empty_output><jupyter_text>Create remote compute clusterWe will use Azure Batch AI GPU cluster to run training and hyper parameter tuning.The cluster is set up for autoscaling. It will start with a single node and can scale to up to 4 nodes. The nodes are NC6 VMs with Tesla K80 GPU.**Creation of the cluster takes approximately 5 minutes.** If the cluster is already in the workspace this code uses it and skips the creation process.<jupyter_code>from azureml.core.compute import ComputeTarget, BatchAiCompute
from azureml.core.compute_target import ComputeTargetException

# choose a name for your cluster
batchai_cluster_name = ws.name + 'gpucls'
vim_size = "STANDARD_NC6"

try:
    # look for the existing cluster by name
    compute_target = ComputeTarget(workspace=ws, name=batchai_cluster_name)
    if type(compute_target) is BatchAiCompute:
        print('found compute target {}, just use it.'.format(batchai_cluster_name))
    else:
        print('{} exists but it is not a Batch AI cluster. Please choose a different name.'.format(batchai_cluster_name))
except ComputeTargetException:
    print('creating a new compute target...')
    compute_config = BatchAiCompute.provisioning_configuration(vm_size=vim_size, # GPU-based VM
                                                                #vm_priority='lowpriority', # optional
                                                                autoscale_enabled=True,
                                                                cluster_min_nodes=1, 
                                                                cluster_max_nodes=4)

    # create the cluster
    compute_target = ComputeTarget.create(ws, batchai_cluster_name, compute_config)
    
    # can poll for a minimum number of nodes and for a specific timeout. 
    # if no min node count is provided it uses the scale settings for the cluster
    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)
    
    # Use the 'status' property to get a detailed status for the current cluster. 
    print(compute_target.status.serialize())<jupyter_output><empty_output><jupyter_text>Configure datastoreThe bottleneck files have been uploaded to the workspace's default datastore during the previous step. We will mount the store on the nodes of the cluster.<jupyter_code>from azureml.core import Datastore

ds = ws.get_default_datastore()
print("Using the default datastore for training data: ")
print(ds.name, ds.datastore_type, ds.account_name, ds.container_name)<jupyter_output><empty_output><jupyter_text>Create an estimatorAs in the previous step of the lab, we will use **Estimator** to submit a run.<jupyter_code>from azureml.train.estimator import Estimator

script_params = {
    '--data_folder': ds.path('bottleneck').as_mount(),
    '--l1': 0.001,
    '--l2': 0.001,
    '--units': 512,
    '--epochs': 50
}

pip_packages = ['h5py','pillow','tensorflow-gpu']

est = Estimator(source_directory=script_folder,
                script_params=script_params,
                compute_target=compute_target,
                entry_script='train.py',
                node_count=1,
                process_count_per_node=1,
                use_gpu=True,
                pip_packages=pip_packages
                )<jupyter_output><empty_output><jupyter_text>Run the jobRun the experiment by submitting the estimator object.<jupyter_code>run = exp.submit(config=est)
run<jupyter_output><empty_output><jupyter_text>Since the call is asynchronous, it returns a **Preparing** or **Running** state as soon as the job is started. Monitor a remote runIn total, the first run takes **approximately 10 minutes**. But for subsequent runs, as long as the script dependencies don't change, the same image is reused and hence the container start up time is much faster.Here is what's happening while you wait:- **Image creation**: A Docker image is created matching the Python environment specified by the estimator. The image is uploaded to the workspace. This stage happens once for each Python environment since the container is cached for subsequent runs.  During image creation, logs are streamed to the run history. You can monitor the image creation progress using these logs.- **Scaling**: If the remote cluster requires more nodes to execute the run than currently available, additional nodes are added automatically. - **Running**: In this stage, the necessary scripts and files are sent to the compute target, then data stores are mounted/copied, then the entry_script is run. While the job is running, stdout and the ./logs directory are streamed to the run history. You can monitor the run's progress using these logs.- **Post-Processing**: The ./outputs directory of the run is copied over to the run history in your workspace so you can access these results.You can check the progress of a running job in multiple ways. This tutorial uses a Jupyter widget as well as a `wait_for_completion` method.  Jupyter widgetWatch the progress of the run with a Jupyter widget.  Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes. Note: Currently, there is a problem with RunDetails widget in DSVM.<jupyter_code>from azureml.train.widgets import RunDetails
RunDetails(run).show()<jupyter_output><empty_output><jupyter_text>Get log results upon completionModel training and monitoring happen in the background. Wait until the model has completed training before running more code.<jupyter_code>run.wait_for_completion(show_output=False) # specify True for a verbose log<jupyter_output><empty_output><jupyter_text>Display run resultsThe training has completed. You can see the logs generated during the run by executing `Run.get_file_names()` method.<jupyter_code>print(run.get_file_names())<jupyter_output><empty_output><jupyter_text>Hyperparameter tuningAs you can see on **Training Accuracy** and **Validation Accuracy** charts, the validation accuracy reaches a plateau while the training accuracy keeps climbing. This is an indication overfitting. We may need to fine tune hyperparameters to achieve better performance.As described before, our network has 4 hyperparameters: - Number of units in the hidden layer- L1 and L2 regularization, and- mini-batch size.So far we have executed a single training run on one set of hyperparameters. Now, we are going to use AML feature called *hyperdrive* to launch multiple runs on multiple cluster nodes using different combinations of hyperparameter values.As we have limited time to complete the lab, we are going to limit a number of hyperparameter combinations.First, let's define the hyperparameter space using grid sampling<jupyter_code>from azureml.train.hyperdrive import *

ps = GridParameterSampling(
    {
        '--units': choice(256, 512),
        '--l1': choice(0.001, 0.01, 0.05),
        '--l2': choice(0.001, 0.01, 0.05)
    }
)<jupyter_output><empty_output><jupyter_text>Next, we will create a new estimator without the above parameters since they will be passed in later.<jupyter_code>script_params = {
    '--data_folder': ds.path('bottleneck').as_mount(),
    '--epochs': 50
}

pip_packages = ['h5py','pillow','tensorflow-gpu']

est = Estimator(source_directory=script_folder,
                script_params=script_params,
                compute_target=compute_target,
                entry_script='train.py',
                node_count=1,
                process_count_per_node=1,
                use_gpu=True,
                pip_packages=pip_packages
                )<jupyter_output><empty_output><jupyter_text>*Hyperdrive* supports early termination policies to limit exploration of hyperparameter combinations that don't show promise of helping reach the target metric. This is feature is especially useful when traversing large hyperparameter spaces. Since we are going to run a small number of jobs we will not apply early termination.<jupyter_code>policy = NoTerminationPolicy()<jupyter_output><empty_output><jupyter_text>Now we are ready to configure a run configuration object, and specify the primary metric as *validation_acc* that's recorded in our training runs. If you go back to visit the training script, you will notice that this value is being logged after every epoch. We also want to tell the service that we are looking to maximizing this value. We also set the number of total runs to 20, and maximal concurrent job to 4, which is the same as the number of nodes in our computer cluster.<jupyter_code>htc = HyperDriveRunConfig(estimator=est, 
                          hyperparameter_sampling=ps,
                          policy=policy,
                          primary_metric_name='validation_acc', 
                          primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, 
                          max_total_runs=12,
                          max_concurrent_runs=4)<jupyter_output><empty_output><jupyter_text>Create a new experiment to capture `hyperdrive` runs.<jupyter_code>experiment_name = 'aerial-classifier-hyperdrive'

from azureml.core import Experiment
exp = Experiment(workspace=ws, name=experiment_name)<jupyter_output><empty_output><jupyter_text>Finally, let's launch the hyperparameter tuning job.<jupyter_code>hdr = exp.submit(config=htc)
hdr
from azureml.train.widgets import RunDetails
RunDetails(hdr).show()
hdr.wait_for_completion(show_output=True) # specify True for a verbose log<jupyter_output><empty_output><jupyter_text>Find and register best modelWhen all the jobs finish, we can find out the one that has the highest accuracy.<jupyter_code>best_run = hdr.get_best_run_by_primary_metric()
best_run_metrics = best_run.get_metrics()
parameter_values = best_run.get_details()['runDefinition']['Arguments']

print('Best Run Id: ', best_run.id)
print('\n Validation Accuracy:', best_run_metrics['validation_acc'])
print('\n Units:',parameter_values[5])
print('\n L1:',parameter_values[7])
print('\n L2:',parameter_values[9])<jupyter_output><empty_output><jupyter_text>Check the output of the best run.<jupyter_code>print(run.get_file_names())<jupyter_output><empty_output><jupyter_text>Register modelThe last step in the training script wrote the file `aerial_classifier.hd5` in the `outputs` directory. As noted before, `outputs` is a special directory in that all content in this  directory is automatically uploaded to your workspace.  This content appears in the run record in the experiment under your workspace. You can register the model so that it can be later queried, examined and deployed.<jupyter_code>model = best_run.register_model(model_name='aerial_classifier', model_path='outputs/aerial_classifier.hd5')
print(model.name, model.id, model.version, sep = '\t')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08053129113318176, 'fraction_numerical': 0.009333492880220175, 'mean_word_length': 4.243099121706399, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '75988', 'license': 'MIT', 'n_tokens_mistral': 4837, 'n_tokens_neox': 4613, 'n_words': 1644, 'path': 'Archive/00-AML-EndToEndWalkthrough/02-train.ipynb', 'repo_name': 'SamIstephan/MTC_AzureAILabs', 'size': '16714'}