{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Search\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This example shows how to perform random search hyperparameter tuning using CNTK with MNIST dataset to train a convolutional neural network (CNN) on a GPU cluster. We make use of the Batch AI Extensions, including the JobFactory module to generate values for hyperparameters, and the ExperimentUtils module for bulk job submission.\n",
    "\n",
    "## Details\n",
    "\n",
    "- We provide a CNTK example [ConvMNIST.py](../ConvMNIST.py) to accept  command line arguments for CNTK dataset, model locations, model file suffix and two hyperparameters for tuning: 1. hidden layer dimension and 2. feedforward constant \n",
    "- For demonstration purposes, MNIST dataset and CNTK training script will be deployed at Azure File Share;\n",
    "- Standard output of the job and the model will be stored on Azure File Share;\n",
    "- MNIST dataset (http://yann.lecun.com/exdb/mnist/) has been preprocessed by usign install_mnist.py available [here](https://batchaisamples.blob.core.windows.net/samples/mnist_dataset.zip?st=2017-09-29T18%3A29%3A00Z&se=2099-12-31T08%3A00%3A00Z&sp=rl&sv=2016-05-31&sr=c&sig=PmhL%2BYnYAyNTZr1DM2JySvrI12e%2F4wZNIwCtf7TRI%2BM%3D)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "### Install Dependencies and Create Configuration file.\n",
    "Follow [instructions](/recipes) to install all dependencies and create configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Configuration and Create Batch AI client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "bfa11f00-8866-4051-bbfe-a9646e004910"
    }
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import sys\n",
    "\n",
    "import azure.mgmt.batchai.models as models\n",
    "from azure.storage.blob import BlockBlobService\n",
    "from azure.storage.file import FileService\n",
    "\n",
    "sys.path.append('../../..')\n",
    "import utilities as utils\n",
    "from utilities.job_factory import ParameterSweep, NumericParameter, DiscreteParameter\n",
    "\n",
    "cfg = utils.config.Configuration('../../configuration.json')\n",
    "client = utils.config.create_batchai_client(cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create Resoruce Group and Batch AI workspace if not exists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "utils.config.create_resource_group(cfg)\n",
    "_ = client.workspaces.create(cfg.resource_group, cfg.workspace, cfg.location).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Prepare Training Dataset and Script in Azure Storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Azure Blob Container\n",
    "\n",
    "We will create a new Blob Container with name `batchaisample` under your storage account. This will be used to store the *input training dataset*\n",
    "\n",
    "**Note** You don't need to create new blob Container for every cluster. We are doing this in this sample to simplify resource management for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "azure_blob_container_name = 'batchaisample'\n",
    "blob_service = BlockBlobService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "blob_service.create_container(azure_blob_container_name, fail_on_exist=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload MNIST Dataset to Azure Blob Container\n",
    "\n",
    "For demonstration purposes, we will download preprocessed MNIST dataset to the current directory and upload it to Azure Blob Container directory named `mnist_dataset`.\n",
    "\n",
    "There are multiple ways to create folders and upload files into Azure Blob Container - you can use [Azure Portal](https://ms.portal.azure.com), [Storage Explorer](http://storageexplorer.com/), [Azure CLI2](/azure-cli-extension) or Azure SDK for your preferable programming language.\n",
    "In this example we will use Azure SDK for python to copy files into Blob."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist_dataset_directory = 'mnist_dataset'\n",
    "utils.dataset.download_and_upload_mnist_dataset_to_blob(\n",
    "    blob_service, azure_blob_container_name, mnist_dataset_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Azure File Share\n",
    "\n",
    "For this example we will create a new File Share with name `batchaisample` under your storage account. This will be used to share the *training script file* and *output file*.\n",
    "\n",
    "**Note** You don't need to create new file share for every cluster. We are doing this in this sample to simplify resource management for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "azure_file_share_name = 'batchaisample'\n",
    "file_service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "file_service.create_share(azure_file_share_name, fail_on_exist=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload the training script [ConvMNIST.py](../ConvMNIST.py) to file share directory named `hyperparam_samples`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cntk_script_path = \"hyperparam_samples\"\n",
    "file_service.create_directory(\n",
    "    azure_file_share_name, cntk_script_path, fail_on_exist=False)\n",
    "file_service.create_file_from_path(\n",
    "    azure_file_share_name, cntk_script_path, 'ConvMNIST.py', '../ConvMNIST.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create Azure Batch AI Compute Cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Compute Cluster\n",
    "\n",
    "- For this example we will use a GPU cluster of `STANDARD_NC6` nodes. Number of nodes in the cluster is configured with `nodes_count` variable;\n",
    "- We will call the cluster `nc6`;\n",
    "\n",
    "So, the cluster will have the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes_count = 4\n",
    "cluster_name = 'nc6'\n",
    "\n",
    "parameters = models.ClusterCreateParameters(\n",
    "    location=cfg.location,\n",
    "    vm_size='STANDARD_NC6',\n",
    "    scale_settings=models.ScaleSettings(\n",
    "        manual=models.ManualScaleSettings(target_node_count=nodes_count)\n",
    "    ),\n",
    "    user_account_settings=models.UserAccountSettings(\n",
    "        admin_user_name=cfg.admin,\n",
    "        admin_user_password=cfg.admin_password or None,\n",
    "        admin_user_ssh_public_key=cfg.admin_ssh_key or None,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Compute Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = client.clusters.create(cfg.resource_group, cfg.workspace, cluster_name, parameters).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor Cluster Creation\n",
    "\n",
    "Monitor the just created cluster. The `utilities` module contains a helper function to print out detail status of the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = client.clusters.get(cfg.resource_group, cfg.workspace, cluster_name)\n",
    "utils.cluster.print_cluster_status(cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Parametric Sweeping using Random Search\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new experiment called ```random_search_experiment```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_name = 'random_search_experiment'\n",
    "experiment = client.experiments.create(cfg.resource_group, cfg.workspace, experiment_name).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define specifications for the hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "param_specs = [\n",
    "    NumericParameter(\n",
    "        parameter_name=\"FEEDFORWARD_CONSTANT\",\n",
    "        data_type=\"REAL\",\n",
    "        start=0.001,\n",
    "        end=10,\n",
    "        scale=\"LOG\"\n",
    "    ),\n",
    "    DiscreteParameter(\n",
    "        parameter_name=\"HIDDEN_LAYERS_DIMENSION\",\n",
    "        values=[100, 200, 300]\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a parameter substitution object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = ParameterSweep(param_specs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the parameter substitution object to specify where we would like to substitute the parameters. We substitute\n",
    "the values for feedforward constant and hidden layers dimension into `models.JobCreateParameters.cntk_settings.command_line_args`. Note that the `parameters` variable is used like a dict, with the `parameter_name` being used as the key to specify which parameter to substitute. When `parameters.generate_jobs` is called, the `parameters[name]` variables will be replaced with actual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "azure_file_share_mount_path = 'afs'\n",
    "azure_blob_mount_path = 'bfs'\n",
    "jcp = models.JobCreateParameters(\n",
    "    cluster=models.ResourceId(id=cluster.id),\n",
    "    node_count=1,\n",
    "    std_out_err_path_prefix='$AZ_BATCHAI_JOB_MOUNT_ROOT/{0}'.format(azure_file_share_mount_path),\n",
    "    input_directories = [\n",
    "        models.InputDirectory(\n",
    "            id='SCRIPT',\n",
    "            path='$AZ_BATCHAI_JOB_MOUNT_ROOT/{0}/{1}'.format(azure_blob_mount_path, mnist_dataset_directory))\n",
    "    ],\n",
    "    output_directories = [\n",
    "        models.OutputDirectory(\n",
    "            id='ALL',\n",
    "            path_prefix='$AZ_BATCHAI_JOB_MOUNT_ROOT/{0}'.format(azure_file_share_mount_path))],\n",
    "    mount_volumes = models.MountVolumes(\n",
    "        azure_file_shares=[\n",
    "            models.AzureFileShareReference(\n",
    "                account_name=cfg.storage_account_name,\n",
    "                credentials=models.AzureStorageCredentialsInfo(\n",
    "                    account_key=cfg.storage_account_key),\n",
    "                azure_file_url='https://{0}.file.core.windows.net/{1}'.format(\n",
    "                    cfg.storage_account_name, azure_file_share_name),\n",
    "                relative_mount_path=azure_file_share_mount_path)\n",
    "        ],\n",
    "        azure_blob_file_systems=[\n",
    "            models.AzureBlobFileSystemReference(\n",
    "                account_name=cfg.storage_account_name,\n",
    "                credentials=models.AzureStorageCredentialsInfo(\n",
    "                    account_key=cfg.storage_account_key),\n",
    "                container_name=azure_blob_container_name,\n",
    "                relative_mount_path=azure_blob_mount_path)\n",
    "        ]\n",
    "    ),\n",
    "    container_settings=models.ContainerSettings(\n",
    "        image_source_registry=models.ImageSourceRegistry(image='microsoft/cntk:2.5.1-gpu-python2.7-cuda9.0-cudnn7.0')\n",
    "    ),\n",
    "    cntk_settings=models.CNTKsettings(\n",
    "        python_script_file_path='$AZ_BATCHAI_JOB_MOUNT_ROOT/{0}/{1}/ConvMNIST.py'.format(azure_file_share_mount_path, cntk_script_path),\n",
    "        command_line_args='--epochs 16 --feedforward_const {0} --hidden_layers_dim {1} --datadir $AZ_BATCHAI_INPUT_SCRIPT --outputdir $AZ_BATCHAI_OUTPUT_ALL --logdir $AZ_BATCHAI_OUTPUT_ALL'\n",
    "            .format(parameters['FEEDFORWARD_CONSTANT'], parameters['HIDDEN_LAYERS_DIMENSION'])  # Substitute hyperparameters\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we generate a list of jobs to submit and then submit the jobs to an experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate Jobs\n",
    "num_configs = 16\n",
    "jobs_to_submit, param_combinations = parameters.generate_jobs_random_search(jcp, num_configs)\n",
    "\n",
    "# Print the parameter combinations generated\n",
    "for idx, comb in enumerate(param_combinations):\n",
    "    print(\"Parameters {0}: {1}\".format(idx + 1, comb))\n",
    "\n",
    "# Submit Jobs\n",
    "experiment_utils = utils.experiment.ExperimentUtils(client, cfg.resource_group, cfg.workspace, experiment_name)\n",
    "jobs = experiment_utils.submit_jobs(jobs_to_submit, 'mnist_hyperparam_job').result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the following metric extractor to extract desired metric from learning log file. \n",
    "- In this example, we extract the number between \"metric =\" and \"%\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_extractor = utils.job.MetricExtractor(\n",
    "                        output_dir_id='ALL',\n",
    "                        logfile='progress.log',\n",
    "                        regex='metric =(.*?)\\%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We wait on the jobs the finish, then get the metric value from the log files of the finished jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait for all jobs to complete\n",
    "experiment_utils.wait_all_jobs()\n",
    "\n",
    "# Get the metrics from the jobs\n",
    "results = experiment_utils.get_metrics_for_jobs(jobs, metric_extractor)\n",
    "results.sort(key=lambda r: r['metric_value'])\n",
    "\n",
    "# Print results\n",
    "for result in results:\n",
    "    print(\"Job {0} completed with metric value {1}\".format(result['job_name'], result['metric_value']))\n",
    "print(\"Best job: {0} with parameters {1}\".format(\n",
    "    results[0]['job_name'], \n",
    "    {ev.name:ev.value for ev in results[0]['job'].environment_variables}\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Clean Up (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Experiment\n",
    "Delete the experiment and jobs inside it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = client.experiments.delete(cfg.resource_group, cfg.workspace, experiment_name).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Cluster\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the cluster using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = client.clusters.delete(cfg.resource_group, cfg.workspace, cluster_name).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Delete File Share\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the file share completely with all files using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "service.delete_share(azure_file_share_name)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
