<jupyter_start><jupyter_text>Tutorial: Bring your own data (Part 3 of 3) IntroductionIn the previous [Tutorial: Train a model in the cloud](2.pytorch-model.ipynb) article, the CIFAR10 data was downloaded using the builtin `torchvision.datasets.CIFAR10` method in the PyTorch API. However, in many cases you are going to want to use your own data in a remote training run. This article focuses on the workflow you can leverage such that you can work with your own data in Azure Machine Learning. By the end of this tutorial you would have a better understanding of:- How to upload your data to Azure- Best practices for working with cloud data in Azure Machine Learning- Working with command-line arguments--- Your machine learning codeBy now you have your training script running in Azure Machine Learning, and can monitor the model performance. Let's _parametrize_ the training script by introducingarguments. Using arguments will allow you to easily compare different hyperparmeters.Presently our training script is set to download the CIFAR10 dataset on each run. The python code in [train-with-cloud-data-and-logging.py](src/train-with-cloud-data-and-logging.py) now uses **`argparse` to parametize the script.**  Understanding your machine learning code changesThe script `train-with-cloud-data-and-logging.py` has leveraged the `argparse` library to set up the `--data-path`, `--learning-rate`, `--momentum`, and `--epochs` arguments:```pythonimport argparse...parser = argparse.ArgumentParser()parser.add_argument("--data-path", type=str, help="Path to the training data")parser.add_argument("--learning-rate", type=float, default=0.001, help="Learning rate for SGD")parser.add_argument("--momentum", type=float, default=0.9, help="Momentum for SGD")parser.add_argument("--epochs", type=int, default=2, help="Number of epochs to train")args = parser.parse_args()```The script was adapted to update the optimizer to use the user-defined parameters:```pythonoptimizer = optim.SGD(    net.parameters(),    lr=args.learning_rate,      get learning rate from command-line argument    momentum=args.momentum,     get momentum from command-line argument)```Similarly the training loop was adapted to update the number of epochs to train to use the user-defined parameters:```pythonfor epoch in range(args.epochs):```  Upload your data to AzureIn order to run this script in Azure Machine Learning, you need to make your training data available in Azure. Your Azure Machine Learning workspace comes equipped with a _default_ **Datastore** - an Azure Blob storage account - that you can use to store your training data.> ! NOTE > Azure Machine Learning allows you to connect other cloud-based datastores that store your data. For more details, see [datastores documentation](./concept-data.md).<jupyter_code>!pip install --upgrade torchvision
from azureml.core import Workspace, Dataset
from torchvision import datasets

ws = Workspace.from_config()

datasets.CIFAR10(".", download=True)

ds = ws.get_default_datastore()
ds.upload(
    src_dir="cifar-10-batches-py",
    target_path="tutorials/datasets/cifar-10-batches-py",
    overwrite=True,
)
import os
import shutil

os.remove("cifar-10-python.tar.gz")
shutil.rmtree("cifar-10-batches-py")<jupyter_output><empty_output><jupyter_text>The `target_path` specifies the path on the datastore where the CIFAR10 data will be uploaded.  Submit your machine learning code to Azure Machine LearningAs you have done previously, create a new Python control script:<jupyter_code>from azureml.core import (
    Experiment,
    Environment,
    ScriptRunConfig,
    Dataset,
)
from azureml.widgets import RunDetails

ds = Dataset.File.from_files(
    (ws.get_default_datastore(), "tutorials/datasets/")
)
env = Environment.from_conda_specification(
    name="pytorch-env-tutorial", file_path="environment.yml",
)

src = ScriptRunConfig(
    source_directory="src",
    script="train-with-cloud-data-and-logging.py",
    compute_target="cpu-cluster",
    environment=env,
    arguments=[
        "--data-path",
        ds.as_mount(),
        "--learning-rate",
        0.003,
        "--momentum",
        0.92,
        "--epochs",
        2,
    ],
)

run = Experiment(
    workspace=ws, name="an-introduction-train-cloud-data-logging-tutorial"
).submit(src)
RunDetails(run).show()<jupyter_output><empty_output><jupyter_text>Understand the control codeThe above control code has the following additional code compared to the control code written in [previous tutorial](2.pytorch-model.ipynb)**`ds = Dataset.File.from_files(path=(datastore, 'tutorials/datasets'))`**: A Dataset is used to reference the data you uploaded to the Azure Blob Store. Datasets are an abstraction layer on top of your data that are designed to improve reliability and trustworthiness.**`src = ScriptRunConfig(...)`**: We modified the `ScriptRunConfig` to include a list of arguments that will be passed into training script. We also specified `ds.as_mount()`, which means the directory specified will be _mounted_ to the compute target. Inspect the 70_driver_log log fileIn the navigate to the 70_driver_log.txt file - you should see the following output:```Processing 'input'.Processing dataset FileDataset{  "source": [    "('workspaceblobstore', 'datasets/cifar10')"  ],  "definition": [    "GetDatastoreFiles"  ],  "registration": {    "id": "XXXXX",    "name": null,    "version": null,    "workspace": "Workspace.create(name='XXXX', subscription_id='XXXX', resource_group='X')"  }}Mounting input to /tmp/tmp9kituvp3.Mounted input to /tmp/tmp9kituvp3 as folder.Exit __enter__ of DatasetContextManagerEntering Run History Context Manager.Current directory:  /mnt/batch/tasks/shared/LS_root/jobs/dsvm-aml/azureml/tutorial-session-3_1600171983_763c5381/mounts/workspaceblobstore/azureml/tutorial-session-3_1600171983_763c5381Preparing to call script [ train.py ] with arguments: ['--data_path', '$input', '--learning_rate', '0.003', '--momentum', '0.92']After variable expansion, calling script [ train.py ] with arguments: ['--data_path', '/tmp/tmp9kituvp3', '--learning_rate', '0.003', '--momentum', '0.92']Script type = None===== DATA =====DATA PATH: /tmp/tmp9kituvp3LIST FILES IN DATA PATH...['cifar-10-batches-py', 'cifar-10-python.tar.gz']```Notice:1. Azure Machine Learning has mounted the blob store to the compute cluster automatically for you2. The ``ds.as_mount()`` used in the control script resolves to the mount point3. In the machine learning code we include a line to list the directorys under the data directory - you can see the list above If you're not going to use what you've created here, delete the resources you just created with this quickstart so you don't incur any charges for storage. In the Azure portal, select and delete your resource group.<jupyter_code>run.wait_for_completion(show_output=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10343322665114926, 'fraction_numerical': 0.01571137620017457, 'mean_word_length': 4.921619293712317, 'pattern_counts': {'":': 7, '<': 13, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '138522', 'license': 'MIT', 'n_tokens_mistral': 2037, 'n_tokens_neox': 1930, 'n_words': 707, 'path': 'tutorials/an-introduction/3.pytorch-model-cloud-data.ipynb', 'repo_name': 'lobrien/azureml-examples', 'size': '6874'}