<jupyter_start><jupyter_text>Working with DataIn this notebook you will learn how to:1. Read data in a job1. Read *and* write data in a job1. Register data as an asset in Azure Machine Learning1. Read registered data assets from Azure Machine Learning in a job Connect to Azure Machine Learning WorkspaceTo connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../jobs/configuration.ipynb) for more details on how to configure credentials and connect to a workspace.```<jupyter_code>from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

# enter details of your AML workspace
subscription_id = "<SUBSCRIPTION_ID>"
resource_group = "<RESOURCE_GROUP>"
workspace = "<AML_WORKSPACE_NAME>"

# get a handle to the workspace
ml_client = MLClient(
    DefaultAzureCredential(), subscription_id, resource_group, workspace
)<jupyter_output><empty_output><jupyter_text>Reading data in a jobIn this example we will use the titanic dataset in this repo - ([./sample_data/titanic.csv](./sample_data/titanic.csv)) and set-up a command that executes the following python code:```pythondf = pd.read_csv(args.input_data)print(df.head(10))```Below is the code for submitting the command to the cloud - note that both the code *and* the data is automatically uploaded to the cloud. Note: The data is only re-uploaded on subsequent job submissions if data has changed.<jupyter_code>from azure.ai.ml import command
from azure.ai.ml.entities import Data
from azure.ai.ml import Input
from azure.ai.ml.constants import AssetTypes

my_job_inputs = {
    "input_data": Input(type=AssetTypes.URI_FILE, path="./sample_data/titanic.csv")
}

job = command(
    code="./src",  # local path where the code is stored
    command="python read_data.py --input_data ${{inputs.input_data}}",
    inputs=my_job_inputs,
    environment="AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:9",
    compute="cpu-cluster",
)

# submit the command
returned_job = ml_client.jobs.create_or_update(job)
# get a URL for the status of the job
returned_job.services["Studio"].endpoint<jupyter_output><empty_output><jupyter_text>Understanding the codeWhen the job has executed, you will see in the log files a print out of the first 10 records of the titanic sample data. The cell above you can see the inputs to the job were defined using a `dict`:```pythonmy_job_inputs = {    "input_data": Input(        type=AssetTypes.URI_FILE,         path='./sample_data/titanic.csv'    )}```The `Input` class allow you to define data inputs where:- `type` can be a `uri_file` (a specific file) or `uri_folder` (a folder location)- `path` can be a local path or a cloud path. Azure Machine Learning supports `https://`, `abfss://`, `wasbs://` and `azureml://` URIs. As you saw above, if the path is local but your compute is defined to be in the cloud, Azure Machine Learning will automatically upload the data to cloud storage for you.The `Input` defaults the `mode` - how the input will be exposed during job runtime - to `InputOutputModes.RO_MOUNT` (read-only mount). Put another way, Azure Machine Learning will mount the file or folder to the compute and set the file/folder to read-only. By design, you cannot *write* to `JobInputs` only `Outputs` - we will cover this later in the notebook. Accessing data already in the cloudAs mentioned above, the `path` in Input supports `https://`, `abfss://`, `wasbs://` and `azureml://` protocols. Therefore, you can simply change the `path` in the above cell to a cloud-based URI.  Reading *and* writing data in a jobBy design, you cannot *write* to `JobInputs` only `Outputs`. Say, you want to read in some data, do some processing and then write the processed data back to the cloud. In the example below you get the URI of the default Azure ML datastore:<jupyter_code>default_dstor = ml_client.datastores.get_default()
output_path = (
    default_dstor.protocol
    + "://"
    + default_dstor.account_name
    + ".blob."
    + default_dstor.endpoint
    + "/"
    + default_dstor.container_name
)

print(output_path)<jupyter_output><empty_output><jupyter_text>The code below creates a `Output` that will mount your Azure Machine Learning default storage (Azure Blob) in Read-*Write* mode. The python code simply takes the CSV as import and exports it as a parquet file, i.e.```pythondf = pd.read_csv(args.input_data)output_path = os.path.join(args.output_folder, "my_output.parquet")df.to_parquet(output_path)```<jupyter_code>from azure.ai.ml import command
from azure.ai.ml.entities import Data
from azure.ai.ml import Input, Output
from azure.ai.ml.constants import AssetTypes

my_job_inputs = {
    "input_data": Input(type=AssetTypes.URI_FILE, path="./sample_data/titanic.csv")
}

my_job_outputs = {"output_folder": Output(type=AssetTypes.URI_FOLDER, path=output_path)}

job = command(
    code="./src",  # local path where the code is stored
    command="python read_write_data.py --input_data ${{inputs.input_data}} --output_folder ${{outputs.output_folder}}",
    inputs=my_job_inputs,
    outputs=my_job_outputs,
    environment="AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:9",
    compute="cpu-cluster",
)

# submit the command
returned_job = ml_client.create_or_update(job)
# get a URL for the status of the job
returned_job.services["Studio"].endpoint<jupyter_output><empty_output><jupyter_text>Registering data as an asset in Azure Machine LearningYou can register data as an asset in Azure Machine Learning. The benefits of registering data are:- Easy to share with other members of the team (no need to remember file locations)- Versioning of the metadata (location, description, etc)- Lineage trackingBelow we show an example of versioning the sample data in this repo. The data is uploaded to cloud storage and registered as an asset.<jupyter_code>from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

my_data = Data(
    path="./sample_data/titanic.csv",
    type=AssetTypes.URI_FILE,
    description="Titanic Data",
    name="titanic",
    version="1",
)

ml_client.data.create_or_update(my_data)<jupyter_output><empty_output><jupyter_text>> Note: Whilst the above example shows a local file. Remember that `path` supports cloud storage (`https`, `abfss`, `wasbs` protocols). Therefore, if you want to register data in a cloud location just specify the path with any of the supported protocols. Consume data assets in an Azure Machine Learning JobBelow we use the previously registered data asset in the job by refering to the long-form ID in the `path`:```txt/subscriptions/XXXXX/resourceGroups/XXXXX/providers/Microsoft.MachineLearningServices/workspaces/XXXXX/datasets/titanic/versions/1```This long-form URI is accessed using:```pythonregistered_data_asset = ml_client.data.get(name='titanic', version='1')registered_data_asset.id```<jupyter_code>from azure.ai.ml import command, Input, Output
from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

registered_data_asset = ml_client.data.get(name="titanic", version="1")

my_job_inputs = {
    "input_data": Input(type=AssetTypes.URI_FILE, path=registered_data_asset.id)
}

job = command(
    code="./src",
    command="python read_data_asset.py --input_data ${{inputs.input_data}}",
    inputs=my_job_inputs,
    environment="AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:9",
    compute="cpu-cluster",
)

# submit the command
returned_job = ml_client.create_or_update(job)
# get a URL for the status of the job
returned_job.services["Studio"].endpoint<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09742554265522463, 'fraction_numerical': 0.00530035335689046, 'mean_word_length': 5.022036474164134, 'pattern_counts': {'":': 6, '<': 28, '<?xml version=': 0, '>': 29, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '549055', 'license': 'MIT', 'n_tokens_mistral': 2398, 'n_tokens_neox': 2326, 'n_words': 835, 'path': 'sdk/assets/data/data.ipynb', 'repo_name': 'jplummer01/azureml-examples', 'size': '7924'}