<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT License. ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/manymodels/01_Data_Preparation/01_Data_Preparation.png)  01b Environment SetupThis notebook uses simulated orange juice sales data to walk you through the process of training many models on Azure Machine Learning using Automated ML. The time series data used in this example was simulated based on the University of Chicago's Dominick's Finer Foods dataset which featured two years of sales of 3 different orange juice brands for individual stores. The full simulated dataset includes 3,991 stores with 3 orange juice brands each thus allowing 11,973 models to be trained to showcase the power of the many models pattern.  In this notebook, two datasets will be created: one with all 11,973 files and one with only 10 files that can be used to quickly test and debug. For each dataset, you'll be walked through the process of:1. Downloading the data from Azure Open Datasets2. Uploading the data to Azure Blob Storage3. Registering a File Dataset to the Workspace PrerequisitesYou will need to setup your workspace using the configuration notebook.  1.0 Connect to your Workspace and DatastoreIn the configuration notebook you created a [Workspace](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.workspace.workspace?view=azure-ml-py). We are going to use that enviroment to register the data. You also set up the Datastore which in this example is a container in Blob storage where we will store the data.<jupyter_code>from azureml.core.workspace import Workspace
ws = Workspace.from_config() 
datastore = ws.get_default_datastore()

# Take a look at Workspace
ws.get_details()<jupyter_output><empty_output><jupyter_text>2.0 Download the data from Azure Open DatasetsTo download the data, import OjSalesSimulated from Azure Open Datasets. Two datasets are being created: one with all 11,973 files and one with 10 files but this can be customized based on your preferences. If you have to install OjSalesSimulated, run the following command and restart the kernal.<jupyter_code>#!pip install azureml-opendatasets<jupyter_output><empty_output><jupyter_text>To use your own data, create a local folder with each group as a separate file. The data has to be presplit into different files by group and for OJ data, it was already split by Store and Brand. For timeseries, the groups must not split up individual timeseries. That is, each group must contain one or more whole time-seriesThen use that folder and directory in section 3.0 to upload your data to the Datastore.<jupyter_code>from azureml.opendatasets import OjSalesSimulated

# Pull all of the data
oj_sales_files = OjSalesSimulated.get_file_dataset()

# Pull the first 10 files
oj_sales_files_small = OjSalesSimulated.get_file_dataset().take(10)<jupyter_output><empty_output><jupyter_text>Next, create the folders that the data will be downloaded to.<jupyter_code>import os

oj_sales_path = "oj_sales_data"
if not os.path.exists(oj_sales_path):
    os.mkdir(oj_sales_path)
    
oj_sales_path_small = "oj_sales_data_small"
if not os.path.exists(oj_sales_path_small):
    os.mkdir(oj_sales_path_small)<jupyter_output><empty_output><jupyter_text>Finally, download the files to the folder you created.<jupyter_code>oj_sales_files.download(oj_sales_path, overwrite=True)
oj_sales_files_small.download(oj_sales_path_small, overwrite=True)<jupyter_output><empty_output><jupyter_text>3.0 Split the data and upload the files to your DatastoreWe will split the data so that we can use part of the data for inferencing. For the current dataset, we will be splitting on time column ('WeekStarting') before and after '1992-5-28' .You are now ready to load the historical orange juice sales data. To create the [FileDataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.file_dataset.filedataset?view=azure-ml-py) needed for the ParallelRunStep, you first need to upload the csv files to your blob datastore.Please *note* that this will take time as we process 11k files to get the train and test data.  The following scripts will create 'upload_train_data' and 'upload_test_data' folders under the 'oj_sales_data' and 'oj_sales_data_small' folders. The data will then be uploaded to the datastore.<jupyter_code>from scripts.helper import  split_data_upload_to_datastore
time_column_name = 'WeekStarting'
target_date = '1992-5-28'


target_path = 'oj_sales_data' 
target_inference_path = 'oj_sales_inference'
split_data_upload_to_datastore(data_path = oj_sales_path,
                               column_name = time_column_name,
                               date = target_date,
                               datastore = datastore,
                               train_ds_target_path = target_path,
                               test_ds_target_path = target_inference_path)


target_path_small = 'oj_sales_data_small'
target_inference_path_small = 'oj_sales_inference_small'
split_data_upload_to_datastore(data_path = oj_sales_path_small,
                               column_name = time_column_name,
                               date = target_date,
                               datastore = datastore,
                               train_ds_target_path = target_path_small,
                               test_ds_target_path = target_inference_path_small)<jupyter_output><empty_output><jupyter_text>4.0 Create the FileDatasets Now that the files exist in the datastore, [FileDatasets](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.file_dataset.filedataset?view=azure-ml-py) can be created. Datasets in Azure Machine Learning are references to specific data in a Datastore.  We are using FileDatasets since we are storing each group as a separate file. This will help to separate the data needed for model training.<jupyter_code>from azureml.core.dataset import Dataset

ds_name = 'oj_data'
input_ds = Dataset.File.from_files(path=datastore.path(target_path + '/'), validate=False)

inference_name = 'oj_inference'
inference_ds = Dataset.File.from_files(path=datastore.path(target_inference_path + '/'), validate=False)

ds_name_small = 'oj_data_small'
input_ds_small = Dataset.File.from_files(path=datastore.path(target_path_small + '/'), validate=False)

inference_name_small = 'oj_inference_small'
inference_ds_small = Dataset.File.from_files(path=datastore.path(target_inference_path_small + '/'), validate=False)<jupyter_output><empty_output><jupyter_text>5.0 Register the FileDataSets to the Workspace Finally, register the dataset to your Workspace so it can be called as an input into the training pipeline in the next notebook. We will use the inference dataset as part of the forecasting pipeline.<jupyter_code>input_ds.register(ws, ds_name, create_new_version=True)
inference_ds.register(ws, inference_name, create_new_version=True)

input_ds_small.register(ws, ds_name_small, create_new_version=True)
inference_ds_small.register(ws, inference_name_small, create_new_version=True)<jupyter_output><empty_output><jupyter_text>6.0 Call the Registered dataset *(Optional)*After registering the data, it can be easily called using the command below. This is how the datasets will be accessed in future notebooks.<jupyter_code>oj_ds = Dataset.get_by_name(ws, name = ds_name_small)
oj_ds<jupyter_output><empty_output><jupyter_text>You can also download the data from the dataset in the future.<jupyter_code>#oj_ds.download()<jupyter_output><empty_output><jupyter_text>7.0 Delete the local files *(Optional)*<jupyter_code>import shutil

shutil.rmtree(oj_sales_path)
shutil.rmtree(oj_sales_path_small)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0648786717752235, 'fraction_numerical': 0.010727969348659003, 'mean_word_length': 5.0192159877017675, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '643112', 'license': 'MIT', 'n_tokens_mistral': 2400, 'n_tokens_neox': 2282, 'n_words': 748, 'path': 'AutoML/01b_Environment_Setup/01b_Environment_Setup.ipynb', 'repo_name': 'hoseinalizadeh/maria-many-model', 'size': '7830'}