<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/using-environments/using-environments.png)  Using environments Contents1. [Introduction](Introduction)1. [Setup](Setup)1. [Use curated environment](Use-curated-environment)1. [Create environment](Create-environment)    1. Add Python packages    1. Specify environment variables1. [Submit run using environment](Submit-run-using-environment)1. [Register environment](Register-environment)1. [List and get existing environments](List-and-get-existing-environments)1. [Other ways to create environments](Other-ways-to-create-environments)    1. From existing Conda environment    1. From Conda or pip files1. [Using environments for inferencing](Using-environments-for-inferencing)1. [Docker settings](Docker-settings)1. [Spark and Azure Databricks settings](Spark-and-Azure-Databricks-settings)1. [Next steps](Next-steps) IntroductionAzure ML environments are an encapsulation of the environment where your machine learning training happens. They define Python packages, environment variables, Docker settings and other attributes in declarative fashion. Environments are versioned: you can update them and retrieve old versions to revisit and review your work.Environments allow you to:* Encapsulate dependencies of your training process, such as Python packages and their versions.* Reproduce the Python environment on your local computer in a remote run on VM or ML Compute cluster* Reproduce your experimentation environment in production setting.* Revisit and audit the environment in which an existing model was trained.Environment, compute target and training script together form run configuration: the full specification of training run. SetupIf you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration notebook](../../../configuration.ipynb) first if you haven't.First, let's validate Azure ML SDK version and connect to workspace.<jupyter_code>import azureml.core
print(azureml.core.VERSION)
from azureml.core.workspace import Workspace
ws = Workspace.from_config()
ws.get_details()<jupyter_output><empty_output><jupyter_text>Use curated environmentsCurated environments are provided by Azure Machine Learning and are available in your workspace by default. They contain collections of Python packages and settings to help you get started different machine learning frameworks.   * The __AzureML-Minimal__ environment contains a minimal set of packages to enable run tracking and asset uploading. You can use it as a starting point for your own environment.  * The __AzureML-Tutorial__ environment contains common data science packages, such as Scikit-Learn, Pandas and Matplotlib, and larger set of azureml-sdk packages. Curated environments are backed by cached Docker images, reducing the run preparation cost. You can get a curated environment using<jupyter_code>from azureml.core import Environment

curated_env = Environment.get(workspace=ws, name="AzureML-Minimal")<jupyter_output><empty_output><jupyter_text>To list curated environments, use following code.**Note**: The name prefixes _AzureML_ and _Microsoft_ are reserved for curated environments. Do not use them for your own environments<jupyter_code>envs = Environment.list(workspace=ws)

for env in envs:
    if env.startswith("AzureML"):
        print("Name",env)
        if envs[env].python.conda_dependencies is not None:
            print("packages", envs[env].python.conda_dependencies.serialize_to_string())<jupyter_output><empty_output><jupyter_text>Create your own environmentYou can create an environment by instantiating ```Environment``` object and then setting its attributes: set of Python packages, environment variables and others. Add Python packagesThe recommended way is to specify Conda packages, as they typically come with complete set of pre-built binaries.<jupyter_code>from azureml.core.environment import CondaDependencies

myenv = Environment(name="myenv")
conda_dep = CondaDependencies()
conda_dep.add_conda_package("scikit-learn==0.22.1")<jupyter_output><empty_output><jupyter_text>You can also add pip packages, and specify the version of package<jupyter_code>conda_dep.add_pip_package("pillow==6.2.1")
myenv.python.conda_dependencies=conda_dep<jupyter_output><empty_output><jupyter_text>Submit run using environmentWhen you submit a run, you can specify which environment to use. On the first run in given environment, Azure ML spends some time building the environment. On the subsequent runs, Azure ML keeps track of changes and uses the existing environment, resulting in faster run completion.<jupyter_code>from azureml.core import ScriptRunConfig, Experiment

myexp = Experiment(workspace=ws, name = "environment-example")<jupyter_output><empty_output><jupyter_text>To submit a run, create a run configuration that combines the script file and environment, and pass it to ```Experiment.submit```. In this example, the script is submitted to local computer, but you can specify other compute targets such as remote clusters as well.<jupyter_code>src = ScriptRunConfig(source_directory=".",
                      script="example.py",
                      compute_target="local",
                      environment=myenv)

run = myexp.submit(config=src)

run.wait_for_completion(show_output=True)<jupyter_output><empty_output><jupyter_text>To audit the environment used by for a run, you can use ```get_environment```.<jupyter_code>run.get_environment()<jupyter_output><empty_output><jupyter_text>Register environmentYou can manage environments by registering them. This allows you to track their versions, and reuse them in future runs. For example, once you've constructed an environment that meets your requirements, you can register it and use it in other experiments so as to standardize your workflow.If you register the environment with same name, the version number is increased by one. Note that Azure ML keeps track of differences between the version, so if you re-register an identical version, the version number is not increased.<jupyter_code>myenv.register(workspace=ws)<jupyter_output><empty_output><jupyter_text>List and get existing environmentsYour workspace contains a dictionary of registered environments. You can then use ```Environment.get``` to retrieve a specific environment with specific version.<jupyter_code>for name,env in ws.environments.items():
    print("Name {} \t version {}".format(name,env.version))

restored_environment = Environment.get(workspace=ws,name="myenv",version="1")

print("Attributes of restored environment")
restored_environment<jupyter_output><empty_output><jupyter_text>Other ways to create environments From existing Conda environmentYou can create an environment from existing conda environment. This make it easy to reuse your local interactive environment in Azure ML remote runs. For example, if you've created conda environment using```conda create -n mycondaenv```you can create Azure ML environment out of that conda environment using```myenv = Environment.from_existing_conda_environment(name="myenv",conda_environment_name="mycondaenv")``` From conda or pip filesYou can create environments from conda specification or pip requirements files using```myenv = Environment.from_conda_specification(name="myenv", file_path="path-to-conda-specification-file")myenv = Environment.from_pip_requirements(name="myenv", file_path="path-to-pip-requirements-file")```  Using environments for inferencingYou can re-use the training environment when you deploy your model as a web service, by specifying inferencing stack version, and adding then environment to ```InferenceConfig```.```from azureml.core.model import InferenceConfigmyenv.inferencing_stack_version = "latest"inference_config = InferenceConfig(entry_script="score.py", environment=myenv)```See [Register Model and deploy as Webservice Notebook](../../deployment/deploy-to-cloud/model-register-and-deploy.ipynb) for an end-to-end example of web service deployment.  Docker settingsDocker container provides an efficient way to encapsulate the dependencies. When you enable Docker, Azure ML builds a Docker image and creates a Python environment within that container, given your specifications. The Docker images are reused: the first run in a new environment typically takes longer as the image is build.**Note:** For runs on local computer or attached virtual machine, that computer must have Docker installed and enabled. Machine Learning Compute has Docker pre-installed.Attribute ```docker.enabled``` controls whether to use Docker container or host OS for execution.<jupyter_code>myenv.docker.enabled = True<jupyter_output><empty_output><jupyter_text>You can specify custom Docker base image and registry. This allows you to customize and control in detail the guest OS in which your training run executes. whether to use GPU, whether to use shared volumes, and shm size.<jupyter_code>myenv.docker.base_image
myenv.docker.base_image_registry<jupyter_output><empty_output><jupyter_text>You can also specify shared volumes, and shm size.<jupyter_code>myenv.docker.shared_volumes
myenv.docker.shm_size<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07241121893995947, 'fraction_numerical': 0.00405246880665458, 'mean_word_length': 6.208301306687163, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '233416', 'license': 'MIT', 'n_tokens_mistral': 2429, 'n_tokens_neox': 2334, 'n_words': 943, 'path': 'how-to-use-azureml/training/using-environments/using-environments.ipynb', 'repo_name': 'rserran/MachineLearningNotebooks', 'size': '9377'}