<jupyter_start><jupyter_text>Getting Started with the Iguazio Data Science Platform   Overview This tutorial is designed to demonstrate some of the main capabilities of the Iguazio Data Science Platform (**"the platform"**).The tutorial walks you through a basic machine-learning (ML) workflow, which includes the following steps; before running the steps, ensure that the required [prerequisites](gs-prerequisites) are met:1. [Create and configure a new project](gs-step-project-create-n-config)2. [Ingest a data set](gs-step-ingest-data)3. [Explore the data](gs-step-explore-data)4. [Run model training at scale](gs-step-run-model-training)5. [Test your model](gs-step-test-model)6. [Serve your model](gs-step-model-serving)7. [Create and run a fully automated ML pipeline](gs-step-create-n-run-ml-pipeline)By the end of this tutorial you will learn how to1. Create a new platform project.2. Create functions and track experiments while running them.3. Store artifacts to be used and managed in a central database.4. Leverage the MLRun functions marketplace.5. Create an automated ML pipeline.6. Run your code on a distributed Kubernetes cluster without any DevOps overhead.7. Create a scalable endpoint for model serving.The tutorial uses the [Iris flower data set](http://archive.ics.uci.edu/ml/datasets/iris) &mdash; a popular data set that is widely used for basic data science tutorials.The data set contains a set of 150 records under five attributes &mdash; petal length, petal width, sepal length, sepal width, and species. Your model will classify the species based on the other four attributes.   Prerequisites The tutorial uses MLRun to create a project, implement and execute an ML pipeline, and track the execution.(For more information about MLRun, see Step 1.)To use MLRun, you must first ensure that it's installed and running as a service on your platform cluster.Look for an `mlrun` service on the **Services** page of the platform dashboard.For more information and additional assistance, contact the Iguazio [support team](mailto:<EMAIL>).To use MLRun from Jupyter Notebook, you need to run the following code to install the `mlrun` Python package.This needs to be done only once per Jupyter Notebook service.> **Note:** You must **restart the Jupyter kernel** to complete the installation.<jupyter_code>import sys
!{sys.executable} -m pip install mlrun<jupyter_output><empty_output><jupyter_text>Step 1: Create and Configure a New Project Start out by creating a new platform project and making some initial configurations.   Create a New Project Projects in the platform are used to package multiple functions, workflows, and artifacts.Projects are created by using the `new_project` MLRun method, which receives the following parameters:- **`name`** (Required) &mdash; the project name.- **`context`** &mdash; the path to a local project directory (the project's context directory).  The project directory contains a project-configuration file (default: **project.yaml**), which defines the project, and additional generated Python code.  The project file is created when you save your project (using the `save` MLRun project method), as demonstrated in Step 6.- **`functions`** &mdash; a list of functions objects or links to function code or objects.- **`init_git`** &mdash; set to `True` to perform Git initialization of the project directory (`context`).  > **Note:** It's customary to store project code and definitions in a Git repository.Projects are visible in the MLRun dashboard only after they're saved to the MLRun database, which happens whenever you run code for a project.The following code creates a project named "getting-started-iris-&lt;V3IO_USERNAME&gt;", where **&lt;V3IO_USERNAME&gt;** is your current running username in the platform, and sets the project directory to a **conf** directory in the current tutorial directory (**/User/getting-started-tutorial/conf**).> **Note:** Platform projects are shared among all users of the parent tenant, to facilitate collaboration. Therefore,>> - Synchronize your projects execution with other users on your platform cluster, as needed, or use unique project names to avoid conflicts.>   You can easily change the default project name for this tutorial by changing the definition of the `project_name` variable in the following code.> - Don't include in your project proprietary information that you don't want to expose to other users.>   Note that while projects are a useful tool, you can easily develop and run code in the platform without using projects.<jupyter_code>from os import path, getenv
from mlrun import new_project

project_name = '-'.join(filter(None, ['getting-started-iris', getenv('V3IO_USERNAME', None)])).lower()
project_path = path.abspath('conf')
project = new_project(project_name, project_path, init_git=True)

print(f'Project path: {project_path}\nProject name: {project_name}')<jupyter_output>Project path: /User/getting-started-tutorial/conf
Project name: getting-started-iris-iguazio<jupyter_text>Configure MLRun [MLRun](https://github.com/mlrun/mlrun) is a generic and convenient mechanism for data scientists and software developers to describe and run tasks related to machine learning in various, scalable runtime environments and ML pipelines while automatically tracking executed code, metadata, inputs, and outputs.MLRun integrates with the Nuclio serverless framework and with the Kubeflow Pipelines framework for running ML pipelines.The tutorial uses MLRun to create a project, implement and execute an ML pipeline, and track the execution.Before running your code, you need to set some MLRun configurations:- **Artifacts path** &mdash; the location for storing versioned data artifacts (such as files, objects, data sets, and models) that are produced or consumed by functions, runs, and workflows.  The path can be defined either as a local directory path or as a URL (of the format `s3://*`, `v3io://*`, etc.).  You can set the artifacts path either by defining an `MLRUN_ARTIFACT_PATH` environment variable (which applies globally throughout the current environment) or as part of the MLRun configuration.   If the target directory doesn't exist, MLRun creates it.  You can use the notation `{{run.uid}}` in the path to signify the current run ID.  For pipelines, you can use the notation `{{workflow.uid}}` to signify the workflow ID.  This allows you to create a unique artifacts directory for each executed job or workflow.  After you run an MLRun job, the artifacts directory might contain one or more of the following directories:   - **plots** &mdash; a directory for storing images, figures, and plotlines.  - **models** &mdash; a directory for storing all trained models.  - **data** &mdash; a directory for storing any other type of data artifact, such as data sets.- **The MLRun DB path** &mdash; the path to an MLRun database (DB) directory or the URL of an MLRun API service.  The MLRun service initializes `mlconf.dbpath` to the default API URL for the current environment (usually `http://mlrun-api:8080`).The following code sets the artifacts path to a **jobs** directory within the tutorial directory (**/User/getting-started-tutorial/jobs**), and the MLRun DB path to the currently configured MLRun DB path, if exists, or to `http://mlrun-api:8080` otherwise.(In Step 7 of the tutorial, you'll create a full pipeline and store the pipeline artifacts in a **pipeline** tutorial directory.)<jupyter_code>from mlrun import run_local, NewTask, mlconf, import_function, mount_v3io

# Target location for storing pipeline artifacts
artifact_path = path.abspath('jobs')
# MLRun DB path or API service URL
mlconf.dbpath = mlconf.dbpath or 'http://mlrun-api:8080'

print(f'Artifacts path: {artifact_path}\nMLRun DB path: {mlconf.dbpath}')<jupyter_output>Artifacts path: /User/getting-started-tutorial/jobs
MLRun DB path: http://mlrun-api:8080<jupyter_text>Import and Configure Nuclio [Nuclio](https://nuclio.io/) is a high-performance open-source and managed serverless framework, which is available as a predefined tenant-wide platform service (`nuclio`).The tutorial uses Nuclio to create and deploy serverless functions.Therefore, you need to import the Nuclio package and configure Nuclio for your project.The platform's Jupyter Notebook service preinstalls the [nuclio-jupyter SDK](https://github.com/nuclio/nuclio-jupyter/blob/master/README.md) for creating and deploying Nuclio functions with Python and Jupyter Notebook.The tutorial uses the Nuclio magic commands and annotation comments of this SDK to automate function code generation.The magic commands are initialized when you import the `nuclio` package.The `%nuclio` magic commands are used to run Nuclio commands from Jupyter notebooks (`%nuclio `).You can also use `%%nuclio` at the start of a cell to identify the entire cell as containing Nuclio code.The magic commands are initialized when you import the `nuclio` package.The ` nuclio: start-code`, ` nuclio: end-code`, and ` nuclio: ignore` section-marker annotations notify Nuclio of the beginning or end of code sections.Nuclio ignores all notebook code before a ` nuclio: start-code` marker or after an ` nuclio: end-code` marker.Nuclio translates all other notebook code sections into function code, except for sections that are marked with the ` nuclio: ignore` marker.  Import Nuclio The following code imports the `nuclio` Python package.<jupyter_code>import nuclio<jupyter_output><empty_output><jupyter_text>Configure Nuclio The following code uses the ` nuclio: start-code` marker to instruct Nuclio to start processing code only from this location, and then performs basic Nuclio function configuration &mdash; defining the name of the function's container image (`mlrun/ml-models`) and the function type (`job`).> **Note:** You can add code to define function dependencies and perform additional configuration after the ` nuclio: start-code` marker.<jupyter_code># nuclio: start-code
%nuclio config spec.image = "mlrun/ml-models"
%nuclio config kind = "job"<jupyter_output>%nuclio: setting spec.image to 'mlrun/ml-models'
%nuclio: setting kind to 'job'<jupyter_text>Step 2: Ingest a Data Set Define and run a Python function that downloads the Iris flower data set from Amazon Web Services (AWS) S3 bucket to the platform's local NoSQL data store, and stores it as an MLRun data set.   Initialization Define the source URL for downloading the data set.> **Note:** As explained earlier, the `nuclio: ignore` annotation is used to instruct Nuclio to ignore this cell.<jupyter_code># nuclio: ignore
# Set the source-data URL
source_url = 'http://iguazio-sample-data.s3.amazonaws.com/iris_dataset.csv'<jupyter_output><empty_output><jupyter_text>Define a Data-Ingestion Function Define a data-ingestion function that downloads the configured data set from an AWS S3 bucket and saves it to the platform's local NoSQL data store.<jupyter_code>from os import path
from mlrun.execution import MLClientCtx
from mlrun.datastore import DataItem

# Ingest a data set into the platform
def get_data(context: MLClientCtx, source_url: DataItem, format: str = 'csv'):

    iris_dataset = source_url.as_df()

    target_path = path.join(context.artifact_path, 'data')
    # Optionally print data to your logger
    context.logger.info('Saving Iris data set to {} ...'.format(target_path))

    # Store the data set in your artifacts database
    context.log_dataset('iris_dataset', df=iris_dataset, format=format,
                        index=False, artifact_path=target_path)<jupyter_output><empty_output><jupyter_text>The following cell uses the ` nuclio: end-code` marker to mark the end of a Nuclio code section and instruct Nuclio to stop parsing the notebook at this point.> **IMPORTANT:** Do not remove the end-code cell.<jupyter_code># nuclio: end-code<jupyter_output><empty_output><jupyter_text>Run the Data-Ingestion Function MLRun enables you to run function code either locally in your preferred development environment (IDE) or web notebook (in this case, Jupyter Notebook), or on a large cluster (in this case, the platform's distributed Kubernetes cluster).For either execution method, you can specify function parameters, inputs, and outputs either by passing them as parameters to the run method (as done in this tutorial) or by configuring them in a task object that is then passed to the run method; (for more information about tasks, see Step 4 of this tutorial).In each run, you can specify new function inputs, parameters/hyperparameters, etc.For more information, see the [**mlrun_basics**](https://github.com/mlrun/mlrun/blob/master/examples/mlrun_basics.ipynb) example notebook.After the code execution completes, review the output, as demonstrated in the next sections.  Run the Function Locally in Jupyter Notebook Start out by running your function locally from within Jupyter Notebook.This is done by using the MLRun `run_local` method.When using this method, the function's inputs and outputs are automatically recorded and stored in the MLRun experiment and data-tracking database (**"the MLRun DB"**).<jupyter_code># Run the data-ingestion function locally in Jupyter Notebook
get_data_run = run_local(name='get_data',
                         handler=get_data,
                         inputs={'source_url': source_url},
                         project=project_name, artifact_path=artifact_path)<jupyter_output>> 2020-08-10 07:39:19,047 [info] starting run get_data uid=adc48fc97e34481b987d2e81e0501ea5  -> http://mlrun-api:8080
> 2020-08-10 07:39:19,228 [info] Saving Iris data set to /User/getting-started-tutorial/jobs/data ...<jupyter_text>Review the Run Output The output cell for your function execution contains a table with run information &mdash; including the state of the execution, all inputs and parameters, and the execution results and artifacts.Click on the `iris_dataset` artifact in the **artifacts** column to see a short summary of the data set, as illustrated in the following image:![getting_started-get_data-details](./assets/getting_started-get_data-details.png) MLRun features a graphical user interface (**"the MLRun dashboard"**), which displays detailed information about executed jobs, artifacts, and functions &mdash; including job states, inputs, parameters, artifacts, execution results, and logs.This dashboard also allows you to filter the display based on different criteria.To open the dashboard, simply select the MLRun service link (`mlrun`) from the **Services** page of the platform dashboard.On the MLRun dashboard's **Projects** home page, select the **Jobs** link for the current project ("getting-started-iris") to open a **Jobs** page with a **Monitor** tab that displays information about the project's executed jobs: Select the `get_data` function to see detailed information about the function and its execution &mdash; including the execution start time and status, metadata, inputs, artifacts, run results, and logs: Return to the dashboard's **Projects** home page and select the **Artifacts** link for the current project to open an **Artifacts** page with a **Monitor** tab that displays information about your project's artifacts.Select the "get_data_iris_dataset" artifact to view information about the "iris_dataset" artifact of the `get_data` function (**iris_dataset.csv**); note that you can also preview the data set and see related metadata: > **Note:** There are also other ways that you can view run artifacts, in addition to the notebook run-output table and the MLRun dashboard:> - In your notebook, use Python print or display methods, such as `IPython.display`, to display the artifact of a run object, as demonstrated later in this tutorial.> - On the platform dashboard, select **Data** from the side navigation menu, and then select your data container from the **Browse** tab; the tutorial stores data in the "users" container.>   Then, navigate to your project's artifacts directory, locate the artifact file, and double-click it to view it in the browse table.>   You can also download platform data files from the dashboard to your local environment.> - In the JupyterLab file browser, navigate to the project's artifacts directory (**/User/getting-started-tutorial/jobs** for this tutorial), locate the artifact file, and select it to view the file from the Jupyter UI.  Run the Function on a Cluster One of the key benefits of Iguazio Data Science Platform is the ability to easily run any code on a distributed Kubernetes cluster and achieve better performance.To do so, you first need to convert your notebook code into a function that can run on the cluster, and then run this function.  Convert the Notebook Code Into a Function Use the MLRun `code_to_function` method to convert your notebook code into a project MLRun function &mdash; a function object with embedded code, which can run on the cluster.This will also come in handy later in the tutorial, after you create an automated pipeline.The following code converts the code of your local `get-data` data-ingestion function into a `gen_func` project function.(For more information about the `set_function` MLRun project method, which is used to add the new project function, see Step 3.)<jupyter_code>from mlrun import code_to_function

# Convert the local get_data function into a gen_func project function
gen_func = code_to_function(name='get-data')
project.set_function(gen_func)<jupyter_output><empty_output><jupyter_text>Run the Function on the Cluster After you create the project function, you can run it on the platform's distributed Kubernetes cluster.The execution code is similar to the code used for the local execution, except that the `run_local` method is replaced with the `run` method.When the execution is done, review the output as you did for the local execution.<jupyter_code>get_data_func = project.func('get-data').apply(mount_v3io())
get_data_run = get_data_func.run(name='get_data',
                                 handler='get_data',
                                 inputs={'source_url': source_url},
                                 artifact_path=artifact_path)<jupyter_output>> 2020-08-10 07:39:25,586 [info] starting run get_data uid=74faf42295d6439d8fe8e083f9f1a940  -> http://mlrun-api:8080
> 2020-08-10 07:39:25,714 [info] Job is running in the background, pod: get-data-5jd8w
> 2020-08-10 07:40:42,952 [info] Saving Iris data set to /User/getting-started-tutorial/jobs/data ...
> 2020-08-10 07:40:43,155 [info] run executed, status=completed
final state: succeeded<jupyter_text>Step 3: Explore the Data Data exploration makes up a big part of the data science work.This step demonstrates how you can use pandas DataFrames or an MLRun marketplace function to read your data set, analyze it to extract statistics, and visualize the information to gain more insight into your data.   Explore the Data with Pandas DataFrames Run the following code to use pandas DataFrames to read your data set, extract some basic statistics, and display them.<jupyter_code># Read your data set
from mlrun.run import get_dataitem
df = get_dataitem(get_data_run.outputs['iris_dataset']).as_df()
# Display a portion of the read data
df.head()
# Calculate and display the number of data-set items
print(f'Total number of rows: {len(df)}')
# Display statistics grouped by label
df.groupby(['label']).describe()<jupyter_output><empty_output><jupyter_text>Explore the Data with an MLRun Marketplace Function You can perform further data exploration by leveraging **the MLRun functions marketplace** (a.k.a. "the MLRun functions hub").This marketplace is a centralized location for open-source contributions of function components that are commonly used in machine-learning development.The location of the marketplace is configured via the `hub_url` MLRun configuration.By default, it points to the [mlrun/functions](https://github.com/mlrun/functions/) GitHub repository.In this step you'll use the [`describe` marketplace function](https://github.com/mlrun/functions/tree/master/describe), which performs data exploration on a provided data set, to extract information from your data set, analyze it, and visualize relevant information in different ways.  Add an Exploration Function Use the `set_function` MLRun project method, which adds or updates a function object in a project, to load the `describe` marketplace function into a new `describe` project function. The tutorial code sets the first two `set_function` method parameters:- **`func`** &mdash; identities the function to load.  This parameter can be set to a function object or to the URL of a function specification or code file.  MLRun supports multiple types of URL formats.  The example uses the `hub://...` format to point to the `describe` function-code directory in the MLRun functions marketplace (`'hub://describe'`).- **`name`** &mdash; the name of the project function that is being set.  The example names the project function `describe` (like the name of the marketplace function, although this isn't required).After you add the function project, create a `describe` function object by using `apply(mount_v3io())` to attach an Iguazio Data Science Platform data volume (a.k.a. "a v3io volume") to your `describe` project function.This connects your function to the platform's shared file system and allows you to pass data from the platform to the project function and get back the results (plots) directly into your notebook.You can also select to use other supported mount methods &mdash; for example, for using NFS or object storage.<jupyter_code>describe = project.set_function('hub://describe', 'describe').apply(mount_v3io())<jupyter_output><empty_output><jupyter_text>View the Function Documentation Use the `doc` method to view the embedded documentation of the `describe` function.<jupyter_code>describe.doc()<jupyter_output>function: describe
describe and visualizes dataset stats
default handler: summarize
entry points:
  summarize: Summarize a table
    context(MLClientCtx)  - the function context, default=
    table(DataItem)  - MLRun input pointing to pandas dataframe (csv/parquet file path), default=
    label_column(str)  - ground truth column label, default=None
    class_labels(List[str])  - label for each class in tables and plots, default=[]
    plot_hist(bool)  - (True) set this to False for large tables, default=True
    plots_dest(str)  - destination folder of summary plots (relative to artifact_path), default=plots
    update_dataset  - when the table is a registered dataset update the charts in-place, default=False<jupyter_text>Run the Exploration Function Run the following code to execute the `describe` project function as a Kubernetes job by using the MLRun `run` method.The returned run object is stored in a `describe_run` variable.The location of the data set is the only input that you need to provide.This information is provided as a `table` input artifact that points to the `iris_dataset` output artifact of the `get_data` job that you ran in the previous steps (using the job's `get_data_run` run object &mdash; see Step 2).<jupyter_code>describe_run = describe.run(params={'label_column': 'label'},
                            inputs={"table":
                                    get_data_run.outputs['iris_dataset']},
                            artifact_path=artifact_path)<jupyter_output>> 2020-08-10 07:41:04,040 [info] starting run describe-summarize uid=a4a250b2a2d04f80a5d36fd04e5384bd  -> http://mlrun-api:8080
> 2020-08-10 07:41:04,170 [info] Job is running in the background, pod: describe-summarize-snf86
> 2020-08-10 07:41:17,478 [info] run executed, status=completed
final state: succeeded<jupyter_text>Review the Run Output The output cell for your code execution contains a run-information table.You can also view run information in the MLRun dashboard; see the output-review information in Step 2, only this time look for the `describe-summarize` job and related artifacts.The `describe` function generates three HTML output artifacts, which provide visual insights for your data set &mdash; `histograms`, `imbalance`, and `correlation`.The artifacts are stored as HTML files in your project's artifacts directory, under **/User/getting-started-tutorial/jobs/plots/**.The following code displays the artifact files in the notebook.<jupyter_code>from IPython.display import display, HTML

# Display the `histograms` artifact
display(HTML('<h3>Histograms</h3>'), HTML(describe_run.outputs['histograms']))
# Display the `imbalance` artifact
display(HTML('<h3>Imbalance<h3>'),
        HTML(filename=describe_run.outputs['imbalance']))
# Display the `correlation` artifact
display(HTML('<h3>Correlation Matrix<h3>'),
        HTML(filename=describe_run.outputs['correlation']))<jupyter_output><empty_output><jupyter_text>Step 4: Run Model Training at Scale It's time to train your model.Start by adding a training project function that uses the [`sklearn_classifier` MLRun marketplace function](https://github.com/mlrun/functions/tree/master/sklearn_classifier) to train three scikit-learn models; (you can use this function to train any scikit-learn model).Then, use an MLRun task to run your function in order to train and evaluate the models and select the optimal model.  Add a Training Function Run the following code to add to your project a `train` function that uses the `sklearn_classifier` marketplace function code, and create a related `train` function object.<jupyter_code>train = project.set_function('hub://sklearn_classifier', 'train').apply(mount_v3io())<jupyter_output><empty_output><jupyter_text>Run a Model-Training Task The following code uses the MLRun `NewTask` method to create a new `train` MLRun task and a related `train_task` task object for executing the `train` project function.An MLRun **task** defines the parameters, inputs, and outputs of a logical job or task to execute.The advantage of tasks is that they're independent of the runtime environment.Hence, you can define a task once and use it in different scenarios.The `NewTask` `inputs` parameter is set to the `iris_dataset` data set, as stored in the output artifacts of the previously executed `get_data` job.The `test_size` training parameter (`params.test_size`) is set to 0.10.This means that the `iris_dataset` data set will make up 90% of the training data set, and the remaining 10% will be the scikit-learn test data set.The `train_task` task (and therefore the `train` function) is executed by using the `run` method.The returned run object is stored in a `train_run` variable.The code uses the `with_hyper_params` task method, which enables running multiple iterations of the same task with different parameter combinations, comparing the results, and selecting the optimal combination based on the specified selection criteria.In this example, there's a single `models` parameter with different values (model names) so as to compare multiple models; the model with the most accurate training results is selected as the optimal model.The `with_hyper_params` method receives the following parameters- **`hyperparams`** &mdash; a dictionary of parameter names and parameter-values lists.  The example defines a single hyperparameter named `model_pkg_class`, and sets its value to a `models` variable that contains a list of three scikit-learn models to train and evaluate. - **`selector`** &mdash; a string of the format `'.'`, which defines a selection criterion for selecting the winning run iteration.  For example, `"max.accuracy"` selects the iteration with the most accurate results while `"min.loss"` selects the iteration with minimum loss.  The example uses the `max.accuracy` selector to select the model with the most accurate training results.<jupyter_code># Configure the models to train
models = ["sklearn.ensemble.RandomForestClassifier", 
          "sklearn.linear_model.LogisticRegression",
          "sklearn.ensemble.AdaBoostClassifier"]
# Create a training task
train_task = NewTask(name="train",
                     params={"sample": -1,
                             "label_column": "label",
                             "test_size": 0.10},
                     inputs={"dataset": get_data_run.outputs['iris_dataset']})

# Run the training task
train_run = train.run(train_task.with_hyper_params({'model_pkg_class': models},
                                                   selector='max.accuracy'),
                      artifact_path=artifact_path)<jupyter_output>> 2020-08-10 07:41:23,595 [info] starting run train uid=5e0690e48b82476cab38ed74485605b6  -> http://mlrun-api:8080
> 2020-08-10 07:41:23,747 [info] Job is running in the background, pod: train-n7ghs
> 2020-08-10 07:41:33,512 [info] best iteration=2, used criteria max.accuracy
> 2020-08-10 07:41:33,846 [info] run executed, status=completed
/opt/conda/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)
final state: succeeded<jupyter_text>Review the Run Output As with the previous executions, you can view extensive run information and artifacts from Jupyter Notebook and the MLRun dashboard, as well as browse the task artifacts from the platform dashboard.See the output-review information in the previous steps.Your project's artifacts directory (**/User/getting-started-tutorial/jobs**) contains an **iteration_results.csv** output artifact with the results for the training-task run iterations, and **&lt;iteration number&gt;** subdirectories with the artifacts for all the run iterations (i.e., for each evaluated model).The **plots** subdirectory has HTML output artifacts for the selected run iteration; (the **tests** subdirectory contains the artifacts for the test data set).Use the following code to extract and display information from the run object &mdash; the name of the selected trained model, the best accuracy that was achieved with this model, and the `confusion` and `roc` HTML output artifacts for the optimal run iteration.<jupyter_code># Display the name of the selected model
# best_iteration is 1 based, get the best model from the model list by substracting 1 
display(HTML(f'<b>Best model:</b> '
             f'{models[train_run.outputs["best_iteration"] - 1]}'))

# Display the accuracy for the optimal run iteration
display(HTML(f'<b>Accuracy:</b> {train_run.outputs["accuracy"]}'))

# Display HTML output artifacts
display(HTML('<h3>Confusion Matrix:</h3>'),
        HTML(filename=train_run.outputs['confusion-matrix']))
display(HTML('<h3>ROC Curve:</h3>'), HTML(filename=train_run.outputs['roc-multiclass']))<jupyter_output><empty_output><jupyter_text>Step 5: Test Your Model Now that you have a trained model, you can test it: run a task that uses the [`test_classifier` marketplace function](https://github.com/mlrun/functions/tree/master/test_classifier) to run the selected trained model against the test data set, as returned for the training task (`train`) in the previous step.  Add a Test Function Run the following code to add to your project a `test` function that uses the `test_classifier` marketplace function code, and create a related `test` function object.<jupyter_code>test = project.set_function('hub://test_classifier', 'test').apply(mount_v3io())<jupyter_output><empty_output><jupyter_text>Run a Model-Testing Task Use a task to run the model-testing function:- Create a new `test` task and a related `test_task` task object for executing the `test` project function, which runs the `test_classifier` marketplace function.  In the `NewTask` call, configure parameters for the test function (`params`), and provide the selected trained model from the `train` task as an input artifact (`inputs`).- Run the task by using the `run` MLRun method, and assign the returned run object to a `test_run` variable.<jupyter_code>test_task = NewTask(name="test",
                    params={"label_column": "label",
                            "plots_dest": path.join("plots", "test")},
                    inputs={"models_path": train_run.outputs['model'],
                            "test_set": train_run.outputs['test_set']}
                    )
test_run = test.run(test_task,
                    artifact_path=artifact_path)<jupyter_output>> 2020-08-10 07:41:43,177 [info] starting run test uid=f22b311ab0174fc2bd69e09a7a49b1e8  -> http://mlrun-api:8080
> 2020-08-10 07:41:43,311 [info] Job is running in the background, pod: test-4kfvw
> 2020-08-10 07:41:50,903 [info] run executed, status=completed
final state: succeeded<jupyter_text>Review the Run Output Check the output information for your run in Jupyter Notebook and on the MLRun dashboard.Use the following code to display information from the run object &mdash; the accuracy of the model, and the `confusion` and `roc` HTML output artifacts.<jupyter_code># Display the model accuracy
display(HTML(f'<b>Test Accuracy:</b> {test_run.outputs["accuracy"]}'))

# Display HTML output artifacts
display(HTML("<h3>Confusion matrix:</h3>"),
        HTML(filename=test_run.outputs['confusion-matrix']))
display(HTML("<h3>ROC Curve:</h3>"), HTML(filename=test_run.outputs['roc-multiclass']))<jupyter_output><empty_output><jupyter_text>Step 6: Serve Your Model Add to your project a real-time model-server function (`serving`) via the [`model_server`](https://github.com/mlrun/functions/tree/master/model_server) MLRun marketplace function.<jupyter_code>serving = project.set_function('hub://model_server', 'serving').apply(mount_v3io())<jupyter_output><empty_output><jupyter_text>Deploy the serving function Set the trained model and deploy the serving function<jupyter_code>model_name = 'iris_dataset_v1'
serving.add_model(model_name, train_run.outputs['model'])
serving_addr = serving.deploy()<jupyter_output>> 2020-08-10 07:41:52,777 [info] deploy started
[nuclio] 2020-08-10 07:41:52,797 project name not found created new (getting-started-iris-iguazio)
[nuclio] 2020-08-10 07:45:06,131 (info) Build complete
[nuclio] 2020-08-10 07:45:12,276 (info) Function deploy complete
[nuclio] 2020-08-10 07:45:12,282 done creating getting-started-iris-iguazio-sklearn-server, function address: 192.168.3.11:32148<jupyter_text>You can now run the model serving function. MLRun leverages Nuclio on Kubernetes, Nuclio scales up and down automatically and will scale to zero if remains unused. Therefore, unlike VMs that consume resources and require you to delete the endpoint when unused, here you do not have to worry about deleting this serving function from resource and cost point of view.   Test the model from the notebook Now define a test vector of inputs `test_data` and send them to the serving function `predict_url`<jupyter_code>test_data = [5.1, 3.5, 1.4, 0.2]
import json
def data_to_serving_input(data):
    return json.dumps({"instances": [data]})
predict_url = f'{serving_addr}/{model_name}/predict'
import requests
resp = requests.put(predict_url, json=data_to_serving_input(test_data))
print(resp.text)<jupyter_output>[0]<jupyter_text>Test via the a model tester function A much more robust way to test your serving function is by creating a test function that sends a test dataset and compares the output to the expected output. Add to your project a REST model API tester function (`serving-tester`) which runs the model_server function with a specified table via the [`model_server_tester`](https://github.com/mlrun/functions/tree/master/model_server_tester) MLRun marketplace function.<jupyter_code>serving_tester = project.set_function('hub://model_server_tester', 'serving-tester').apply(mount_v3io())<jupyter_output><empty_output><jupyter_text>Run the serving tester, with the test set<jupyter_code>tester = serving_tester.run(name='serving-tester',
                            params={'addr': serving_addr,
                                    'model': model_name},
                            inputs={'table': train_run.outputs['test_set']},
                            artifact_path=artifact_path)<jupyter_output>> 2020-08-10 07:45:12,539 [info] starting run serving-tester uid=61c14b69d4364219afeb0e84dc8ca986  -> http://mlrun-api:8080
> 2020-08-10 07:45:12,669 [info] Job is running in the background, pod: serving-tester-lkbth
> 2020-08-10 07:45:17,120 [info] testing with dataset against http://3.18.89.74:32148, model: iris_dataset_v1
> 2020-08-10 07:45:17,231 [info] run 15 tests, 0 errors and 14 match expected value
> 2020-08-10 07:45:17,254 [info] run executed, status=completed
final state: succeeded<jupyter_text>Test the model via the UI You can also run the model-serving function via the UI. The function name is the project name concatenated with `-sklearn server`. Use the following code to obtain the function name<jupyter_code>serving.spec.base_spec.get('metadata').get('name')<jupyter_output><empty_output><jupyter_text>Append `iris_dataset_v1/predict` to the function's URL to create the `POST` URL; change the body type to `JSON`; and add the following text to the request body:```json{"instances": [[5.1, 3.5, 1.4, 0.2]]}```You can perform these steps from the platform dashboard: select **Projects** from the side navigation menu, select your project ("getting-started-iris-iguazio"), and then select the `getting-stated-iris-iguazio-sklearn-server` function and execute the required steps, as demonstrated in the following dashboard flow:![getting_started-pipeline](./assets/getting_started-model_serving_test.gif)   Step 7: Create and Run a Fully Automated ML Pipeline You're now ready to create a full ML pipeline.This is done by using [Kubeflow Pipelines](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/), which is integrated into the Iguazio Data Science Platform.Kubeflow Pipelines is an open-source framework for building and deploying portable, scalable machine-learning workflows based on Docker containers.MLRun leverages this framework to take your existing code and deploy it as steps in the pipeline.   Define and Save a Pipeline Workflow A pipeline is created by running an MLRun **"workflow"**.The following code defines a workflow and writes it to a file in your project directory (**/User/getting-started-tutorial/jobs/workflow.py**).The workflow describes a directed acyclic graph (DAG) for execution using Kubeflow Pipelines, and depicts the connections between the functions and the data as part of an end-to-end pipeline.The workflow file has two parts &mdash; initialization of the function objects, and definition of a pipeline DSL (domain-specific language) for connecting the function inputs and outputs.Examine the code to see how functions objects are initialized and used (by name) within the workflow.The defined pipeline includes the following steps:- Ingest the Iris flower data set (`ingest`).- Analyze the data set (`describe`).- Train and test the model with hyperparameters (`train`).- Deploy the model as a real-time serverless function (`deploy`).- Test the serverless serving-model function with a test data set by using REST API calls (`Tester`).> **Note**: A pipeline can also include continuous build integration and deployment (CI/CD) steps, such as building container images and deploying models.<jupyter_code>%%writefile {path.join(project_path, 'workflow.py')}

from kfp import dsl
from mlrun import mount_v3io

funcs = {}
DATASET = 'iris_dataset'
LABELS = "label"
MODELS = ["sklearn.ensemble.RandomForestClassifier",
          "sklearn.linear_model.LogisticRegression",
          "sklearn.ensemble.AdaBoostClassifier"]


# Configure function resources and local settings
def init_functions(functions: dict, project=None, secrets=None):
    for f in functions.values():
        f.apply(mount_v3io())

    functions['serving'].metadata.name = 'getting-started-serving'


# Create a Kubeflow Pipelines pipeline
@dsl.pipeline(
    name="Getting-started tutorial",
    description="This tutorial is designed to demonstrate some of the main "
                "capabilities of the Iguazio Data Science Platform.\n"
                "The tutorial uses the Iris flower data set."
)
def kfpipeline(source_url='http://iguazio-sample-data.s3.amazonaws.com/iris_dataset.csv'):

    # Ingest the data set
    ingest = funcs['get-data'].as_step(
        name="get-data",
        handler='get_data',
        inputs={'source_url': source_url},
        params={'format': 'pq'},
        outputs=[DATASET])

    # Analyze the dataset
    describe = funcs["describe"].as_step(
        name="summary",
        params={"label_column": LABELS},
        inputs={"table": ingest.outputs[DATASET]})

    # Train a model with hyperparemeters
    train = funcs["train"].as_step(
        name="train",
        params={"sample": -1,
                "label_column": LABELS,
                "test_size": 0.10},
        hyperparams={'model_pkg_class': MODELS},
        selector='max.accuracy',
        inputs={"dataset": ingest.outputs[DATASET]},
        outputs=['model', 'test_set'])

    # Test and visualize the model
    test = funcs["test"].as_step(
        name="test",
        params={"label_column": LABELS},
        inputs={"models_path": train.outputs['model'],
                "test_set": train.outputs['test_set']})

    # Deploy the model as a serverless function
    deploy = funcs["serving"].deploy_step(
        models={f"{DATASET}_v1": train.outputs['model']})

    # Test the new model server (via REST API calls)
    tester = funcs["serving-tester"].as_step(
        name='serving-tester',
        params={'addr': deploy.outputs['endpoint'], 'model': f"{DATASET}_v1"},
        inputs={'table': train.outputs['test_set']})<jupyter_output>Writing /User/getting-started-tutorial/conf/workflow.py<jupyter_text>Register the Workflow Use the `set_workflow` MLRun project method to register your workflow with MLRun.The following code sets the `name` parameter to the selected workflow name ("main") and the `code` parameter to the name of the workflow file that is found in your project directory (**workflow.py**).<jupyter_code># Register the workflow file as "main"
project.set_workflow('main', 'workflow.py')<jupyter_output><empty_output><jupyter_text>Save Your Project Configuration Use the `save` MLRun project method to save your project definitions to a project-configuration file in your project directory (**/User/getting-started-tutorial/conf**).The default name of the project file is **project.yaml**, but you can optionally change it by setting the `filepath` parameter of the `save` method.> **Note:** It's recommended that you commit your project-configuration file and any future changes to this file to a Git repository.<jupyter_code>project.save()<jupyter_output><empty_output><jupyter_text>Run Your Pipeline Workflow Use the `run` MLRun project method to execute your workflow pipeline with Kubeflow Pipelines.The tutorial code sets the following method parameters; (for the full parameters list, see the MLRun documentation or embedded help):- **`name`** &mdash; the workflow name (in this case, "main" &mdash; see the previous step).- **`arguments`** &mdash; A dictionary of Kubeflow Pipelines arguments (parameters).  The tutorial code sets this parameter to an empty arguments list (`{}`), but you can edit the code to add arguments.- **`artifact_path`** &mdash; a path or URL that identifies a location for storing the workflow artifacts.  You can use `{{workflow.uid}}` in the path to signify the ID of the current workflow run iteration.  The tutorial code sets the artifacts path to a **&lt;worker ID&gt;** directory (`{{workflow.uid}}`) in a **pipeline** directory within the current tutorial directory (**/User/getting-started-tutorial/pipeline/&lt;worker ID&gt;**).- **`dirty`** &mdash; set to `True` to allow running the workflow also when the project's Git repository is dirty (i.e., contains uncommitted changes).  (When the notebook that contains the execution code is in the same Git directory as the executed workflow, the directory will always be dirty during the execution.)The `run` method returns the ID of the executed workflow, which the code stores in a `run_id` variable.You can use this ID to track the progress or your workflow, as demonstrated in the following sections.> **Note**: You can also run the workflow from a command-line shell by using the `mlrun` CLI.> The following CLI command defines a similar execution logic as that of the `run` call in the tutorial:> ```> mlrun project /User/getting-started-tutorial/conf -r main -p "$V3IO_HOME_URL/getting-started-tutorial/pipeline/{{workflow.uid}}/"> ```<jupyter_code>run_id = project.run(
    'main',
    arguments={}, 
    artifact_path=path.abspath(path.join('pipeline','{{workflow.uid}}')), 
    dirty=True)<jupyter_output>> 2020-08-10 07:45:18,864 [info] using in-cluster config.<jupyter_text>Review the Run Output The run output contains hyperlinks to the relevant sections for your workflow in the platform dashboard's pipelines dashboard (**Pipelines**):Select the **Experiments link** to view information about the executed experiments.![getting_started-pipeline](./assets/getting_started-pipeline-experiment.png)Select the **Run link** to view the pipeline execution.The following image illustrates the execution graph for a completed run:![getting_started-pipeline](./assets/getting_started-pipeline.png)   Track the Pipeline Workflow Execution You can use the following code to view the information for your pipeline workflow execution &mdash; including inputs and outputs, run results, and artifacts.The code uses the `get_run_db` MLRun method to get an MLRun DB object for your project's MLRun database/API service ("the MLRun DB"), and the `connect` method to connect to the DB.The returned database object is stored in a `db` variable, which can then be used to run additional methods for performing different operations on the database.The code uses the `list_runs` method to list all runs for the current workflow.The workflow is referenced by using the workflow ID that was returned for the workflow execution and saved to the `run_id` variable.<jupyter_code>from mlrun import get_run_db
db = get_run_db().connect()
db.list_runs(project=project.name, labels=f'workflow={run_id}').show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08499989259553627, 'fraction_numerical': 0.016561768307090843, 'mean_word_length': 4.949392971246007, 'pattern_counts': {'":': 21, '<': 140, '<?xml version=': 0, '>': 193, 'https://': 13, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '482106', 'license': 'Apache-2.0', 'n_tokens_mistral': 13284, 'n_tokens_neox': 12354, 'n_words': 5409, 'path': 'getting-started-tutorial/getting-started-tutorial.ipynb', 'repo_name': 'michaelk-igz/tutorials', 'size': '46563'}