<commit_before># Setting up an MLOps Environment with Cloud Composer and MLflow on Google Cloud

We describe how to provision and MLOps environment using [Cloud Composer](https://cloud.google.com/composer)
and [MLflow](https://mlflow.org/) on Google Cloud. This environment enables deploying and executing
ML continuous training pipelines, as well as managing and tracking ML experiments, metadata, and artifacts.

We assume that you already have a [GCP Project](https://cloud.google.com/cloud-resource-manager), with 
[billing enabled](https://cloud.google.com/billing/docs/how-to/modify-project). You will also need
[Project Editor](https://cloud.google.com/iam/docs/understanding-roles) permission to provision this environment.

## Environment overview

The diagram shows the overall MLOps environment

![architecture](../../images/mlops-composer-mlflow.png)

The proposed MLOps environment consists of the following components:

| Component         | Description |
|-----------------|-------------|
|[AI Platform Notebooks](https://cloud.google.com/ai-platform-notebooks) | A managed JupyterLab instances for ML experimentation. The instance uses a custom container with MLflow installed and Cloud Proxy to connect to Cloud SQL
|[Cloud Composer](https://cloud.google.com/composer) | A managed [Airflow](https://airflow.apache.org/) service to deploy, schedule, and orchestration  run ML training pipelines. It runs on [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine).
|[MLflow](https://mlflow.org/) | An open source platform for managing the ML ifecycle.  In this environment, MLflow used for tracking ML metadata and artifacts.
|[Cloud SQL](https://cloud.google.com/sql/docs) | A managed relational database service. In this environment, Cloud SQL used as a backend for the [MLFlow tracking](https://www.mlflow.org/docs/latest/tracking.html#tracking).
|[Cloud Storage](https://cloud.google.com/storage) | A simple, reliable, and highly-durable object store. In this environment, Cloud Storage is used as an artifact store, where the outputs of ML pipeline steps are saved.
|[Cloud Build](https://cloud.google.com/cloud-build)| A managed service for executing CI/CD routines on Google Cloud infrastructure. In this environment, Cloud Build is used to build the AI Notebooks instance custom container image, as well as  ML model training runtimes.
|[Container Registry](https://cloud.google.com/container-registry)| A single place for a team to store and manage Docker images. In this environment, Container Registry is used to store the container images that are produced by Cloud Build.

## Installation Script

Provisioning of the environment has been automated with the [install.sh](install.sh) script.

The following tables describes the parameters required for the installation:

| Parameter       | Optional | Default       | Description                                                                                                                       |
|-----------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------------------------|
| PROJECT_ID      | Required |               | The project id of your GCP project                                                                                                    |
| SQL_PASSWORD    | Required |               | The password for the Cloud SQL root user                                                                                          |
| DEPLOYMENT_NAME | Optional | mlops         | Short name prefix of infrastructure element and folder names                                                                      |
| REGION          | Optional | us-central1   | A GCP region across the globe. Best to select one of the nearest.                                                                 |
| ZONE            | Optional | us-central1-a | A zone is an isolated location within a region. Available Regions and Zones: https://cloud.google.com/compute/docs/regions-zones' |


The installation script performs the following steps:

1. Enables necessary APIs
2. Creates a Cloud Storage bucket as an artifact store
3. Creates Cloud SQL instance
4. Provisions a Composer cluster environment in GKE
5. Deploys MLflow server to Composer GKE cluster
6. Create a AI Platform Notebooks instance using a custom image.

### 1. Enable the required APIs

In addition to the [services enabled by default](https://cloud.google.com/service-usage/docs/enabled-service), the following additional services must be enabled

- Compute Engine
- Container Registry
- Cloud Build
- Cloud Composer


### 2. Create a Cloud Storage bucket

A Cloud Storage bucket is required as an ML **artifact store**. The artifacts produced by the various ML steps, such as 
data splits, trained models, evaluation metrics, will be saved in this bucket.

The Cloud Storage bucket name will be set to **<DEPLOYMENT_NAME>-artifact-store**.

### 3. Provisioning a Cloud SQL instance

A Cloud SQL instance is used as a the MLflow backend to store ML metadata and pointers to artifacts stored
in the Cloud Storage bucket.

The Cloud SQL instance name will be set to **<DEPLOYMENT_NAME>-sql**, and the **root** user password will be set to <SQL_PASSWORD>.
A database named **mlflow** will be created in the Cloud SQL instance.

For more information, see [Cloud SQL setup and available machine types](https://cloud.google.com/sql/docs/mysql/create-instance#gcloud>).


### 4. Provisioning Cloud Composer

Cloud Composer will run ML training pipelines, implemented in  Airflow, on a managed environment.
Cloud Composer installation created a GKE cluster in the `REGION` and `ZONE` provided.

The Cloud Composer cluster name will be set to **<DEPLOYMENT_NAME>-af**

By default, the `--machine-type` is set to `n1-standard-2`, `--node-count` is set to 3, and `--python-version` is set to 3.

After the Cloud Composer is provisioned, Python packages in the [requirements.txt](requirements.txt) file are installed
to the Composer runtime. The file includes MLflow, SciPy, and Scikit-learn. For more information, 
see [Creating environments](https://cloud.google.com/composer/docs/how-to/managing/creating) in the Cloud Composer documentation.

> Note that the ML steps like data pre-processing and model training and evaluation, where large compute<commit_msg>Removed Notebook step from script steps list.<commit_after># Setting up an MLOps Environment with Cloud Composer and MLflow on Google Cloud

We describe how to provision and MLOps environment using [Cloud Composer](https://cloud.google.com/composer)
and [MLflow](https://mlflow.org/) on Google Cloud. This environment enables deploying and executing
ML continuous training pipelines, as well as managing and tracking ML experiments, metadata, and artifacts.

We assume that you already have a [GCP Project](https://cloud.google.com/cloud-resource-manager), with 
[billing enabled](https://cloud.google.com/billing/docs/how-to/modify-project). You will also need
[Project Editor](https://cloud.google.com/iam/docs/understanding-roles) permission to provision this environment.

## Environment overview

The diagram shows the overall MLOps environment

![architecture](../../images/mlops-composer-mlflow.png)

The proposed MLOps environment consists of the following components:

| Component         | Description |
|-----------------|-------------|
|[AI Platform Notebooks](https://cloud.google.com/ai-platform-notebooks) | A managed JupyterLab instances for ML experimentation. The instance uses a custom container with MLflow installed and Cloud Proxy to connect to Cloud SQL
|[Cloud Composer](https://cloud.google.com/composer) | A managed [Airflow](https://airflow.apache.org/) service to deploy, schedule, and orchestration  run ML training pipelines. It runs on [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine).
|[MLflow](https://mlflow.org/) | An open source platform for managing the ML ifecycle.  In this environment, MLflow used for tracking ML metadata and artifacts.
|[Cloud SQL](https://cloud.google.com/sql/docs) | A managed relational database service. In this environment, Cloud SQL used as a backend for the [MLFlow tracking](https://www.mlflow.org/docs/latest/tracking.html#tracking).
|[Cloud Storage](https://cloud.google.com/storage) | A simple, reliable, and highly-durable object store. In this environment, Cloud Storage is used as an artifact store, where the outputs of ML pipeline steps are saved.
|[Cloud Build](https://cloud.google.com/cloud-build)| A managed service for executing CI/CD routines on Google Cloud infrastructure. In this environment, Cloud Build is used to build the AI Notebooks instance custom container image, as well as  ML model training runtimes.
|[Container Registry](https://cloud.google.com/container-registry)| A single place for a team to store and manage Docker images. In this environment, Container Registry is used to store the container images that are produced by Cloud Build.

## Installation Script

Provisioning of the environment has been automated with the [install.sh](install.sh) script.

The following tables describes the parameters required for the installation:

| Parameter       | Optional | Default       | Description                                                                                                                       |
|-----------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------------------------|
| PROJECT_ID      | Required |               | The project id of your GCP project                                                                                                    |
| SQL_PASSWORD    | Required |               | The password for the Cloud SQL root user                                                                                          |
| DEPLOYMENT_NAME | Optional | mlops         | Short name prefix of infrastructure element and folder names                                                                      |
| REGION          | Optional | us-central1   | A GCP region across the globe. Best to select one of the nearest.                                                                 |
| ZONE            | Optional | us-central1-a | A zone is an isolated location within a region. Available Regions and Zones: https://cloud.google.com/compute/docs/regions-zones' |


The installation script performs the following steps:

1. Enables necessary APIs
2. Creates a Cloud Storage bucket as an artifact store
3. Creates Cloud SQL instance
4. Provisions a Composer cluster environment in GKE
5. Deploys MLflow server to Composer GKE cluster

### 1. Enable the required APIs

In addition to the [services enabled by default](https://cloud.google.com/service-usage/docs/enabled-service), the following additional services must be enabled

- Compute Engine
- Container Registry
- Cloud Build
- Cloud Composer
- Dataflow
- SQL Admin
- Notebooks

### 2. Create a Cloud Storage bucket

A Cloud Storage bucket is required as an ML **artifact store**. The artifacts produced by the various ML steps, such as 
data splits, trained models, evaluation metrics, will be saved in this bucket.

The Cloud Storage bucket name will be set to **<DEPLOYMENT_NAME>-artifact-store**.

### 3. Provisioning a Cloud SQL instance

A Cloud SQL instance is used as a the MLflow backend to store ML metadata and pointers to artifacts stored
in the Cloud Storage bucket.

The Cloud SQL instance name will be set to **<DEPLOYMENT_NAME>-sql**, and the default **root** user password will be set to <SQL_PASSWORD>.
A database named **mlflow** will be created in the Cloud SQL instance.

For more information, see [Cloud SQL setup and available machine types](https://cloud.google.com/sql/docs/mysql/create-instance#gcloud>).


### 4. Provisioning Cloud Composer

Cloud Composer will run ML training pipelines, implemented in  Airflow, on a managed environment.
Cloud Composer installation created a GKE cluster in the `REGION` and `ZONE` provided.

The Cloud Composer cluster name will be set to **<DEPLOYMENT_NAME>-af**

By default, the `--machine-type` is set to `n1-standard-2`, `--node-count` is set to 3, and `--python-version` is set to 3.

After the Cloud Composer is provisioned, Python packages in the [requirements.txt](requirements.txt) file are installed
to the Composer runtime. The file includes MLflow, SciPy, and Scikit-learn. For more information, 
see [Creating environments](https://cloud.google.com/composer/docs/how-to/managing/creating) in the Cloud Composer documentation.

> Note that the ML steps like data pre-processing and model training and evaluation, where large compute
{'fraction_non_alphanumeric': 0.10629829079840787, 'fraction_numerical': 0.0024194177788183875, 'mean_word_length': 2.7239174658529497, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 15, 'https://': 38, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 34, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4933435', 'n_tokens_mistral': 3032, 'n_tokens_neox': 2904, 'n_words': 1375}