<jupyter_start><jupyter_text>"Azure Databricks and MLflow Tutorial"> "In this post, I will show how to use Azure Databricks and MLflow to deploy a machine learning model."- toc: true- branch: master- badges: true- comments: true- categories: [databricks, mlflow, machine_learning, jupyter, ai]- image: images/databricks.png- hide: false- search_exclude: true- metadata_key1: metadata_value1- metadata_key2: metadata_value2  IntroductionThe dataset that I'll be using can be found on the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Heart+Disease). The goal is to train, register and deploy an ML model that will perform a classification and predict whether a person has a heart disease or not. Cluster & Notebook Creation Cluster CreationIn Azure Databricks, under **Clusters**, click on **Create Cluster**.You can use the following configuration or any other configuration depending on your business requirements.![](https://github.com/aeidme/blog/raw/master/images/notebooks/cluster-creation.png)After the cluster has been created, make sure it's running as follows:![](https://github.com/aeidme/blog/raw/master/images/notebooks/cluster-running.png) Notebook CreationMove back to the main dashboard and click on **New Notebook** under **Common Tasks**. There are also other ways to do that.You will be prompted with this dialog.![](https://github.com/aeidme/blog/raw/master/images/notebooks/nb-creation.png)Make sure to select **Python** as the language and the cluster that was created in the previous step.  Importing DataThe dataset that I'll be using can be found on the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Heart+Disease).To import a file to Databricks notebook, select **File** -> **Upload Data** and follow the instructions. At the end, you will receive a **File Path**. Make sure to copy it and save it somewhere safe.The file that I uploaded is called "**processed_cleveland.data**". It doesn't contain headers therefore I will:1. Create a list of header names called **column_names**.2. Use *header=None* and *names=column_names* with Panda's **read_csv()**.<jupyter_code>import pandas as pd

column_names = ["age", "sex", "chest_pain", "resting_blood_pressure", "cholestoral", "fasting_blood_sugar", "rest_ECG", "maximum_heart_rate", "exercice_angina", "depression_ST", "slope_ST", "major_vessels", "thal", "y"]
data = pd.read_csv("/dbfs/FileStore/shared_uploads/username/processed_cleveland.data", header=None, names=column_names)<jupyter_output><empty_output><jupyter_text>Data Analysis & PreparationThe dataset contains 14 attributes of which one is the predicted attribute 'y'. There are some missing values denoted by '?' that I will deal with at a later stage.<jupyter_code>data<jupyter_output><empty_output><jupyter_text>Classes DistributionI will plot the distribution of the classes that we have. The "goal" field refers to the presence of heart disease in the patient. It is integer valued from 0 (no presence) to 4.<jupyter_code>import seaborn as sns
sns.distplot(data.y, kde=True, hist=False)<jupyter_output><empty_output><jupyter_text>Transforming LabelsI will distinguish presence (values 1,2,3,4) from absence (value 0) of the disease. This can be done by changing the 'y' column.<jupyter_code>disease = (data.y > 0).astype(int)
data.y = disease
data<jupyter_output><empty_output><jupyter_text>Classes DistributionAfter plotting the distribution, we can clearly see that the dataset is now balanced between positive and negative classes.<jupyter_code>sns.distplot(data.y, kde=False)<jupyter_output><empty_output><jupyter_text>Correlation PlotIn order to plot the correlation matrix of the features, I will:1. Use Panda's **corr()** function after dropping the label.2. Use Seaborn's **heatmap()** function.<jupyter_code>import numpy as np
# Compute the correlation matrix
corr = data.drop(["y"], axis=1).corr()
# Plot the correlation heatmap
sns.heatmap(corr, square=True, linewidths=2)<jupyter_output><empty_output><jupyter_text>Missing ValuesThe dataset has missing values denoted with '?'. I will drop all the rows containing such values.<jupyter_code>data = data[~data.eq('?').any(1)]
data<jupyter_output><empty_output><jupyter_text>Splitting DatasetBefore training, I will split the dataset into train and test sub-datasets. This can be done using Scikit-Learn.<jupyter_code>from sklearn.model_selection import train_test_split

train, test = train_test_split(data, random_state=123)
X_train = train.drop(["y"], axis=1)
X_test = test.drop(["y"], axis=1)
y_train = train.y
y_test = test.y<jupyter_output><empty_output><jupyter_text>MLflow & TrainingIn this section, I will:1. Create a random forest classifier using Scikit-Learn.2. Use MLflow to keep track of the model metrics and save it for further usage.The *predict* function of Scikit-Learn's **RandomForestClassifier** normally returns a binary classification (0 or 1). In our case, we need to use *predict_proba* when using MLflow. Therefore, I will created a wrapper around the model called **SklearnModelWrapper**.A few things to note:1. *mlflow.start_run()* creates a new MLflow run to track the performance of this model.2. I will call *mlflow.log_param* to keep track of the number of trees in the forest.3. I will call *mlflow.log_metric* to record the area under the ROC curve.<jupyter_code>import mlflow
import mlflow.pyfunc
import mlflow.sklearn
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score 

class SklearnModelWrapper(mlflow.pyfunc.PythonModel):
  def __init__(self, model):
    self.model = model
    
  def predict(self, context, model_input):
    return self.model.predict_proba(model_input)[:,1]

with mlflow.start_run(run_name='untuned_random_forest'):
  n_estimators = 10
  model = RandomForestClassifier(n_estimators=n_estimators, random_state=np.random.RandomState(123))
  model.fit(X_train, y_train)
  # predict_proba returns [prob_negative, prob_positive], so slice the output with [:, 1]
  predictions_test = model.predict_proba(X_test)[:,1]
  auc_score = roc_auc_score(y_test, predictions_test)
  mlflow.log_param('n_estimators', n_estimators)
  mlflow.log_metric('auc', auc_score)
  wrappedModel = SklearnModelWrapper(model)
  mlflow.pyfunc.log_model("random_forest_model", python_model=wrappedModel)<jupyter_output><empty_output><jupyter_text>Feature ImportanceIn many business cases, it's important to interpret the models. Feature importances give us a clear idea of the important features that determines the prediction (in our case, whether a person has a heart disease or not).A few conclusions can be drawn from this:1. Number of major vessels, Thal and maximum heart rate play a huge role in determining whether the person has a disease or not.2. Sex and ECG at rest aren't very important.<jupyter_code>feature_importances = pd.DataFrame(model.feature_importances_, index=X_train.columns.tolist(), columns=['importance'])
feature_importances.sort_values('importance', ascending=False)<jupyter_output><empty_output><jupyter_text>Registering the model in the MLflow Model RegistryThis step can be done using the UI or programmatically. However, I will do it using the following code.First step is getting the *run_id* for the previously ran mlflow. Getting the run_id<jupyter_code>run_id = mlflow.search_runs(filter_string='tags.mlflow.runName = "untuned_random_forest"').iloc[0].run_id
run_id<jupyter_output><empty_output><jupyter_text>Registering the model<jupyter_code>model_name = "heart_disease"
model_version = mlflow.register_model(f"runs:/{run_id}/random_forest_model", model_name)<jupyter_output><empty_output><jupyter_text>Perform a sanity check by going to **Models**, you will be able to see the registered model as follows:![](https://github.com/aeidme/blog/raw/master/images/notebooks/registered-model.png)  Deploying to productionThe next step is transitioning the model version to *Production*. This can be done using the MLflowClient as follows.<jupyter_code>from mlflow.tracking import MlflowClient

client = MlflowClient()
client.transition_model_version_stage(
  name=model_name,
  version=model_version.version,
  stage="Production"
)
model = mlflow.pyfunc.load_model(f"models:/{model_name}/production")

# Sanity-check: This should match the AUC logged by MLflow
print(f'AUC: {roc_auc_score(y_test, model.predict(X_test))}')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09562225649543243, 'fraction_numerical': 0.005220073555581919, 'mean_word_length': 6.401229148375768, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 56, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '316460', 'license': 'Apache-2.0', 'n_tokens_mistral': 2627, 'n_tokens_neox': 2508, 'n_words': 842, 'path': '_notebooks/2020-03-14-databricks-heart-disease-using-mlflow.ipynb', 'repo_name': 'aeidme/blog', 'size': '8429'}