<jupyter_start><jupyter_text>XGBoost trainingThis is an auto-generated notebook. To reproduce these results, attach this notebook to the **10-3-ML-Cluster** cluster and rerun it.- Compare trials in the [MLflow experiment](mlflow/experiments/406583024052808/s?orderByKey=metrics.%60val_f1_score%60&orderByAsc=false)- Navigate to the parent notebook [here](notebook/406583024052798) (If you launched the AutoML experiment using the Experiments UI, this link isn't very useful.)- Clone this notebook into your project folder by selecting **File > Clone** in the notebook toolbar.Runtime Version: _10.3.x-cpu-ml-scala2.12_<jupyter_code>import mlflow
import databricks.automl_runtime

# Use MLflow to track experiments
mlflow.set_experiment("/Users/<EMAIL>/databricks_automl/label_news_articles_csv-2022_03_12-15_38")

target_col = "label"<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>from mlflow.tracking import MlflowClient
import os
import uuid
import shutil
import pandas as pd

# Create temp directory to download input data from MLflow
input_temp_dir = os.path.join(os.environ["SPARK_LOCAL_DIRS"], "tmp", str(uuid.uuid4())[:8])
os.makedirs(input_temp_dir)


# Download the artifact and read it into a pandas DataFrame
input_client = MlflowClient()
input_data_path = input_client.download_artifacts("c2dfe80b419d4a8dbc88a90e3274369a", "data", input_temp_dir)

df_loaded = pd.read_parquet(os.path.join(input_data_path, "training_data"))
# Delete the temp data
shutil.rmtree(input_temp_dir)

# Preview data
df_loaded.head(5)
df_loaded.head(1).to_dict()<jupyter_output><empty_output><jupyter_text>Select supported columnsSelect only the columns that are supported. This allows us to train a model that can predict on a dataset that has extra columns that are not used in training.`[]` are dropped in the pipelines. See the Alerts tab of the AutoML Experiment page for details on why these columns are dropped.<jupyter_code>from databricks.automl_runtime.sklearn.column_selector import ColumnSelector
supported_cols = ["text_without_stopwords", "published", "language", "main_img_url", "site_url", "hasImage", "title_without_stopwords", "text", "title", "type", "author"]
col_selector = ColumnSelector(supported_cols)<jupyter_output><empty_output><jupyter_text>Preprocessors<jupyter_code>transformers = []<jupyter_output><empty_output><jupyter_text>Categorical columns  Low-cardinality categoricalsConvert each low-cardinality categorical column into multiple binary columns through one-hot encoding.For each input categorical column (string or numeric), the number of output columns is equal to the number of unique values in the input column.<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder

one_hot_encoder = OneHotEncoder(handle_unknown="ignore")

transformers.append(("onehot", one_hot_encoder, ["published", "language", "site_url", "hasImage", "title", "title_without_stopwords", "text_without_stopwords"]))<jupyter_output><empty_output><jupyter_text>Medium-cardinality categoricalsConvert each medium-cardinality categorical column into a numerical representation.Each string column is hashed to 1024 float columns.Each numeric column is imputed with zeros.<jupyter_code>from sklearn.feature_extraction import FeatureHasher
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

for feature in ["text", "main_img_url"]:
    hash_transformer = Pipeline(steps=[
        ("imputer", SimpleImputer(missing_values=None, strategy="constant", fill_value="")),
        (f"{feature}_hasher", FeatureHasher(n_features=1024, input_type="string"))])
    transformers.append((f"{feature}_hasher", hash_transformer, [feature]))<jupyter_output><empty_output><jupyter_text>Text featuresConvert each feature to a fixed-length vector using TF-IDF vectorization. The length of the outputvector is equal to 1024. Each column corresponds to one of the top word n-gramswhere n is in the range [1, 2].<jupyter_code>import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer

for col in {'type', 'author'}:
    vectorizer = Pipeline(steps=[
        ("imputer", SimpleImputer(missing_values=None, strategy="constant", fill_value="")),
        # Reshape to 1D since SimpleImputer changes the shape of the input to 2D
        ("reshape", FunctionTransformer(np.reshape, kw_args={"newshape":-1})),
        ("tfidf", TfidfVectorizer(decode_error="ignore", ngram_range = (1, 2), max_features=1024))])

    transformers.append((f"text_{col}", vectorizer, [col]))
from sklearn.compose import ColumnTransformer

preprocessor = ColumnTransformer(transformers, remainder="passthrough", sparse_threshold=0)<jupyter_output><empty_output><jupyter_text>Feature standardizationScale all feature columns to be centered around zero with unit variance.<jupyter_code>from sklearn.preprocessing import StandardScaler

standardizer = StandardScaler()<jupyter_output><empty_output><jupyter_text>Train - Validation - Test SplitSplit the input data into 3 sets:- Train (60% of the dataset used to train the model)- Validation (20% of the dataset used to tune the hyperparameters of the model)- Test (20% of the dataset used to report the true performance of the model on an unseen dataset)<jupyter_code>df_loaded.columns
from sklearn.model_selection import train_test_split

split_X = df_loaded.drop([target_col], axis=1)
split_y = df_loaded[target_col]

# Split out train data
X_train, split_X_rem, y_train, split_y_rem = train_test_split(split_X, split_y, train_size=0.6, random_state=799811440, stratify=split_y)

# Split remaining data equally for validation and test
X_val, X_test, y_val, y_test = train_test_split(split_X_rem, split_y_rem, test_size=0.5, random_state=799811440, stratify=split_y_rem)<jupyter_output><empty_output><jupyter_text>Train classification model- Log relevant metrics to MLflow to track runs- All the runs are logged under [this MLflow experiment](mlflow/experiments/406583024052808/s?orderByKey=metrics.%60val_f1_score%60&orderByAsc=false)- Change the model parameters and re-run the training cell to log a different trial to the MLflow experiment- To view the full list of tunable hyperparameters, check the output of the cell below<jupyter_code>from xgboost import XGBClassifier

help(XGBClassifier)
import mlflow
import sklearn
from sklearn import set_config
from sklearn.pipeline import Pipeline

set_config(display="diagram")

xgbc_classifier = XGBClassifier(
  colsample_bytree=0.7324555878929649,
  learning_rate=0.007636627530856404,
  max_depth=7,
  min_child_weight=6,
  n_estimators=106,
  n_jobs=100,
  subsample=0.6972187716458148,
  verbosity=0,
  random_state=799811440,
)

model = Pipeline([
    ("column_selector", col_selector),
    ("preprocessor", preprocessor),
    ("standardizer", standardizer),
    ("classifier", xgbc_classifier),
])

# Create a separate pipeline to transform the validation dataset. This is used for early stopping.
pipeline = Pipeline([
    ("column_selector", col_selector),
    ("preprocessor", preprocessor),
    ("standardizer", standardizer),
])

mlflow.sklearn.autolog(disable=True)
X_val_processed = pipeline.fit_transform(X_val, y_val)

model
# Enable automatic logging of input samples, metrics, parameters, and models
mlflow.sklearn.autolog(log_input_examples=True, silent=True)

with mlflow.start_run(run_name="xgboost") as mlflow_run:
    model.fit(X_train, y_train, classifier__early_stopping_rounds=5, classifier__eval_set=[(X_val_processed,y_val)], classifier__verbose=False)
    
    # Training metrics are logged by MLflow autologging
    # Log metrics for the validation set
    xgbc_val_metrics = mlflow.sklearn.eval_and_log_metrics(model, X_val, y_val, prefix="val_")

    # Log metrics for the test set
    xgbc_test_metrics = mlflow.sklearn.eval_and_log_metrics(model, X_test, y_test, prefix="test_")

    # Display the logged metrics
    xgbc_val_metrics = {k.replace("val_", ""): v for k, v in xgbc_val_metrics.items()}
    xgbc_test_metrics = {k.replace("test_", ""): v for k, v in xgbc_test_metrics.items()}
    display(pd.DataFrame([xgbc_val_metrics, xgbc_test_metrics], index=["validation", "test"]))
# Patch requisite packages to the model environment YAML for model serving
import os
import shutil
import uuid
import yaml

None

import xgboost
from mlflow.tracking import MlflowClient

xgbc_temp_dir = os.path.join(os.environ["SPARK_LOCAL_DIRS"], str(uuid.uuid4())[:8])
os.makedirs(xgbc_temp_dir)
xgbc_client = MlflowClient()
xgbc_model_env_path = xgbc_client.download_artifacts(mlflow_run.info.run_id, "model/conda.yaml", xgbc_temp_dir)
xgbc_model_env_str = open(xgbc_model_env_path)
xgbc_parsed_model_env_str = yaml.load(xgbc_model_env_str, Loader=yaml.FullLoader)

xgbc_parsed_model_env_str["dependencies"][-1]["pip"].append(f"xgboost=={xgboost.__version__}")

with open(xgbc_model_env_path, "w") as f:
  f.write(yaml.dump(xgbc_parsed_model_env_str))
xgbc_client.log_artifact(run_id=mlflow_run.info.run_id, local_path=xgbc_model_env_path, artifact_path="model")
shutil.rmtree(xgbc_temp_dir)<jupyter_output><empty_output><jupyter_text>Feature importanceSHAP is a game-theoretic approach to explain machine learning models, providing a summary plotof the relationship between features and model output. Features are ranked in descending order ofimportance, and impact/color describe the correlation between the feature and the target variable.- Generating SHAP feature importance is a very memory intensive operation, so to ensure that AutoML can run trials without  running out of memory, we disable SHAP by default.  You can set the flag defined below to `shap_enabled = True` and re-run this notebook to see the SHAP plots.- To reduce the computational overhead of each trial, a single example is sampled from the validation set to explain.  For more thorough results, increase the sample size of explanations, or provide your own examples to explain.- SHAP cannot explain models using data with nulls; if your dataset has any, both the background data and  examples to explain will be imputed using the mode (most frequent values). This affects the computed  SHAP values, as the imputed samples may not match the actual data distribution.For more information on how to read Shapley values, see the [SHAP documentation](https://shap.readthedocs.io/en/latest/example_notebooks/overviews/An%20introduction%20to%20explainable%20AI%20with%20Shapley%20values.html).<jupyter_code># Set this flag to True and re-run the notebook to see the SHAP plots
shap_enabled = True
if shap_enabled:
    from shap import KernelExplainer, summary_plot
    # SHAP cannot explain models using data with nulls.
    # To enable SHAP to succeed, both the background data and examples to explain are imputed with the mode (most frequent values).
    mode = X_train.mode().iloc[0]

    # Sample background data for SHAP Explainer. Increase the sample size to reduce variance.
    train_sample = X_train.sample(n=min(100, len(X_train.index))).fillna(mode)

    # Sample a single example from the validation set to explain. Increase the sample size and rerun for more thorough results.
    example = X_val.sample(n=1).fillna(mode)

    # Use Kernel SHAP to explain feature importance on the example from the validation set.
    predict = lambda x: model.predict_proba(pd.DataFrame(x, columns=X_train.columns))
    explainer = KernelExplainer(predict, train_sample, link="logit")
    shap_values = explainer.shap_values(example, l1_reg=False)
    summary_plot(shap_values, example, class_names=model.classes_)<jupyter_output><empty_output><jupyter_text>Inference[The MLflow Model Registry](https://docs.databricks.com/applications/mlflow/model-registry.html) is a collaborative hub where teams can share ML models, work together from experimentation to online testing and production, integrate with approval and governance workflows, and monitor ML deployments and their performance. The snippets below show how to add the model trained in this notebook to the model registry and to retrieve it later for inference.> **NOTE:** The `model_uri` for the model already trained in this notebook can be found in the cell below Register to Model Registry```model_name = "Example"model_uri = f"runs:/{ mlflow_run.info.run_id }/model"registered_model_version = mlflow.register_model(model_uri, model_name)``` Load from Model Registry```model_name = "Example"model_version = registered_model_version.versionmodel = mlflow.pyfunc.load_model(model_uri=f"models:/{model_name}/{model_version}")model.predict(input_X)``` Load model without registering```model_uri = f"runs:/{ mlflow_run.info.run_id }/model"model = mlflow.pyfunc.load_model(model_uri)model.predict(input_X)```<jupyter_code># model_uri for the generated model
print(f"runs:/{ mlflow_run.info.run_id }/model")<jupyter_output><empty_output><jupyter_text>Loading model to make prediction<jupyter_code>model_uri = f"runs:/51c0348482e042ea8e4b7983ab6bff99/model"

model = mlflow.pyfunc.load_model(model_uri)
#model.predict(input_X)
import pandas as pd
data = {'author': {0: 'bigjim.com'},
 'published': {0: '2016-10-27T18:05:26.351+03:00'},
 'title': {0: 'aliens are coming to invade earth'},
 'text': {0: 'aliens are coming to invade earth'},
 'language': {0: 'english'},
 'site_url': {0: 'cnn.com'},
 'main_img_url': {0: 'https://2.bp.blogspot.com/-0mdp0nZiwMI/UYwYvexmW2I/AAAAAAAAVQM/7C_X5WRE_mQ/w1200-h630-p-nu/Edison-Stock-Ticker.jpg'},
 'type': {0: 'bs'},
 'title_without_stopwords': {0: 'aliens are coming to invade earth'},
 'text_without_stopwords': {0: 'aliens are coming to invade earth'},
 'hasImage': {0: 1.0}}
df = pd.DataFrame(data=data)
df.head()
model.predict(df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09372064455524243, 'fraction_numerical': 0.02283401979911843, 'mean_word_length': 6.525829255029907, 'pattern_counts': {'":': 1, '<': 54, '<?xml version=': 0, '>': 56, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '487512', 'license': 'CC0-1.0', 'n_tokens_mistral': 4563, 'n_tokens_neox': 4303, 'n_words': 1243, 'path': 'XGBoost-fake-news-automl.ipynb', 'repo_name': 'xzhnshng/mlflow-project-best-practices', 'size': '13851'}