<jupyter_start><jupyter_text>Inventory prediction on ecommerce data using Vertex AI Table of contents* [Overview](section-1)* [Dataset](section-2)* [Objective](section-3)* [Costs](section-4)* [Load the required data from BigQuery](section-5)* [Explore and analyze the dataset](section-6)* [Feature preprocessing](section-7)* [Model building](section-8)   * [Train the model](section-9)  * [Evaluate the model](section-10)* [Save the model to a Cloud Storage bucket](section-11) * [Create a model in Vertex AI](section-12) * [Create an endpoint](section-13) * [Deploy the model to the created endpoint](section-14)* [What-If Tool](section-15) * [Clean up](section-16)  OverviewThis notebook explores how to build a machine learning model for inventory prediction on an ecommerce dataset. This notebook includes steps for deploying the model on Vertex AI using the Vertex AI SDK and analyzing the deployed model using the [What-If Tool](https://pair-code.github.io/what-if-tool/).*Note: This notebook file was designed to run in a Vertex AI Workbench managed notebooks instance using the TensorFlow 2 (Local) kernel. Some components of this notebook may not work in other notebook environments.* DatasetThe dataset used in this notebook consists of inventory data since 2018 for an ecommerce store. This dataset is publicly available as a BigQuery table named `looker-private-demo.ecomm.inventory_items`, which can be accessed by pinning the `looker-private-demo` project in BigQuery. The table consists of various fields related to ecommerce inventory items such as `id`, `product_id`, `cost`, when the item arrived at the store, and when it was sold. This notebook makes use of the following fields assuming their purpose is as described below:- `id`: The ID of the inventory item- `product_id`: The ID of the product- `created_at`: When the item arrived in the inventory/at the store- `sold_at`: When the item was sold (*Null if still unsold*)- `cost`: Cost at which the item was sold- `product_category`: Category of the product- `product_brand`: Brand of the product (dropped later as there are too many values)- `product_retail_price`: Price of the product- `product_department`: Department to which the product belonged to- `product_distribution_center_id`: Which distribution center (an approximation of regions) the product was sold fromThe dataset is encoded to hide any private information. For example, the distribution centers have been assigned ID numbers ranging from 1 to 10. ObjectivesThe objectives of this notebook include:* Load the dataset from BigQuery using the "BigQuery in Notebooks" integration.* Analyze the dataset.* Preprocess the features in the dataset.* Build a random forest classifier model that predicts whether a product will get sold in the next 60 days.* Evaluate the model.* Deploy the model using Vertex AI.* Configure and test the What-If Tool. CostsThis tutorial uses the following billable components of Google Cloud:- Vertex AI- BigQuery- Cloud StorageLearn about [Vertex AIpricing](https://cloud.google.com/vertex-ai/pricing), [BigQuery pricing](https://cloud.google.com/bigquery/pricing) and [Cloud Storagepricing](https://cloud.google.com/storage/pricing), and use the [PricingCalculator](https://cloud.google.com/products/calculator/)to generate a cost estimate based on your projected usage. Before you begin Set your project ID**If you don't know your project ID**, you may be able to get your project ID using `gcloud`.<jupyter_code>import os

PROJECT_ID = ""

# Get your Google Cloud project ID from gcloud
if not os.getenv("IS_TESTING"):
    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID: ", PROJECT_ID)<jupyter_output><empty_output><jupyter_text>Otherwise, set your project ID here.<jupyter_code>if PROJECT_ID == "" or PROJECT_ID is None:
    PROJECT_ID = "[your-project-id]"  # @param {type:"string"}<jupyter_output><empty_output><jupyter_text>TimestampIf you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial.<jupyter_code>from datetime import datetime

TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")<jupyter_output><empty_output><jupyter_text>Create a Cloud Storage bucketThe following steps are required, regardless of your notebook environment.When you submit a training job using the Cloud SDK, you upload a Python package containing your training code to a Cloud Storage bucket. Vertex AI runs the code from this package. In this tutorial, Vertex AI needs the trained model to be saved to Cloud Storage bucket for deployment. Using the model artifact, you can then create Vertex AI model and endpoint resources in order to serve the online predictions.Set the name of your Cloud Storage bucket below. It must be unique across all Cloud Storage buckets.You may also change the `REGION` variable, which is used for operations throughout the rest of this notebook. Make sure to choose a region where Vertex AI services are available. You may not use a Multi-Regional Storage bucket for training with Vertex AI.<jupyter_code>BUCKET_NAME = "[your-bucket-name]"
BUCKET_URI = f"gs://{BUCKET_NAME}"
REGION = "[your-region]"
if BUCKET_NAME == "" or BUCKET_NAME is None or BUCKET_NAME == "[your-bucket-name]":
    TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")
    BUCKET_NAME = PROJECT_ID + "aip-" + TIMESTAMP
    BUCKET_URI = "gs://" + BUCKET_NAME<jupyter_output><empty_output><jupyter_text>**Only if your bucket doesn't already exist:** Run the following cell to create your Cloud Storage bucket.<jupyter_code>! gsutil mb -l $REGION $BUCKET_NAME<jupyter_output><empty_output><jupyter_text>Finally, validate access to your Cloud Storage bucket by examining its contents:<jupyter_code>! gsutil ls -al $BUCKET_NAME<jupyter_output><empty_output><jupyter_text>Tutorial Import the required libraries<jupyter_code>import os

import joblib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn.metrics as metrics
from google.cloud import storage
from google.cloud.bigquery import Client
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from witwidget.notebook.visualization import WitConfigBuilder, WitWidget<jupyter_output><empty_output><jupyter_text>Load the required data from BigQueryThe following cell integrates with BigQuery data from the same project through the Vertex AI's "BigQuery in Notebooks" integration. It can run an SQL query as it would run in the BigQuery console. *Note:* This feature only works in a notebook running on a Vertex AI Workbench managed-notebook instance. @bigquerySELECT     id,    product_id,     created_at,    sold_at,    cost,    product_category,    product_brand,    product_retail_price,    product_department,    product_distribution_center_idFROM looker-private-demo.ecomm.inventory_items After executing the above cell, clicking **Query and load as DataFrame** button adds the following python cell that loads the queried data into a pandas dataframe.<jupyter_code># The following two lines are only necessary to run once.
# Comment out otherwise for speed-up.
client = Client()

query = """SELECT 
    id,
    product_id, 
    created_at,
    sold_at,
    cost,
    product_category,
    product_brand,
    product_retail_price,
    product_department,
    product_distribution_center_id
FROM 
looker-private-demo.ecomm.inventory_items"""
job = client.query(query)
df = job.to_dataframe()<jupyter_output><empty_output><jupyter_text>Explore and analyze the datasetCheck the first five rows of the dataset.<jupyter_code>df.head(5)<jupyter_output><empty_output><jupyter_text>Check the fields in the dataset and their data types and number of null values.<jupyter_code>df.info()<jupyter_output><empty_output><jupyter_text>Apart from the `sold_at` datetime field, there aren't any fields that consist of null values in the dataset. As you are dealing with the inventory-item data, it is absolutely plausible that there will be some items that haven't been sold yet and hence the null values.Next, convert the date fields to a proper date format to process them in the next steps.<jupyter_code># convert to proper date columns
df["created_at"] = pd.to_datetime(df["created_at"], format="%Y-%m-%d")
df["sold_at"] = pd.to_datetime(df["sold_at"].dt.strftime("%Y-%m-%d"))<jupyter_output><empty_output><jupyter_text>Check the date ranges.<jupyter_code># check the date ranges
print("Min-sold_at : ", df["sold_at"].min())
print("Max-sold_at : ", df["sold_at"].max())

print("Min-created_at : ", df["created_at"].min())
print("Max-created_at : ", df["created_at"].max())<jupyter_output><empty_output><jupyter_text>Extract the month from the date field `created_at`.<jupyter_code># calculate the month when the item has arrived
df["arrival_month"] = df["created_at"].dt.month<jupyter_output><empty_output><jupyter_text>Calculate the average number of days a product had been in the inventory until it was sold.<jupyter_code># calculate the number of days the item hasn't been sold.
df["shelf_days"] = (df["sold_at"] - df["created_at"]).dt.days<jupyter_output><empty_output><jupyter_text>Calculate the discount percentages that apply to the products.<jupyter_code># calculate the discount offered
df["discount_perc"] = (df["product_retail_price"] - df["cost"]) / df[
    "product_retail_price"
]<jupyter_output><empty_output><jupyter_text>Check the unique products and their brands in the data.<jupyter_code># check total unique items
df["product_id"].unique().shape, df["product_brand"].unique().shape<jupyter_output><empty_output><jupyter_text>The fields `product_id` and `product_brand` seem to have a lot of unique values. For the purpose of prediction, use `product_id` as the primary-key and `product_brand` is dropped as it has too many values/levels. Segregate the required numerical and categorical fields to analyze the dataset.<jupyter_code>categ_cols = [
    "product_category",
    "product_department",
    "product_distribution_center_id",
    "arrival_month",
]
num_cols = ["cost", "product_retail_price", "discount_perc", "shelf_days"]<jupyter_output><empty_output><jupyter_text>Check the count of individual categories for each categorical field.<jupyter_code>for i in categ_cols:
    print(i, " - ", df[i].unique().shape[0])<jupyter_output><empty_output><jupyter_text>Check the distribution of the numerical fields.<jupyter_code>df[num_cols].describe().T<jupyter_output><empty_output><jupyter_text>Generate bar plots for categorical fields and histograms and box plots for numerical fields to check their distributions in the dataset.<jupyter_code>for i in categ_cols:
    df[i].value_counts(normalize=True).plot(kind="bar")
    plt.title(i)
    plt.show()

for i in num_cols:
    _, ax = plt.subplots(1, 2, figsize=(10, 4))
    df[i].plot(kind="box", ax=ax[0])
    df[i].plot(kind="hist", ax=ax[1])
    ax[0].set_title(i + "-Boxplot")
    ax[1].set_title(i + "-Histogram")
    plt.show()<jupyter_output><empty_output><jupyter_text>Most of the fields like discount, department, distribution center-id have a decent distribution. For the field `product_category`, there are some categories that don't constitute 2% of the dataset at least. Although there are outliers in some numerical fields, they are exempted from removing as there can be products that are expensive or belonging to a particular category that doesn't often see many sales.  Feature preprocessingNext, aggregate the data based on suitable categorical fields in the data and take the average number of days it took for the product to get sold. For a given `product_id`, there can be multiple item `id`'s in this dataset and you want to predict at the product level whether that particular product is going to be sold in the next couple of months. You are aggregating the data based on each of the product configurations present in this dataset like the price, cost, category and at which center it is being sold. This way the model can predict *whether a product with so and so properties is going to be sold in the next couple of months*.For the number of days a product got sold in, find the average of the `shelf_days` field.<jupyter_code>groupby_cols = [
    "product_id",
    "product_distribution_center_id",
    "product_category",
    "product_department",
    "arrival_month",
    "product_retail_price",
    "cost",
    "discount_perc",
]
value_cols = ["shelf_days"]


df_prod = df[groupby_cols + value_cols].groupby(by=groupby_cols).mean().reset_index()<jupyter_output><empty_output><jupyter_text>Check the aggregated product level data.<jupyter_code>df_prod.head()<jupyter_output><empty_output><jupyter_text>Look for null values in the data.<jupyter_code>df_prod.isna().sum() / df.shape[0]<jupyter_output><empty_output><jupyter_text>Only the `shelf_days` field has null values that correspond to the `product_id`'s that have no sold items. Plot the distribution of the aggregated `shelf_days` field by generating a box plot.<jupyter_code>df_prod["shelf_days"].plot(kind="box")<jupyter_output><empty_output><jupyter_text>Here, you can see that most of the products are sold within 60 days since they've arrived in the inventory/store. In this tutorial, you will train a machine learning model that predicts the probability of a product being sold within 60 days. Encode the categorical fieldsEncode the the `shelf_days` field to generate the target field `sold_in_2mnt` indicating whether the product was sold in 60 days.<jupyter_code>df_prod["sold_in_2mnt"] = df_prod["shelf_days"].apply(
    lambda x: 1 if x >= 0 and x < 60 else 0
)
df_prod["sold_in_2mnt"].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Segregate the features into variables for model building.<jupyter_code>target = "sold_in_2mnt"
categ_cols = [
    "product_category",
    "product_department",
    "product_distribution_center_id",
    "arrival_month",
]
num_cols = ["product_retail_price", "cost", "discount_perc"]<jupyter_output><empty_output><jupyter_text>Encode the `product_department` field.<jupyter_code>df["product_deprtment"] = (
    df["product_department"].apply(lambda x: 1 if x == "Women" else 0).value_counts()
)<jupyter_output><empty_output><jupyter_text>Encode the rest of the categorical fields for model building.<jupyter_code># Create dummy variables for each categ. variable
for i in categ_cols:
    ml = pd.get_dummies(df_prod[i], prefix=i + "_", drop_first=True)
    df_new = pd.concat([df_prod, ml], axis=1)

df_new.drop(columns=categ_cols, inplace=True)
df_new.shape<jupyter_output><empty_output><jupyter_text>Normalize the numerical fieldsNormalize the fields `product_retail_price` and `cost` to the 0-1 scale using Min-Max normalization technique.<jupyter_code>scaler = MinMaxScaler()
scaler = scaler.fit(df_new[["product_retail_price", "cost"]])
df_new[["product_retail_price_norm", "cost_norm"]] = scaler.transform(
    df_new[["product_retail_price", "cost"]]
)<jupyter_output><empty_output><jupyter_text>Model building Train the modelCollect the required fields from the dataframe.<jupyter_code>cols = [
    "discount_perc",
    "arrival_month__2",
    "arrival_month__3",
    "arrival_month__4",
    "arrival_month__5",
    "arrival_month__6",
    "arrival_month__7",
    "arrival_month__8",
    "arrival_month__9",
    "arrival_month__10",
    "arrival_month__11",
    "arrival_month__12",
    "product_retail_price_norm",
    "cost_norm",
]<jupyter_output><empty_output><jupyter_text>Split the data into train(80%) and test(20%) sets.<jupyter_code>X = df_new[cols].copy()
y = df_new[target].copy()
train_X, test_X, train_y, test_y = train_test_split(
    X, y, train_size=0.8, test_size=0.2, random_state=7
)<jupyter_output><empty_output><jupyter_text>Create the classifier and fit it on the training data.<jupyter_code>model = RandomForestClassifier(random_state=7, n_estimators=100)
model.fit(train_X[cols], train_y)<jupyter_output><empty_output><jupyter_text>Evaluate the modelPredict on the test set and check the accuracy of the model.<jupyter_code>pred_y = model.predict(test_X[cols])

# Calculate the accuracy as our performance metric
accuracy = metrics.accuracy_score(test_y, pred_y)
print("Accuracy: ", accuracy)<jupyter_output><empty_output><jupyter_text>Generate the confusion-matrix on the test set.<jupyter_code>confusion = metrics.confusion_matrix(test_y, pred_y)
print(f"Confusion matrix:\n{confusion}")

print("\nNormalized confusion matrix:")
for row in confusion:
    print(row / row.sum())<jupyter_output><empty_output><jupyter_text>The model performance can be stated in terms of specificity (True-negative rate) and sensitivity (True-positive rate). In the normalized confusion matrix, the top left value represents the True-negative rate and the bottom right value represents the True-positive rate. Save the model to a Cloud Storage bucketNext, save the model to the created Cloud Storage bucket for deployment.<jupyter_code># save the trained model to a local file "model.joblib"
FILE_NAME = "model.joblib"
joblib.dump(model, FILE_NAME)

# Upload the saved model file to Cloud Storage
BLOB_PATH = "inventory_prediction/"
BLOB_NAME = os.path.join(BLOB_PATH, FILE_NAME)

bucket = storage.Client().bucket(BUCKET_NAME)

blob = bucket.blob(BLOB_NAME)
blob.upload_from_filename(FILE_NAME)<jupyter_output><empty_output><jupyter_text>Create a model in Vertex AISpecify the corresponding model parameters.<jupyter_code>MODEL_DISPLAY_NAME = "inventory_prediction_model"
ARTIFACT_GCS_PATH = f"gs://{BUCKET_NAME}/{BLOB_PATH}"<jupyter_output><empty_output><jupyter_text>Create a Vertex AI model resource.<jupyter_code>from google.cloud import aiplatform

aiplatform.init(project=PROJECT_ID, location=REGION)

model = aiplatform.Model.upload(
    display_name=MODEL_DISPLAY_NAME,
    artifact_uri=ARTIFACT_GCS_PATH,
    serving_container_image_uri="us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-0:latest",
)

model.wait()

print(model.display_name)
print(model.resource_name)<jupyter_output><empty_output><jupyter_text>Create an EndpointSet the display name for the endpoint.<jupyter_code>ENDPOINT_DISPLAY_NAME = "inventory_prediction_endpoint"<jupyter_output><empty_output><jupyter_text>Create an endpoint resource on Vertex AI.<jupyter_code>endpoint = aiplatform.Endpoint.create(display_name=ENDPOINT_DISPLAY_NAME)

print(endpoint.display_name)
print(endpoint.resource_name)<jupyter_output><empty_output><jupyter_text>Deploy the model to the created EndpointConfigure the deployment name, machine type, and other parameters for the deployment.<jupyter_code>DEPLOYED_MODEL_NAME = "inventory_prediction_deployment"
MACHINE_TYPE = "n1-standard-2"<jupyter_output><empty_output><jupyter_text>Deploy the model to the created endpoint.<jupyter_code>model.deploy(
    endpoint=endpoint,
    deployed_model_display_name=DEPLOYED_MODEL_NAME,
    machine_type=MACHINE_TYPE,
)

model.wait()

print("Model display-name - ", model.display_name)
print("Model resource-name - ", model.resource_name)
endpoint.list_models()<jupyter_output><empty_output><jupyter_text>Note the `DEPLOYED_MODEL_ID` for deleting the deployment during clean up.<jupyter_code>DEPLOYED_MODEL_ID = ""<jupyter_output><empty_output><jupyter_text>What-If ToolThe What-If Tool can be used to analyze the model predictions on test data. See a brief introduction to the [What-If Tool](https://pair-code.github.io/what-if-tool/get-started/). In this tutorial, the What-If Tool is configured and run on the model deployed on Vertex AI Endpoints in the previous steps.WitConfigBuilder provides the set_ai_platform_model() method to configure the What-If Tool with a model deployed as a version on Ai Platform models. This feature currently supports only Ai Platform but not Vertex AI models. Fortunately, there is also an option to pass a custom function for generating predictions through the set_custom_predict_fn() method where either the locally trained model or a function that returns predictions from a Vertex AI model can be passed. Prepare test samplesSave some samples from the test data for both the available classes (Fraud/not-Fraud) to analyze the model using the What-If Tool.<jupyter_code># collect some samples for each class-label from the test data
sample_size = 200
pos_samples = test_y[test_y == 1].sample(sample_size).index
neg_samples = test_y[test_y == 0].sample(sample_size).index
test_samples_y = pd.concat([test_y.loc[pos_samples], test_y.loc[neg_samples]])
test_samples_X = test_X.loc[test_samples_y.index].copy()<jupyter_output><empty_output><jupyter_text>Running the What-If Tool on the deployed Vertex AI modelDefine a function to fetch the predictions from the deployed model and run it on the created test data configuring the What-If tool.<jupyter_code># configure the target and class-labels
TARGET_FEATURE = target
LABEL_VOCAB = ["not-sold", "sold"]

# function to return predictions from the deployed Model


def endpoint_predict_sample(instances: list):
    prediction = endpoint.predict(instances=instances)
    preds = [[1 - i, i] for i in prediction.predictions]
    return preds


# Combine the features and labels into one array for the What-If Tool
test_examples = np.hstack(
    (test_samples_X.to_numpy(), test_samples_y.to_numpy().reshape(-1, 1))
)

# Configure the WIT with the prediction function
config_builder = (
    WitConfigBuilder(test_examples.tolist(), test_samples_X.columns.tolist() + [target])
    .set_custom_predict_fn(endpoint_predict_sample)
    .set_target_feature(TARGET_FEATURE)
    .set_label_vocab(LABEL_VOCAB)
)

# run the WIT-widget
WitWidget(config_builder, height=800)<jupyter_output><empty_output><jupyter_text>Understanding the What-If toolIn the **Datapoint editor** tab, you can highlight a dot in the result set and ask the What-If Tool to pick the "nearest counterfactual". This is a row of data closest to the row of data you selected but with the opposite outcome. Features in the left-hand table are editable and can show what tweaks are needed to get a particular row of data to flip from one outcome to another. For example, altering the *discount_percentage* feature would show how it impacts the prediction. Under the **Performance & Fairness** tab, you can slice the prediction results by a second variable. This allows digging deeper and understanding how different segments of the data react to the model's predictions.  For example, in the following image, the higher the *discount_percentage*, the lesser the false negatives and the lower the *discount_percentage*, the higher the false positives. The **Features** tab in the end provides you an intuitive and interactive way to understand the features present in the data. Similar to the exploratory data analysis steps performed in this notebook, the What-If Tool provides a visual and statistical description on the features.  Clean upTo clean up all Google Cloud resources used in this project, you can delete the Google Cloud project you used for the tutorial.Otherwise, you can delete the individual resources you created in this tutorial:Undeploy the model.<jupyter_code>endpoint.undeploy(deployed_model_id=DEPLOYED_MODEL_ID)<jupyter_output><empty_output><jupyter_text>Delete the endpoint.<jupyter_code>endpoint.delete()<jupyter_output><empty_output><jupyter_text>Delete the model.<jupyter_code>model.delete()<jupyter_output><empty_output><jupyter_text>Remove the contents of the Cloud Storage bucket.<jupyter_code>! gsutil -m rm -r $BUCKET_URI<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08747473045822103, 'fraction_numerical': 0.004632749326145553, 'mean_word_length': 6.033471563981043, 'pattern_counts': {'":': 1, '<': 194, '<?xml version=': 0, '>': 195, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '48', 'dir': 'jupyter-structured-clean-dedup', 'id': '480985', 'license': 'Apache-2.0', 'n_tokens_mistral': 7199, 'n_tokens_neox': 6982, 'n_words': 2420, 'path': 'notebooks/community/managed_notebooks/inventory-prediction/inventory_prediction.ipynb', 'repo_name': 'nayaknishant/vertex-ai-samples', 'size': '23744'}