<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f7086876-72ed-4ac8-aeba-f05dbc641798",
   "metadata": {},
   "source": [
    "# Training Large Recommender Models with NVIDIA HugeCTR and Vertex AI\n",
    "\n",
    "This notebook demonstrates how to use Vertex AI Training to operationalize training and hyperparameter tuning of large scale deep learning models developed with NVIDIA HugeCTR framework. The notebook compiles prescriptive guidance for the following tasks:\n",
    "\n",
    "- Building a custom Vertex training container derived from NVIDIA NGC Merlin Training image\n",
    "- Configuring, submitting and monitoring a Vertex custom training job\n",
    "- Configuring, submitting and monitoring a Vertex hyperparameter tuning job\n",
    "- Retrieving and analyzing results of a hyperparameter tuning job\n",
    "\n",
    "The deep learning model used in this sample is [DeepFM](https://arxiv.org/abs/1703.04247) - a Factorization-Machine based Neural Network for CTR Prediction. The HugeCTR implementation of this model used in this notebook has been configured for the [Criteo dataset](https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/). \n",
    "\n",
    "## 1. HugeCTR Overview\n",
    "\n",
    "[HugeCTR](https://github.com/NVIDIA-Merlin/HugeCTR) is NVIDIA's GPU-accelerated, highly scalable recommender framework. We highly encourage reviewing the [HugeCTR User Guide](https://github.com/NVIDIA-Merlin/HugeCTR/blob/master/docs/hugectr_user_guide.md) before proceeding with this notebook.\n",
    "\n",
    "NVIDIA HugeCTR facilitates highly scalable implementations of leading deep learning recommender models including Google's [Wide and Deep](https://arxiv.org/abs/1606.07792), Facebook's [DLRM](https://ai.facebook.com/blog/dlrm-an-advanced-open-source-deep-learning-recommendation-model/), and the [DeepFM](https://arxiv.org/abs/1703.04247) model used in this notebook.\n",
    "\n",
    "A unique feature of HugeCTR is support for model-parallel embedding tables. Applying model-parallelizm for embedding tables enables massive scalability. Industrial grade deep learning recommendation systems most often employ very large embedding tables. User and item embedding tables - cornerstones of any recommender - can easily exceed tens of millions of rows. Without model-parallelizm it would be impossible to fit embedding tables this large in the memory of a single device - especially when using large embeddng vectors. In HugeCTR, embedding tables can span device memory across multiple GPUs on a single node or even across GPUs in a large distributed cluster. \n",
    "\n",
    "HugeCTR supports multiple model-parallelizm configurations for embedding tables. Refer to [the HugeCTR API reference](https://github.com/NVIDIA-Merlin/HugeCTR/blob/master/docs/hugectr_layer_book.md#embedding-types-detail) for detailed descriptions. In the DeepFM implementation used in this notebook, we utilize the `LocalizedSlotSparseEmbeddingHash` embedding type. In this embedding type, an embedding table is segmented into multiple slots or feature fields. Each slot stores embeddings for a single categorical feature. A given slot is allocated to a single GPU and it does not span multiple GPUs. However; in a multi GPU environment different slots can be allocated to different GPUs. \n",
    "\n",
    "The following diagram demonstrates an example configuration on a single node with multiple GPU - a hardware topology used by Vertex jobs in this notebook.\n",
    "\n",
    "\n",
    "<img src=\"./images/deepfm.png\" alt=\"Model parallel embeddings\" style=\"width: 70%;\"/>\n",
    "\n",
    "\n",
    "The Criteo dataset has 24 categorical features so there are 24 slots in the embedding table. Cardinalities of categorical variables vary from tens of milions to low teens so the dimensions of slots vary accordingly. Each slot in the embedding table utilizes an embedding vector of the same size. Note that the distribution of slots across GPUs is handled by HugeCTR; you don't have to explicitly pin a slot to a GPU.\n",
    "\n",
    "Dense layers of the DeepFM models are replicated on all GPUs using a canonical data-parallel pattern. \n",
    "\n",
    "A choice of an optimizer is critical when training large deep learning recommender systems. Different optimizers may result in significantly different convergence rates impacting both time (cost) to train and a final model performance. Since large recommender systems are often retrained on frequent basis minimizing time to train is one of the key design objectives for a training workflow. In this notebook we use the Adam optimizer that has been proved to work well with many deep learning recommeder system architectures.\n",
    "\n",
    "You can find the code that implements the DeepFM model in the [src/training/hugectr/trainer/model.py](https://github.com/GoogleCloudPlatform/nvidia-merlin-on-vertex-ai/blob/main/src/training/hugectr/model.py) file. \n",
    "\n",
    "\n",
    "## 2. Model Training Overview\n",
    "\n",
    "The training workflow has been optimized for Vertex AI Training. \n",
    "- Google Cloud Storage (GCS) and Vertex Training GCS Fuse are used for accessing training and validation data\n",
    "- A single node, multiple GPUs worker pool is used for Vertex Training jobs.\n",
    "- Training code has been instrumented to support hyperparameter tuning using Vertex Training Hyperparameter Tuning Job. \n",
    "\n",
    "You can find the code that implements the training workflow in the [src/training/hugectr/trainer/task.py](https://github.com/GoogleCloudPlatform/nvidia-merlin-on-vertex-ai/blob/main/src/training/hugectr/task.py) file.\n",
    "\n",
    "### Training data access\n",
    "\n",
    "Large deep learning  recommender systems are trained on massive datasets often hundreds of terabytes in size. Maintaining high-throughput when streaming training data to GPU workers is of critical importance. HugeCTR features a highly efficient multi-threaded data reader that parallelizes data reading and model computations. The reader accesses training data through a file system interface. The reader cannot directly access object storage systems like Google Cloud Storage, which is a canonical storage system for large scale training and validation datasets in Vertex AI Training. To expose Google Cloud Storage through a file system interface, the notebook uses an integrated feature of Vertex AI  - Google Cloud Storage FUSE. Vertex AI GCS FUSE provides a high performance file system interface layer to GCS that is self-tuning and requires minimal configuration. The following diagram depicts the training data access configuration:\n",
    "\n",
    "<img src=\"images/gcsfuse.png\" alt=\"GCS Fuse\" style=\"width:70%\"/>\n",
    "\n",
    "\n",
    "### Vertex AI Training worker pool configuration\n",
    "\n",
    "HugeCTR supports both single node, multiple GPU configurations and multiple node, multiple GPU distributed cluster topologies. In this sample, we use a single node, multiple GPU configuration. Due to the computational complexity of modern deep learning recommender models we recommend using Vertex Training A2 series machines for large models implemented with HugeCTR. The A2 machines can be configured with up to 16 A100 GPUs, 96 vCPUs, and 1,360GBs RAM. Each A100 GPU has 40GB of device memory. These are powerful configurations that can handle complex models with large embeddings.\n",
    "\n",
    "In this sample we use the `a2-highgpu-4g` machine type. \n",
    "\n",
    "Both custom training and hyperparameter tuning Vertex AI jobs demonstrated in this notebook are configured to use a [custom training container](https://cloud.google.com/vertex-ai/docs/training/containers-overview). The container is a derivative of [NVIDIA NGC Merlin training container](https://ngc.nvidia.com/catalog/containers/nvidia:merlin:merlin-training). The definition of the container image is found in the [Dockefile.hugectr](https://github.com/GoogleCloudPlatform/nvidia-merlin-on-vertex-ai/blob/main/src/Dockerfile.hugectr) file.\n",
    "\n",
    "### HugeCTR hyperparameter tuning with Vertex AI\n",
    "\n",
    "The training module has been instrumented to support [hyperparameter tuning with Vertex AI](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview). The custom container includes the [cloudml-hypertune package](https://github.com/GoogleCloudPlatform/cloudml-hypertune), which is used to report the results of model evaluations to Vertex AI hypertuning service. The following diagram depicts the training flow implemeted by the training module.\n",
    "\n",
    "\n",
    "<img src=\"images/hugectrtrainer.png\" alt=\"Training regime\" style=\"height:15%; width:40%\"/>\n",
    "\n",
    "\n",
    "Note that as of HugeCTR v3.2 release, the `hugectr.inference.InferenceSession.evaluate` method used in the trainer module only supports the *AUC* evaluation metric.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df66aa44-bcc2-4f16-96f3-385673c40869",
   "metadata": {},
   "source": [
    "## 3. Executing Model Training on Vertex AI\n",
    "\n",
    "This notebook assumes that the Criteo dataset has been preprocessed using the preprocessing workflow detailed in the [01-dataset-preprocessing.ipynb](https://github.com/GoogleCloudPlatform/nvidia-merlin-on-vertex-ai/blob/main/03-model-inference-hugectr.ipynb) notebook and the resulting Parquet training and validation splits, and the processed data schema have been stored in Google Cloud Storage.\n",
    "\n",
    "As you walk through the notebook you will execute the following steps:\n",
    "- Configure notebook environment settings, including GCP project, compute region, and the GCS locations of training and validation data splits.\n",
    "- Build a custom Vertex training container based on NVIDIA NGC Merlin Training container\n",
    "- Configure and submit a Vertex custom training job\n",
    "- Configure and submit a Vertex hyperparameter training job\n",
    "- Retrieve the results of the hyperparameter tuning job"
   ]
  },
  {
   "cell_type": "markdown",<commit_msg>Fix Links<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f7086876-72ed-4ac8-aeba-f05dbc641798",
   "metadata": {},
   "source": [
    "# Training Large Recommender Models with NVIDIA HugeCTR and Vertex AI\n",
    "\n",
    "This notebook demonstrates how to use Vertex AI Training to operationalize training and hyperparameter tuning of large scale deep learning models developed with NVIDIA HugeCTR framework. The notebook compiles prescriptive guidance for the following tasks:\n",
    "\n",
    "- Building a custom Vertex training container derived from NVIDIA NGC Merlin Training image\n",
    "- Configuring, submitting and monitoring a Vertex custom training job\n",
    "- Configuring, submitting and monitoring a Vertex hyperparameter tuning job\n",
    "- Retrieving and analyzing results of a hyperparameter tuning job\n",
    "\n",
    "The deep learning model used in this sample is [DeepFM](https://arxiv.org/abs/1703.04247) - a Factorization-Machine based Neural Network for CTR Prediction. The HugeCTR implementation of this model used in this notebook has been configured for the [Criteo dataset](https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/). \n",
    "\n",
    "## 1. HugeCTR Overview\n",
    "\n",
    "[HugeCTR](https://github.com/NVIDIA-Merlin/HugeCTR) is NVIDIA's GPU-accelerated, highly scalable recommender framework. We highly encourage reviewing the [HugeCTR User Guide](https://github.com/NVIDIA-Merlin/HugeCTR/blob/master/docs/hugectr_user_guide.md) before proceeding with this notebook.\n",
    "\n",
    "NVIDIA HugeCTR facilitates highly scalable implementations of leading deep learning recommender models including Google's [Wide and Deep](https://arxiv.org/abs/1606.07792), Facebook's [DLRM](https://ai.facebook.com/blog/dlrm-an-advanced-open-source-deep-learning-recommendation-model/), and the [DeepFM](https://arxiv.org/abs/1703.04247) model used in this notebook.\n",
    "\n",
    "A unique feature of HugeCTR is support for model-parallel embedding tables. Applying model-parallelizm for embedding tables enables massive scalability. Industrial grade deep learning recommendation systems most often employ very large embedding tables. User and item embedding tables - cornerstones of any recommender - can easily exceed tens of millions of rows. Without model-parallelizm it would be impossible to fit embedding tables this large in the memory of a single device - especially when using large embeddng vectors. In HugeCTR, embedding tables can span device memory across multiple GPUs on a single node or even across GPUs in a large distributed cluster. \n",
    "\n",
    "HugeCTR supports multiple model-parallelizm configurations for embedding tables. Refer to [the HugeCTR API reference](https://github.com/NVIDIA-Merlin/HugeCTR/blob/master/docs/hugectr_layer_book.md#embedding-types-detail) for detailed descriptions. In the DeepFM implementation used in this notebook, we utilize the `LocalizedSlotSparseEmbeddingHash` embedding type. In this embedding type, an embedding table is segmented into multiple slots or feature fields. Each slot stores embeddings for a single categorical feature. A given slot is allocated to a single GPU and it does not span multiple GPUs. However; in a multi GPU environment different slots can be allocated to different GPUs. \n",
    "\n",
    "The following diagram demonstrates an example configuration on a single node with multiple GPU - a hardware topology used by Vertex jobs in this notebook.\n",
    "\n",
    "\n",
    "<img src=\"./images/deepfm.png\" alt=\"Model parallel embeddings\" style=\"width: 70%;\"/>\n",
    "\n",
    "\n",
    "The Criteo dataset has 24 categorical features so there are 24 slots in the embedding table. Cardinalities of categorical variables vary from tens of milions to low teens so the dimensions of slots vary accordingly. Each slot in the embedding table utilizes an embedding vector of the same size. Note that the distribution of slots across GPUs is handled by HugeCTR; you don't have to explicitly pin a slot to a GPU.\n",
    "\n",
    "Dense layers of the DeepFM models are replicated on all GPUs using a canonical data-parallel pattern. \n",
    "\n",
    "A choice of an optimizer is critical when training large deep learning recommender systems. Different optimizers may result in significantly different convergence rates impacting both time (cost) to train and a final model performance. Since large recommender systems are often retrained on frequent basis minimizing time to train is one of the key design objectives for a training workflow. In this notebook we use the Adam optimizer that has been proved to work well with many deep learning recommeder system architectures.\n",
    "\n",
    "You can find the code that implements the DeepFM model in the [src/training/hugectr/trainer/model.py](https://github.com/GoogleCloudPlatform/nvidia-merlin-on-vertex-ai/blob/main/src/training/hugectr/model.py) file. \n",
    "\n",
    "\n",
    "## 2. Model Training Overview\n",
    "\n",
    "The training workflow has been optimized for Vertex AI Training. \n",
    "- Google Cloud Storage (GCS) and Vertex Training GCS Fuse are used for accessing training and validation data\n",
    "- A single node, multiple GPUs worker pool is used for Vertex Training jobs.\n",
    "- Training code has been instrumented to support hyperparameter tuning using Vertex Training Hyperparameter Tuning Job. \n",
    "\n",
    "You can find the code that implements the training workflow in the [src/training/hugectr/trainer/task.py](https://github.com/GoogleCloudPlatform/nvidia-merlin-on-vertex-ai/blob/main/src/training/hugectr/task.py) file.\n",
    "\n",
    "### Training data access\n",
    "\n",
    "Large deep learning  recommender systems are trained on massive datasets often hundreds of terabytes in size. Maintaining high-throughput when streaming training data to GPU workers is of critical importance. HugeCTR features a highly efficient multi-threaded data reader that parallelizes data reading and model computations. The reader accesses training data through a file system interface. The reader cannot directly access object storage systems like Google Cloud Storage, which is a canonical storage system for large scale training and validation datasets in Vertex AI Training. To expose Google Cloud Storage through a file system interface, the notebook uses an integrated feature of Vertex AI  - Google Cloud Storage FUSE. Vertex AI GCS FUSE provides a high performance file system interface layer to GCS that is self-tuning and requires minimal configuration. The following diagram depicts the training data access configuration:\n",
    "\n",
    "<img src=\"images/gcsfuse.png\" alt=\"GCS Fuse\" style=\"width:70%\"/>\n",
    "\n",
    "\n",
    "### Vertex AI Training worker pool configuration\n",
    "\n",
    "HugeCTR supports both single node, multiple GPU configurations and multiple node, multiple GPU distributed cluster topologies. In this sample, we use a single node, multiple GPU configuration. Due to the computational complexity of modern deep learning recommender models we recommend using Vertex Training A2 series machines for large models implemented with HugeCTR. The A2 machines can be configured with up to 16 A100 GPUs, 96 vCPUs, and 1,360GBs RAM. Each A100 GPU has 40GB of device memory. These are powerful configurations that can handle complex models with large embeddings.\n",
    "\n",
    "In this sample we use the `a2-highgpu-4g` machine type. \n",
    "\n",
    "Both custom training and hyperparameter tuning Vertex AI jobs demonstrated in this notebook are configured to use a [custom training container](https://cloud.google.com/vertex-ai/docs/training/containers-overview). The container is a derivative of [NVIDIA NGC Merlin training container](https://ngc.nvidia.com/catalog/containers/nvidia:merlin:merlin-training). The definition of the container image is found in the [Dockefile.hugectr](https://github.com/GoogleCloudPlatform/nvidia-merlin-on-vertex-ai/blob/main/src/Dockerfile.hugectr) file.\n",
    "\n",
    "### HugeCTR hyperparameter tuning with Vertex AI\n",
    "\n",
    "The training module has been instrumented to support [hyperparameter tuning with Vertex AI](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview). The custom container includes the [cloudml-hypertune package](https://github.com/GoogleCloudPlatform/cloudml-hypertune), which is used to report the results of model evaluations to Vertex AI hypertuning service. The following diagram depicts the training flow implemeted by the training module.\n",
    "\n",
    "\n",
    "<img src=\"images/hugectrtrainer.png\" alt=\"Training regime\" style=\"height:15%; width:40%\"/>\n",
    "\n",
    "\n",
    "Note that as of HugeCTR v3.2 release, the `hugectr.inference.InferenceSession.evaluate` method used in the trainer module only supports the *AUC* evaluation metric.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df66aa44-<PASSWORD>",
   "metadata": {},
   "source": [
    "## 3. Executing Model Training on Vertex AI\n",
    "\n",
    "This notebook assumes that the Criteo dataset has been preprocessed using the preprocessing workflow detailed in the [01-dataset-preprocessing.ipynb](https://github.com/GoogleCloudPlatform/nvidia-merlin-on-vertex-ai/blob/main/01-dataset-preprocessing.ipynb) notebook and the resulting Parquet training and validation splits, and the processed data schema have been stored in Google Cloud Storage.\n",
    "\n",
    "As you walk through the notebook you will execute the following steps:\n",
    "- Configure notebook environment settings, including GCP project, compute region, and the GCS locations of training and validation data splits.\n",
    "- Build a custom Vertex training container based on NVIDIA NGC Merlin Training container\n",
    "- Configure and submit a Vertex custom training job\n",
    "- Configure and submit a Vertex hyperparameter training job\n",
    "- Retrieve the results of the hyperparameter tuning job"
   ]
  },
  {
   "cell_type": "markdown",
{'fraction_non_alphanumeric': 0.08692228183294778, 'fraction_numerical': 0.009942412757789044, 'mean_word_length': 4.133400707427994, 'pattern_counts': {'":': 20, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 32, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2738617', 'n_tokens_mistral': 5552, 'n_tokens_neox': 5272, 'n_words': 2376}