<jupyter_start><jupyter_text>Unified API: Train and evaluation This notebook presents the solution for training and evaluating both the __GANITE__ and __CMGP__ algorithms over the [Twins](https://bitbucket.org/mvdschaar/mlforhealthlabpub/src/master/data/twins/) dataset, using a unified API.For details about each algorithm, please refer to their dedicated notebooks: - [GANITE(Tensorflow) notebook](https://github.com/bcebere/ite-api/blob/main/notebooks/ganite_train_evaluation.ipynb). - [GANITE(PyTorch) notebook](https://github.com/bcebere/ite-api/blob/main/notebooks/ganite_pytorch_train_evaluation.ipynb). - [CMGP notebook](https://github.com/bcebere/ite-api/blob/main/notebooks/cmgp_train_evaluation.ipynb).  SetupFirst, make sure that all the depends are installed in the current environment.```pip install -r requirements.txtpip install .```Next, we import all the dependencies necessary for the task.<jupyter_code># Double check that we are using the correct interpreter.
import sys
print(sys.executable)

# Import depends
from ite.algs.model import Model # the unified API
import ite.datasets as ds

from matplotlib import pyplot as plt
from IPython.display import HTML, display
import tabulate<jupyter_output>/home/bcebere/anaconda3/envs/cambridge/bin/python
WARNING:tensorflow:From /home/bcebere/anaconda3/envs/cambridge/lib/python3.8/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term<jupyter_text>Load the DatasetThe example is done using the [Twins](https://bitbucket.org/mvdschaar/mlforhealthlabpub/src/master/data/twins/) dataset.Next, we load the dataset, process the data, and sample a training set and a test set.The logic is implemented [here](https://github.com/bcebere/ite-api/tree/main/src/ite/datasets), and it adapted from the original [GANITE pre-processing implementation](https://bitbucket.org/mvdschaar/mlforhealthlabpub/src/master/alg/ganite/data_preprocessing_ganite.py).For CGMP, we have to downsample to 1000 training items. For the rest, we load without downsampling.<jupyter_code>train_ratio = 0.8
 
full_dataloader = ds.load("twins", train_ratio)
cmgp_dataloader = ds.load("twins", train_ratio, downsample=1000)<jupyter_output><empty_output><jupyter_text>Load and train GANITE(Tensorflow version)The constructor requires the name of the chosen algorithm for the first parameter - `GANITE`.The constructor supports the same parameters as the "native" version: - `dim`: The number of features in X. - `dim_outcome`: The number of potential outcomes. - `dim_hidden`: hyperparameter for tuning the size of the hidden layer. - `depth`: hyperparameter for the number of hidden layers in the generator and inference blocks. - `num_iterations`: hyperparameter for the number of training epochs. - `alpha`: hyperparameter used for the Generator block loss. - `beta`: hyperparameter used for the ITE block loss. - `num_discr_iterations`: number of iterations executed by the discriminator.  The hyperparameters used in this experiment are from Table 7 in the paper.<jupyter_code>dim = len(full_dataloader[0][0])
dim_outcome = full_dataloader[-1].shape[1]
 
ganite_model = Model(
    "GANITE",
    dim, 
    dim_outcome, 
    dim_hidden=8, 
    num_iterations=10000, 
    alpha=2, 
    beta=2, 
    minibatch_size=128, 
    num_discr_iterations=3, 
    depth=5,
)

ganite_tf_metrics = ganite_model.train(*full_dataloader)
ganite_tf_metrics.print()

ganite_tf_metrics.plot(plt, thresholds = [0.2, 0.25, 0.3, 0.35])<jupyter_output>100%|██████████| 10000/10000 [01:00<00:00, 164.14it/s]
100%|██████████| 10000/10000 [00:13<00:00, 764.22it/s]<jupyter_text>Load and train GANITE(PyTorch version)The constructor requires the name of the chosen algorithm for the first parameter - `GANITE_TORCH`.The constructor supports the same parameters as the "native" version: - `dim`: The number of features in X. - `dim_outcome`: The number of potential outcomes. - `dim_hidden`: hyperparameter for tuning the size of the hidden layer. - `depth`: hyperparameter for the number of hidden layers in the generator and inference blocks. - `num_iterations`: hyperparameter for the number of training epochs. - `alpha`: hyperparameter used for the Generator block loss. - `beta`: hyperparameter used for the ITE block loss. - `num_discr_iterations`: number of iterations executed by the discriminator.  The hyperparameters used in this experiment are computed using the [hyperparameter tuning notebook](https://github.com/bcebere/ite-api/blob/main/notebooks/hyperparam_tuning.ipynb).<jupyter_code>ganite_torch_model = Model(
    "GANITE_TORCH",
    dim, 
    dim_outcome, 
    dim_hidden=30,
    num_iterations=3000,
    alpha=1,
    beta=10,
    minibatch_size=256,
    num_discr_iterations=6,
    depth=5,
)

ganite_torch_metrics = ganite_torch_model.train(*full_dataloader)
ganite_torch_metrics.print()

ganite_torch_metrics.plot(plt, thresholds = [0.2, 0.25, 0.3, 0.35])<jupyter_output>0%|          | 0/3000 [00:00<?, ?it/s] /home/bcebere/anaconda3/envs/cambridge/lib/python3.8/site-packages/torch/autograd/__init__.py:130: UserWarning:CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
100%|██████████| 3000/3000 [03:07<00:00, 16.02it/s]
100%|██████████| 3000/3000 [00:30<00:00, 99.90it/s]<jupyter_text>Load and train CMGPThe constructor requires the name of the chosen algorithm for the first parameter - `CMGP`.The constructor supports the following parameters: - `dim`: The number of features in X. - `dim_outcome`: The number of potential outcomes. - `max_gp_iterations`: Maximum number of GP iterations before stopping the training.<jupyter_code>cmgp_model = Model(
    "CMGP",
    dim=dim,
    dim_outcome=dim_outcome,
    max_gp_iterations=1000, # (optional) Maximum number of interations for the Gaussian Process
)

for experiment in range(5):
    cmgp_dataloader = ds.load("twins", train_ratio, downsample=1000)
    cmgp_metrics = cmgp_model.train(*cmgp_dataloader)

cmgp_metrics.print()

cmgp_metrics.plot(plt, thresholds = [0.2, 0.25, 0.3, 0.35])<jupyter_output>/home/bcebere/anaconda3/envs/cambridge/lib/python3.8/site-packages/GPy/kern/src/stationary.py:243: RuntimeWarning:invalid value encountered in true_divide
 /home/bcebere/anaconda3/envs/cambridge/lib/python3.8/site-packages/GPy/kern/src/stationary.py:166: RuntimeWarning:overflow encountered in true_divide
 /home/bcebere/anaconda3/envs/cambridge/lib/python3.8/site-packages/GPy/kern/src/stationary.py:138: RuntimeWarning:invalid value encountered in add
 /home/bcebere/anaconda3/envs/cambridge/lib/python3.8/site-packages/GPy/kern/src/stationary.py:137: RuntimeWarning:overflow encountered in square<jupyter_text>Evaluate the models on the test setWe evaluate each algorithm on their test sets, using the [PEHE](https://github.com/bcebere/ite-api/blob/main/src/ite/utils/numpy.pyL19) and [ATE](https://github.com/bcebere/ite-api/blob/main/src/ite/utils/numpy.pyL58) metrics.<jupyter_code>for scenario in ["in-sample", "out-sample"]:
    print(f"{scenario} metrics:")

    results = [
        [
            "GANITE",
            "{:0.3f} +/- {:0.3f}".format(
                *ganite_tf_metrics.mean_confidence_interval("sqrt_PEHE", f"ITE Block {scenario} metrics")
            ),
            "{:0.3f} +/- {:0.3f}".format(
                *ganite_tf_metrics.mean_confidence_interval("ATE", f"ITE Block {scenario} metrics")
            ),
            "{:0.3f} +/- {:0.3f}".format(
                *ganite_tf_metrics.mean_confidence_interval("MSE", f"ITE Block {scenario} metrics")
            ),
        ],
        [
            "GANITE_TORCH",
            "{:0.3f} +/- {:0.3f}".format(
                *ganite_torch_metrics.mean_confidence_interval("sqrt_PEHE", f"ITE Block {scenario} metrics")
            ),
            "{:0.3f} +/- {:0.3f}".format(
                *ganite_torch_metrics.mean_confidence_interval("ATE", f"ITE Block {scenario} metrics")
            ),
            "{:0.3f} +/- {:0.3f}".format(
                *ganite_torch_metrics.mean_confidence_interval("MSE", f"ITE Block {scenario} metrics")
            ),
        ],
        [
            "CMGP",
            "{:0.3f} +/- {:0.3f}".format(
                *cmgp_metrics.mean_confidence_interval("sqrt_PEHE", f"{scenario} metrics")
            ),
            "{:0.3f} +/- {:0.3f}".format(*cmgp_metrics.mean_confidence_interval("ATE", f"{scenario} metrics")),
            "{:0.3f} +/- {:0.3f}".format(*cmgp_metrics.mean_confidence_interval("MSE", f"{scenario} metrics")),
        ],
    ]

    display(
        HTML(tabulate.tabulate(results, headers=["Model", "sqrt_PEHE", "ATE", "MSE"], tablefmt="html"))
    )<jupyter_output>in-sample metrics:
{'fraction_non_alphanumeric': 0.12276413731807577, 'fraction_numerical': 0.030107765803799577, 'mean_word_length': 4.22460824143935, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 20, 'https://': 10, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '42122', 'license': 'Apache-2.0', 'n_tokens_mistral': 3184, 'n_tokens_neox': 2929, 'n_words': 693, 'path': 'notebooks/unified_api_train_evaluation.ipynb', 'repo_name': 'bcebere/ite-api', 'size': '9001'}