<jupyter_start><jupyter_text>This notebook demonstrates the usage of the `convert_to_onnx` helper function for converting a model from any supported framework into ONNX for APS inference. The function simply wraps the individual conversion functions from [ONNXMLTools](https://github.com/onnx/onnxmltools). This notebook provides examples for conversion from PyTorch, Sklearn, Keras, Tensorflow, and LightGBM.<jupyter_code>import numpy as np
import pandas as pd
import onnxruntime as rt
import pickle

from IVaps import convert_to_onnx
from IVaps import estimate_aps_onnx<jupyter_output><empty_output><jupyter_text>In general all models can be converted with the same basic function call, as demonstrated by the minimum examples below. All that is typically needed for conversion is a model, the framework string, and dummy input data. Aside from PyTorch, conversion from all other frameworks will return the converted model, which can then be sent directly into APS estimation or an InferenceSession.  Basic examples: Sklearn, Keras, LightGBM Note on the dummy input data: the data can be in any format or shape as long as it is exactly how you would pass it into a model for inference. `convert_to_onnx` allows for models that take continuous and discrete inputs as separate arrays -- please refer to the Pytorch section for an in-depth example.<jupyter_code># Minimal example with Sklearn iris 
data = pd.read_csv(f"data/iris_data.csv")
data = data.iloc[:,1:]
model = pickle.load(open(f"models/iris_logreg.pickle", 'rb'))
onnx = convert_to_onnx(model, "sklearn", data)
# We can save the ONNX model to file as well
onnx = convert_to_onnx(model, "sklearn", data, path = "../tests/test_models/iris_test.onnx") 

# We can estimate APS/run inference directly with the converted model
aps_skl = estimate_aps_onnx(onnx.SerializeToString(), X_c = data)
print(aps_skl[:5])
# We can also specify a label for the ONNX input node
convert_to_onnx(model, "sklearn", data, path = "../tests/test_models/iris_test_named_node.onnx", input_names = ('new_name',))

sess1 = rt.InferenceSession("../tests/test_models/iris_test.onnx")
print("Default input name:", sess1.get_inputs()[0].name)

sess2 = rt.InferenceSession("../tests/test_models/iris_test_named_node.onnx")
print("Manual input name:", sess2.get_inputs()[0].name)
# Minimal example with Keras iris
import keras

model = keras.models.load_model(f"models/keras_example")
print(model.summary())
onnx = convert_to_onnx(model, "keras", data)
print(onnx)
# Minimal example with LightGBM
import lightgbm as lgb

data = pd.read_csv(f"data/lgbm_regression.test", header=None, sep='\t')
data = data.drop(0, axis = 1) # First column is the test Y variable
model = lgb.Booster(model_file= f"models/lgbm_example.txt")
onnx = convert_to_onnx(model, "lightgbm", data)

# APS estimation -- note that the return array is 2D for LightGBM 
estimate_aps_onnx(onnx.SerializeToString(), X_c = data[0:5])<jupyter_output>/Users/rl874/opt/anaconda3/envs/IVaps/lib/python3.7/site-packages/lightgbm/__init__.py:48: UserWarning: Starting from version 2.2.1, the library file in distribution wheels for macOS is built by the Apple Clang (Xcode_8.3.3) compiler.
This means that in case of installing LightGBM from PyPI via the ``pip install lightgbm`` command, you don't need to install the gcc compiler anymore.
Instead of that, you need to install the OpenMP library, which is required for running LightGBM on the system with the Apple Clang compiler.
You can install the OpenMP library by the following command: ``brew install libomp``.
  "You can install the OpenMP library by the following command: ``brew install libomp``.", UserWarning)
The maximum opset needed by this model is only 1.<jupyter_text>LightGBM models will return predictions in a 2D array when converted to ONNX. Otherwise the outputs will be the same as inference with the original model.<jupyter_code>sess = rt.InferenceSession(onnx.SerializeToString())
label_name = sess.get_outputs()[0].name
input_name = sess.get_inputs()[0].name

onnx_out = sess.run([label_name], {input_name: np.array(data)})[0]
print(onnx_out.shape)

og_out = model.predict(data)
print(og_out.shape)

np.testing.assert_array_almost_equal(onnx_out.flatten(), og_out, decimal = 5)<jupyter_output>(500, 1)
(500,)<jupyter_text>Tensorflow (In progress...)  PyTorch<jupyter_code>import torch
import torch.nn as nn<jupyter_output><empty_output><jupyter_text>We will read in a pretrained model with the below framework.<jupyter_code>class CatModel(nn.Module):

    def __init__(self, embedding_size, num_numerical_cols, output_size, layers, p=0.4):
        super().__init__()
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
        self.embedding_dropout = nn.Dropout(p)
        self.batch_norm_num = nn.BatchNorm1d(num_numerical_cols)

        all_layers = []
        num_categorical_cols = sum((nf for ni, nf in embedding_size))
        input_size = num_categorical_cols + num_numerical_cols

        for i in layers:
            all_layers.append(nn.Linear(input_size, i))
            all_layers.append(nn.ReLU(inplace=True))
            all_layers.append(nn.BatchNorm1d(i))
            all_layers.append(nn.Dropout(p))
            input_size = i

        all_layers.append(nn.Linear(layers[-1], output_size))

        self.layers = nn.Sequential(*all_layers)

        self.m = nn.Softmax(dim=1)

    def forward(self, x_categorical, x_numerical):
        embeddings = []
        for i,e in enumerate(self.all_embeddings):
            embeddings.append(e(x_categorical[:,i]))
        x = torch.cat(embeddings, 1)
        x = self.embedding_dropout(x)

        x_numerical = self.batch_norm_num(x_numerical)
        x = torch.cat([x, x_numerical], 1)
        x = self.layers(x)
        x = self.m(x)

        return x[:,1]
model = CatModel([(3, 2), (2, 1), (2, 1), (2, 1)], 6, 2, [200,100,50], p=0.4)
model.load_state_dict(torch.load(f"models/churn_categorical.pt"))
model.eval()<jupyter_output><empty_output><jupyter_text>You will notice that this model takes two separate inputs, `x_categorical` and `x_numerical` for discrete and continuous variables, respectively. Our conversion function is well equipped to handle such cases.<jupyter_code># Read in input data 
data = pd.read_csv("data/churn_data.csv")

# Split discrete and continuous data
categorical_cols = ['Geography', 'Gender', 'HasCrCard', 'IsActiveMember']
numerical_cols = ['CreditScore', 'Age', 'Tenure', 'Balance', 'NumOfProducts', 'EstimatedSalary']

for category in categorical_cols:
    data[category] = data[category].astype('category')
cat = [data[c].cat.codes.values for c in categorical_cols]

cat_data = np.stack(cat, 1)
num_data = np.array(data[numerical_cols])

cat_tensor = torch.tensor(cat_data).long()
num_tensor = torch.tensor(num_data).float()<jupyter_output><empty_output><jupyter_text>PyTorch conversion is slightly different from the other frameworks in that it makes use of a package-specific `export` function. This function will not return the converted onnx model and requires a file path to save to.<jupyter_code># Without a file path, the conversion will throw an error
try:
    convert_to_onnx(model, "pytorch", cat_data, num_data)
except ValueError as e:
    print(e)<jupyter_output>For PyTorch to ONNX conversion a file path must be given.<jupyter_text>`convert_to_onnx` allows for models with up to two separate inputs. Keep in mind that the order that these inputs are passed should be the same as their order in the model's `forward` function. We can also pass input names that can be referenced during ONNX inferencing, but these are not required.Models are typically sensitive to data types, so it is important to cast the input data to the expected type before passing them into conversion.<jupyter_code># Dummy data can be in any format, as long as it is convertible to a tensor and multi-dimensional
# The below calls will all result in the same outcome
ret = convert_to_onnx(model, "pytorch", dummy_input1 = cat_data.astype(np.int64), dummy_input2 = num_data.astype(np.float32), 
                      path = "../tests/test_models/churn_cat_test.onnx", input_names=("d_inputs", "c_inputs"))
ret2 = convert_to_onnx(model, "pytorch", dummy_input1 = cat_data[0,np.newaxis].astype(np.int64), dummy_input2 = num_data[0,np.newaxis].astype(np.float32), 
                       path = "../tests/test_models/churn_cat_test.onnx", input_names=("d_inputs", "c_inputs"))
ret3 = convert_to_onnx(model, "pytorch", dummy_input1 = cat_tensor, dummy_input2 = num_tensor, 
                       path = "../tests/test_models/churn_cat_test.onnx", input_names=("d_inputs", "c_inputs"))

print(ret, ret2, ret3)<jupyter_output>True True True<jupyter_text>We can also set the name(s) of the output node.<jupyter_code>convert_to_onnx(model, "pytorch", dummy_input1 = cat_tensor, dummy_input2 = num_tensor, output_names = ["cat_out"], 
                       path = "../tests/test_models/churn_cat_test.onnx", input_names=("d_inputs", "c_inputs"))

sess = rt.InferenceSession("../tests/test_models/churn_cat_test.onnx")
sess.get_outputs()[0].name<jupyter_output><empty_output><jupyter_text>We compare inference between ONNX and the original model to validate successful conversion<jupyter_code>with torch.no_grad():
    torch_preds = model(cat_tensor, num_tensor).numpy()
sess = rt.InferenceSession("../tests/test_models/churn_cat_test.onnx")
output_name = sess.get_outputs()[0].name
onnx_preds = sess.run([output_name], {"c_inputs": num_data.astype(np.float32),
                                      "d_inputs": cat_data.astype(np.int64)})[0]

np.testing.assert_array_almost_equal(torch_preds, onnx_preds, decimal=5)<jupyter_output><empty_output><jupyter_text>Now we can run APS inference<jupyter_code>estimate_aps_onnx("../tests/test_models/churn_cat_test.onnx", X_c = num_data.astype(np.float32), X_d = cat_data.astype(np.int64), 
                  input_type = 2)
num_data.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10476285771462877, 'fraction_numerical': 0.010506303782269362, 'mean_word_length': 4.855301698886936, 'pattern_counts': {'":': 2, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '194410', 'license': 'Apache-2.0', 'n_tokens_mistral': 3304, 'n_tokens_neox': 3216, 'n_words': 924, 'path': 'examples/ONNX Conversion Tutorial.ipynb', 'repo_name': 'factoryofthesun/MLisNE', 'size': '9994'}