<jupyter_start><jupyter_text>Adult income dataset Here we wll study the Adult Income Dataset and see how we can apply Homomorphic Random Forest to it.  Data loading<jupyter_code>import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv(
    "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",
    header=None)

df.columns = [
    "Age", "WorkClass", "fnlwgt", "Education", "EducationNum",
    "MaritalStatus", "Occupation", "Relationship", "Race", "Gender",
    "CapitalGain", "CapitalLoss", "HoursPerWeek", "NativeCountry", "Income"
]

# df = df.sample(frac=0.1, random_state=1)
train_cols = df.columns[0:-1]
label = df.columns[-1]
X = df[train_cols]
y = df[label].apply(lambda x: 0 if x == " <=50K" else 1) #Turning response into 0 and 1

seed = 1
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=seed)<jupyter_output><empty_output><jupyter_text>Data preprocessing Because we need our data to be in the range$[-1,1]$ at the beginning, we will create a preprocessing pipeline using sklearn Pipelines. It will apply a LabelEncoder on categorical columns, before using it in a MinMaxScaler.<jupyter_code>from cryptotree.preprocessing import Featurizer<jupyter_output><empty_output><jupyter_text>Now that we have defined the featurizer, we can use it on our dataset.<jupyter_code>categorical_columns = ["WorkClass","Education","MaritalStatus", "Occupation", "Relationship", 
                       "Race", "Gender", "NativeCountry"]

pipe = Featurizer(categorical_columns)
X_train_normalized = pipe.fit_transform(X_train)<jupyter_output><empty_output><jupyter_text>We can have a look and check that input is in $[-1,1]$.<jupyter_code>X_train_normalized[:2]<jupyter_output><empty_output><jupyter_text>Neural Random Forest<jupyter_code>import torch
import numpy as np<jupyter_output><empty_output><jupyter_text>Now that data has been preprocessed, we can feed it to a Neural Random Forest. Here we can have a look at the activation function we will use, for instance sigmoid.<jupyter_code>from cryptotree.tree import NeuralRandomForest, SigmoidTreeMaker, TanhTreeMaker
from cryptotree.polynomials import plot_graph_function_approximation

max_depth = 4

dilatation_factor = 16
polynomial_degree = dilatation_factor

plot_graph_function_approximation(torch.sigmoid,
                                  dilatation_factor=dilatation_factor,polynomial_degree=polynomial_degree)<jupyter_output><empty_output><jupyter_text>Now that we have te parameters of dilation and polynomial degree for the activation functions, we can create the tree makers.<jupyter_code>sigmoid_tree_maker = SigmoidTreeMaker(use_polynomial=True,
                                  dilatation_factor=dilatation_factor, polynomial_degree=polynomial_degree)

tanh_tree_maker = TanhTreeMaker(use_polynomial=True,
                                  dilatation_factor=dilatation_factor, polynomial_degree=polynomial_degree)<jupyter_output><empty_output><jupyter_text>We can fit a regular sklearn Random Forest now.<jupyter_code>from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(max_depth=max_depth, random_state=0)
rf.fit(X_train_normalized, y_train)<jupyter_output><empty_output><jupyter_text>From the regular RF, we can define the Neural Random Forest, with sigmoid and tanh activations.  Without finetuning<jupyter_code>estimators = rf.estimators_
sigmoid_neural_rf = NeuralRandomForest(estimators, sigmoid_tree_maker)
tanh_neural_rf = NeuralRandomForest(estimators, tanh_tree_maker)
with torch.no_grad():
    sigmoid_neural_pred = sigmoid_neural_rf(torch.tensor(X_train_normalized).float()).argmax(dim=1).numpy()
    tanh_neural_pred = tanh_neural_rf(torch.tensor(X_train_normalized).float()).argmax(dim=1).numpy()

pred = rf.predict(X_train_normalized)
print(f"Original accuracy : {(pred == y_train).mean()}")

print(f"Accuracy of sigmoid  : {(sigmoid_neural_pred == y_train).mean()}")
print(f"Accuracy of tanh : {(tanh_neural_pred == y_train).mean()}")

print(f"Match between sigmoid and original : {(sigmoid_neural_pred == pred).mean()}")
print(f"Match between tanh and original : {(tanh_neural_pred == pred).mean()}")<jupyter_output>Original accuracy : 0.8373003685503686
Accuracy of sigmoid  : 0.24393427518427518
Accuracy of tanh : 0.7560657248157249
Match between sigmoid and original : 0.10565110565110565
Match between tanh and original : 0.8943488943488943<jupyter_text>We can see that there is a discrepancy between the original Random Forest and the neural ones, especially for the sigmoid one.Nonetheless we can correct this error by finetuning the final layer of our neural networks to take into account the fact that due to using activation functions not sharp enough.  With finetuning Because we need to make sure that at each step of the Neural Network the output is in $[-1,1]$, to be able to use the polynomial approximation of the activation, we can only safely train the last layer. To do so we will use Pytorch and [fastai](https://github.com/fastai/fastai). Here we first define our Pytorch Dataset.<jupyter_code>from torch.utils import data
import numpy as np

class TabularDataset(data.Dataset):
    'Characterizes a dataset for PyTorch'
    def __init__(self, X: np.ndarray, y: np.ndarray):
        'Initialization'
        self.X, self.y = X,y

    def __len__(self):
        'Denotes the total number of samples'
        return len(self.X)

    def __getitem__(self, index):
        'Generates one sample of data'

        # Load data and get label
        X = torch.tensor(self.X[index]).float()
        y = torch.tensor(self.y[index])

        return X, y<jupyter_output><empty_output><jupyter_text>Then we split our training data into training and validation.<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=seed)
X_train_normalized, X_valid_normalized, y_train, y_valid = train_test_split(pipe.transform(X_train), 
                                                                            y_train,
                                                                            train_size=0.8)<jupyter_output><empty_output><jupyter_text>Then we create the Pytorch dataloaders.<jupyter_code>train_ds = TabularDataset(X_train_normalized, y_train.values)
valid_ds = TabularDataset(X_valid_normalized, y_valid.values)
bs = 128

train_dl = data.DataLoader(train_ds, batch_size=bs, shuffle=True)
valid_dl = data.DataLoader(valid_ds, batch_size=bs)
fix_dl = data.DataLoader(train_ds, batch_size=bs, shuffle=False)<jupyter_output><empty_output><jupyter_text>Here we will just define the model, which is a sigmoid Neural Random Forest.<jupyter_code>tree_maker = tanh_tree_maker

model = NeuralRandomForest(rf.estimators_, tree_maker=tree_maker)<jupyter_output><empty_output><jupyter_text>Because we only want to train the last layer, we will freeze the first two layers and check they are frozen.<jupyter_code>model.freeze_layer("comparator")
model.freeze_layer("matcher")

for p in model.parameters():
    print(p.shape, p.requires_grad)
pred = rf.predict(X_train_normalized)

with torch.no_grad():
    neural_pred = model(torch.tensor(X_train_normalized).float()).argmax(dim=1).numpy()

print(f"Original accuracy : {(pred == y_train).mean()}")
print(f"Accuracy : {(neural_pred == y_train).mean()}")
print(f"Same output : {(neural_pred == pred).mean()}")<jupyter_output>Original accuracy : 0.8379882906229005
Accuracy : 0.7569824359343507
Same output : 0.8951914771091276<jupyter_text>Now we can define our fastai Learner, with the dataset, the model, and the loss function, which is a Label Smoothing Cross Entropy here.<jupyter_code>from fastai.basic_data import DataBunch
from fastai.tabular.learner import Learner
from fastai.metrics import accuracy

from cryptotree.tree import CrossEntropyLabelSmoothing
import torch.nn as nn

data = DataBunch(train_dl, valid_dl,fix_dl=fix_dl)

criterion = CrossEntropyLabelSmoothing()

learn = Learner(data, model, loss_func=criterion, metrics=accuracy)<jupyter_output><empty_output><jupyter_text>We will use fastai lr finder to have an idea of what learning rate to choose.<jupyter_code>learn.lr_find(num_it=500)
learn.recorder.plot()<jupyter_output><empty_output><jupyter_text>Here we can see that a good learning rate should be around 1e-1.We can now fine tune our model.<jupyter_code>learn.fit_one_cycle(5,1e-1 / 2)<jupyter_output><empty_output><jupyter_text>Here we can have a look at the performance of the Neural Random Forest tuned with respect to the original sklearn Random Forest.<jupyter_code>pred = rf.predict(X_valid_normalized)

with torch.no_grad():
    neural_pred = model(torch.tensor(X_valid_normalized).float()).argmax(dim=1).numpy()

print(f"Original accuracy : {(pred == y_valid).mean()}")
print(f"Accuracy : {(neural_pred == y_valid).mean()}")
print(f"Same output : {(neural_pred == pred).mean()}")<jupyter_output>Original accuracy : 0.8345489443378119
Accuracy : 0.8456813819577735
Same output : 0.9274472168905951<jupyter_text>Homomorphic Random Forest Now that we have seen how a Neural Random Forest can be obtained and fine tuned from a regular sklearn Random Forest, we can now see how its Homomorphic Random Forest counterpart performs. To do so, we must first initialize the parameters of the CKKS SEAL library, and choose the polynomial activation function we want. Here we will use the same activation function as the above Neural Random Forest.<jupyter_code>import numpy as np 

dilatation_factor = 16
degree = dilatation_factor

PRECISION_BITS = 28
UPPER_BITS = 9

polynomial_multiplications = int(np.ceil(np.log2(degree))) + 1
n_polynomials = 2
matrix_multiplications = 3

depth = matrix_multiplications + polynomial_multiplications * n_polynomials

poly_modulus_degree = 16384

moduli = [PRECISION_BITS + UPPER_BITS] + (depth) * [PRECISION_BITS] + [PRECISION_BITS + UPPER_BITS]
print(moduli)
print(sum(moduli))<jupyter_output>[37, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 37]
438<jupyter_text>Now we just need to use the parameters defined above to populate our environment with SEAL context.<jupyter_code>from cryptotree.seal_helper import create_seal_globals, append_globals_to_builtins
import builtins

create_seal_globals(globals(), poly_modulus_degree, moduli, PRECISION_BITS, use_symmetric_key=False)
append_globals_to_builtins(globals(), builtins)<jupyter_output><empty_output><jupyter_text>We can then create our Homomorphic Random Forest from the Neural Random Forest we fine tuned earlier.We first have to extract the weights using the Homomorphic Random Forest class, and pass it to the Homomorphic Tree Evaluator which will do the computation, using the polynomial activation given, and the SEAL context.A featurizer is also created for the client side, in order to preprocess, encode and encrypt the data.<jupyter_code>from cryptotree.cryptotree import HomomorphicNeuralRandomForest, HomomorphicTreeEvaluator, HomomorphicTreeFeaturizer
from cryptotree.polynomials import polyeval_tree

h_rf = HomomorphicNeuralRandomForest(model)

tree_evaluator = HomomorphicTreeEvaluator.from_model(h_rf, tree_maker.coeffs, 
                                                   polyeval_tree, evaluator, encoder, relin_keys, galois_keys, 
                                                   scale)

homomorphic_featurizer = HomomorphicTreeFeaturizer(h_rf.return_comparator(), encoder, encryptor, scale)<jupyter_output><empty_output><jupyter_text>Now we can take some data, encrypt it, and pass it to our Homorphic Tree Evaluator :<jupyter_code>i = 0

x = X_train_normalized[i]
ctx = homomorphic_featurizer.encrypt(x)

outputs = tree_evaluator(ctx)<jupyter_output><empty_output><jupyter_text>We can decrypt and decode the data to see the output :<jupyter_code>import tenseal.sealapi as seal

ptx = seal.Plaintext()
decryptor.decrypt(outputs, ptx)

homomorphic_pred = encoder.decode_double(ptx)[:2]<jupyter_output><empty_output><jupyter_text>We can see what was the original output, if we had used only the Neural Random Forest :<jupyter_code>x = X_train_normalized[i]

pred = rf.predict_proba(x.reshape(1,-1))
neural_pred = model(torch.tensor(x).float().unsqueeze(0))

print(f"Original Random Forest output : {pred}")
print(f"Neural Random Forest output : {neural_pred.detach()}")
print(f"Homomorphic Random Forest output : {homomorphic_pred}")<jupyter_output>Original Random Forest output : [[0.830971 0.169029]]
Neural Random Forest output : tensor([[ 1.0418, -0.8052]])
Homomorphic Random Forest output : [1.0742285425491416, -0.8491698612169115]<jupyter_text>Here we can see our Neural Random Forest, and Homomorphic Random Forest have very similar outputs.  Comparison between models We will now compare a linear model, a RF, a NRF and a HRF. Because computation are done one at a time with HRF, we will use Fastai `parallel` to speedup inference taking advantage of the multiple CPUs.<jupyter_code>from fastai.core import parallel
import multiprocessing
import tenseal.sealapi as seal

def predict(x, index):
    """Performs HRF prediction"""
    
    # We first encrypt and evaluate our model on it
    ctx = homomorphic_featurizer.encrypt(x)
    outputs = tree_evaluator(ctx)
    
    # We then decrypt it and get the first 2 values which are the classes scores
    ptx = seal.Plaintext()
    decryptor.decrypt(outputs, ptx)
    
    homomorphic_pred = encoder.decode_double(ptx)[:2]
    homomorphic_pred = np.argmax(homomorphic_pred)
    
    return index, homomorphic_pred

# We get the number of cores
cores = multiprocessing.cpu_count()

# We compute the outputs
hrf_pred = parallel(predict, X_valid_normalized, max_workers=cores)

# Because the outputs are unordered we must first sort by index then take the predictions
hrf_pred = np.array(sorted(hrf_pred, key = lambda x:x[0]))[:,1]<jupyter_output><empty_output><jupyter_text>We compute the predictions of the NRF.<jupyter_code>outputs = []

for x,y in valid_dl:
    with torch.no_grad():
        pred = model(x)
    outputs.append(pred)
    
nrf_pred = torch.cat(outputs).argmax(dim=1).numpy()<jupyter_output><empty_output><jupyter_text>Finally we compute the predictions of logistic regression and RF.<jupyter_code>from sklearn.linear_model import LogisticRegression

linear = LogisticRegression()
linear.fit(X_train_normalized, y_train)

# We compute the linear preds
linear_pred = linear.predict(X_valid_normalized)

# We compute the random forest predictions
rf_pred = rf.predict(X_valid_normalized)<jupyter_output><empty_output><jupyter_text>We can compute all metrics now.<jupyter_code>from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

def compute_metrics(pred, y):
    """Computes all the metrics between predictions and real values"""
    accuracy = accuracy_score(pred,y)
    precision = precision_score(pred,y)
    recall = recall_score(pred,y)
    f1 = f1_score(pred, y)
    return dict(accuracy=accuracy, precision=precision, recall=recall, f1=f1)

models = dict(nrf=nrf_pred, hrf=hrf_pred, rf=rf_pred, linear=linear_pred)

outputs = []
for name, pred in models.items():
    metrics = compute_metrics(pred, y_valid)
    metrics["model"] = name
    outputs.append(metrics)
outputs = pd.DataFrame(outputs)
outputs
outputs.to_csv("results.csv", index = None)<jupyter_output><empty_output><jupyter_text>Finally we can have a look at how similar NRF and HRF predictions are.<jupyter_code>(nrf_pred == hrf_pred).mean()<jupyter_output><empty_output><jupyter_text>Exporting featurizer and models We will now see how we can export the model and the featurizer for the server and the client. The preprocessing pipeline can be saved for later use on the client side. We can save and load it with pickle.<jupyter_code>import pickle

pickle.dump(pipe, open("pipe.pkl", "wb"))
pipe = pickle.load(open("pipe.pkl" , "rb"))<jupyter_output><empty_output><jupyter_text>We can also store as well the featurizer, used to shuffle the features before the comparisons. This will be sent to the client side to prepare the data before sending it.<jupyter_code>homomorphic_featurizer.save("homomorphic_featurizer.pkl")<jupyter_output><empty_output><jupyter_text>Finally, we can store the Homomorphic Random Forest to use on the server side.<jupyter_code>pickle.dump(h_rf, open("h_rf.pkl", "wb"))<jupyter_output><empty_output><jupyter_text>For the demo built using Streamlit, we need to know what were the values of the categorical features, and also one example of values.<jupyter_code>for col in X_train.columns.values:
    if col in categorical_columns:
        X_train[col] = X_train[col].astype("category")
    else:
        X_train[col] = X_train[col].astype(np.float32)
example = X_train.iloc[[0]]
example.to_csv("example.csv", index=None)
dtypes = X_train.dtypes
pickle.dump(dtypes, open("dtypes.pkl", "wb"))
dtypes = pickle.load(open("dtypes.pkl", "rb"))
dtypes<jupyter_output><empty_output><jupyter_text>Using exported models for inference<jupyter_code>import pickle
from cryptotree.cryptotree import HomomorphicTreeFeaturizer, HomomorphicTreeEvaluator
import numpy as np 

dilatation_factor = 16
degree = dilatation_factor

PRECISION_BITS = 28
UPPER_BITS = 9

polynomial_multiplications = int(np.ceil(np.log2(degree))) + 1
n_polynomials = 2
matrix_multiplications = 3

depth = matrix_multiplications + polynomial_multiplications * n_polynomials

poly_modulus_degree = 16384

moduli = [PRECISION_BITS + UPPER_BITS] + (depth) * [PRECISION_BITS] + [PRECISION_BITS + UPPER_BITS]
print(moduli)
print(sum(moduli))
from cryptotree.seal_helper import create_seal_globals, append_globals_to_builtins
import builtins

create_seal_globals(globals(), poly_modulus_degree, moduli, PRECISION_BITS, use_symmetric_key=False)
append_globals_to_builtins(globals(), builtins)
pipe = pickle.load(open("pipe.pkl","rb"))
homomorphic_featurizer = HomomorphicTreeFeaturizer.load("homomorphic_featurizer.pkl",
                                                        encoder, encryptor, scale)
from cryptotree.tree import SigmoidTreeMaker
from cryptotree.polynomials import polyeval_tree

dilatation_factor = 16
polynomial_degree = dilatation_factor

sigmoid_tree_maker = SigmoidTreeMaker(use_polynomial=True,
                                  dilatation_factor=dilatation_factor, polynomial_degree=polynomial_degree)

h_rf = pickle.load(open("h_rf.pkl","rb"))

tree_evaluator = HomomorphicTreeEvaluator.from_model(h_rf, sigmoid_tree_maker.coeffs, 
                                                   polyeval_tree, evaluator, encoder, relin_keys, galois_keys, 
                                                   scale)
import pandas as pd

x = pd.read_csv("example.csv")
x = pipe.transform(x).reshape(-1)
ctx = homomorphic_featurizer.encrypt(x)
outputs = tree_evaluator(ctx)
import tenseal.sealapi as seal

ptx = seal.Plaintext()
decryptor.decrypt(outputs, ptx)

homomorphic_pred = encoder.decode_double(ptx)[:2]

print(f"Homomorphic Random Forest output : {homomorphic_pred}")
import tenseal.sealapi as seal

ptx = seal.Plaintext()
decryptor.decrypt(outputs, ptx)

homomorphic_pred = encoder.decode_double(ptx)[:2]

pred = rf.predict_proba(x.reshape(1,-1))
neural_pred = model(torch.tensor(x).float().unsqueeze(0))

print(f"Original Random Forest output : {pred}")
print(f"Neural Random Forest output : {neural_pred.detach()}")
print(f"Homomorphic Random Forest output : {homomorphic_pred}")
torch.save(model.state_dict(), "model.pkl")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0905088563530255, 'fraction_numerical': 0.01955564656496365, 'mean_word_length': 5.386073880353057, 'pattern_counts': {'":': 0, '<': 133, '<?xml version=': 0, '>': 132, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '525918', 'license': 'Apache-2.0', 'n_tokens_mistral': 6612, 'n_tokens_neox': 6272, 'n_words': 1647, 'path': 'examples/adult_income/adult_income.ipynb', 'repo_name': 'dhuynh95/cryptotree', 'size': '19534'}