<jupyter_start><jupyter_text>Privacy-preserving machine learning with two agents Using homomorphic encryption over the titanic challengeAuthor:<NAME> - <EMAIL> ObjectivesThis notebook aims to explore the feasibility of private machine learning with homomorphic encryption.We got two agents, Bob and Alice. Bob has data, and wants to extract value from it. But Bob has no clue about data science and needs someone to help him to properly exploit his data. Bob contacts Alice, data scientist, to do so. Here, Bob's project is to create a model that predicts which passengers survived the Titanic shipwreck. He initially has a dataset which contains the details of a subset of the passengers on board (891 to be exact) and importantly, will reveal whether they survived or not.dataset which contains the details of a subset of the passengers on board (891 to be exact) and more importantly, will reveal whether they survived or not. As the Titanic was an european ship, one can ask if this data is under the GRPD legislation. In any case, Bob wants to keep this data private.Therefore the problematic is how Alice can do her job of datascientist without compromising the privacy of Bob's data.How train a model on this private data without seeing them ?Currently there are few possibilities, like differential privacy, multi-party computing, or homomorphic encryption.We choose to focus on homomorphic encryption. This kind of encryption allows one to make computation on encrypted data, without decrypting it, to obtain encrypted results.Here Bob will preprocess his data, then will encrypt it using the CKKS encryption scheme.Alice will receive the encrypted data, and train a model on it. Finally Alice will be able to use this model to make (encrypted) prediction for Bob.Once the model trained, Bob will send new (encrypted) data, (like test data, or unlabeled data),and got a encrypted prediction from Alice, that he will be able to decrypt.   This notebook shows the Alice's side  Setup All modules are mandatory. Make sure to import them all by running the cell below<jupyter_code>import logging
import os
import socket
import sys
import time

import numpy as np
import pickle as pk
import tenseal as ts

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, f1_score, recall_score, precision_score, accuracy_score
                           
os.chdir("/home/apignet/homomorphic-encryption/ckks_titanic/")
from src.features import build_features
from src.models import Alice_LR
from src.communication import Actors
%load_ext memory_profiler
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
1<jupyter_output><empty_output><jupyter_text>Definition of parametersThe data will be sent later by Bob, and will not be stored at all. We choose to not store the model for now. So the only path which one can be interested in is the log's one.  Paths<jupyter_code>WORKING_DIR = os.getcwd()

LOG_PATH = WORKING_DIR+"/reports/log/"
LOG_FILENAME = "Alice"<jupyter_output><empty_output><jupyter_text>Paths<jupyter_code>fileHandler = logging.FileHandler("{0}/{1}.log".format(LOG_PATH, LOG_FILENAME))
streamHandler = logging.StreamHandler(sys.stdout)
logging.basicConfig(format="%(asctime)s  [%(levelname)-8.8s]  %(message)s", 
                    datefmt='%m/%d/%Y %I:%M:%S %p', 
                    level = logging.INFO, 
                    handlers=[fileHandler, streamHandler]
                   )<jupyter_output><empty_output><jupyter_text>Training variablesHere we choose the training parameters. The epoch number, learning rate, are set for a classic gradient descent. The regularization rate is a multiplicative factor of the l2 norm of the weight in the loss. This regularization is really important, as we will see later. Every VERBOSE epoch, the loss will be computed and stored. Note that it is really time consumingEvery SAVE_WEIGHT epoch, the weights will be stored. This is useful for debug, and monitoring the encryption/decryption precision (which can be really bad if the safety parameters are not set properly by Bob).N_JOBS defines the number of processes which will be created and split on cores. A good setting is the number of processeurs.<jupyter_code>EPOCH = 40
LEARNING_RATE = 0.9
MOMENTUM_RATE = 0.6
REGULARIZATION_RATE = 0.5
VERBOSE = 4
SAVE_WEIGHT = 4
N_JOBS = 8<jupyter_output><empty_output><jupyter_text>Socket VariablesBob and Alice will communicate though the socket module, and so will use these parameters.The port is the port on which the communication is going to be made.The packet_size defines the size (in bytes) of the batch that will be sent and received.The sentinel is a binary that will be sent to warn the other side that the message is complete.<jupyter_code>HOST = '127.0.0.1'  # The server's hostname or IP address
PORT = 65431        # The port used by the server
PACKETS_SIZE = 16384
SENTINEL = b'BREAK'<jupyter_output><empty_output><jupyter_text>Establishing the communication with AliceThe computation, the training of the model will be done by Alice. We need to establish a communication between the two actors to pass encrypted data, encrypted prediction, and the keys which are mandatory for the computation (in fact Bob will pass the full public context (ie without the secret key). ActorsHere we use the homemade class actors. When initialized an actor tries to connect to the address/ort which is passed as parameter (client role). If nobody is listening, the actor bind a socket and listen to it, waiting for an other actor (server role).For the transmission, an actor sends packet of the size PACKET_SIZE, and wait that the other actor replied b'next', which means that the packet had been received, and that it is waiting for the next one. If the message is not completely sent, the actors iterate the process. Else, the transmitting actor sends b'DONE' to signal to the other actor that all the packets had been sent, and the full message can be deserialize, or use...<jupyter_code>BOB = Actors.Actor(HOST, PORT, PACKETS_SIZE, SENTINEL)<jupyter_output>08/06/2020 03:45:34 PM  [INFO    ]  b'Connection accepted'<jupyter_text>Receiving the public contextThe first object Bob is going to send is a tenseal context object. The context is the main tenseal object. When created, it holds the secret and public keys. So it is needed to encrypt a vector. Bob will add to this context the relin and Galois keys. These keys are mandatory for every computation, therefore every vector encrypted with a specific context holds an attribute which is a pointer to the context. Moreover,if the context is private (ie holds the secret key), the method decrypt() on a vector will not need the secret key, as it can be accessed with vector.context.secret_key.It really convenient for computations but raise on issue on serialization. Alice will need the relin and Galois keys, so Bob will send the context to Alice, obviously without the secret key. So Bob will store the secret key, and will have the context drop it. Then Bob can generate the relin/galois keys, and sent it to Alice.Alice will also need the context for deserialization of the encrypted vectors. As we explained before, to allow computation, a vector needs to point to his context, but is serialized without his context, to save memory. Therefore for the the deserialization, Alice needs to provide the context.<jupyter_code>logging.info('Waiting for serialized context')

b_context = BOB.reception()
logging.info('Context received. Starting deserialization.')
context = ts.context_from(b_context)
logging.info('Context successfully deserialized')<jupyter_output>08/06/2020 02:31:03 PM  [INFO    ]  Waiting for serialized context
08/06/2020 02:31:04 PM  [INFO    ]  Context received. Starting deserialization.
08/06/2020 02:31:08 PM  [INFO    ]  Context successfully deserialized<jupyter_text>Receiving serialized dataNow Bob is going to sent data to Alice. First, Bob sent the amount of data that is going to be sent. He also sent the number of features for each instance of data, which will be useful for weight initialization of the model<jupyter_code>len_X, nb_features = pk.loads(BOB.reception())
b_Y = [None for _ in range(len_X)]
b_X = [None for _ in range(len_X)]<jupyter_output><empty_output><jupyter_text>Then Alice will receive the data, which is packed in a pickled python dict, holding the data, the true label, and the id of the data. We use the pickle module to serialize and deserialize the dict. As the tenseal vector cannot be serialize by pickle for the reasons we evocated few lines ago, we have to first call pickle to deserialize a dict containing references to strings, int and binaries and then deserialize the binary into tenseal object with tenseal's methods.<jupyter_code>while True:
    b_data = BOB.reception()
    if b_data==b'DONE': break
    data = pk.loads(b_data)
    b_X[data['id']]=data['X']
    b_Y[data['id']]=data['label']
    
    if data[0] == len(b_X) // 4:
        logging.info("25 % ...")
    elif data[0] == len(b_X) // 2 :
        logging.info("50 % ...")
    elif data[0] == 3* len(b_X)//4:
        logging.info("75% ...")
    
assert not (None in b_X), 'Missing value: ' + str([idx for idx,e in enumerate(b_X) if e is None])<jupyter_output>08/06/2020 02:32:14 PM  [INFO    ]  25 % ...
08/06/2020 02:33:20 PM  [INFO    ]  50 % ...
08/06/2020 02:34:25 PM  [INFO    ]  75% ...<jupyter_text>Initialize the weightThe weight will be updated by an encrypted gradient (computed from encrypted data), so they have to be encrypted.We initialize them to be small, around 0 (but not exactly 0). As homomorphic encryption scheme CKKS allows only multiplication and addition, we have to approximate non-linear functions (as sigmoid and log) as polynomials. We use a minmax approximation, which is developed in the notebook Appendix B.The sigmoid approximation is relevant between -5 and 5. To ensure that we stay between these values, we initialize the weight to small value (here the weight are randomly sampled from a normal distribution centered on 0, with a standard deviation of 0.2. ). To avoid that these weights explode during the training, we impose a l2 regularization.<jupyter_code>%%memit
unencrypted_weight = np.random.normal(loc=0,
                                      scale=0.2, size =(nb_features))
logging.info(unencrypted_weight)
weight = ts.ckks_vector(context, unencrypted_weight.tolist())
unencrypted_bias = np.random.random((1))
bias = ts.ckks_vector(context, unencrypted_bias.tolist())<jupyter_output>08/06/2020 02:35:31 PM  [INFO    ]  [ 0.14119233  0.13322082  0.10677418  0.14946615 -0.17499522  0.11817762
 -0.09094475  0.05006976  0.1208017  -0.05478917  0.06556012 -0.05331097
 -0.04307331  0.12412614 -0.42812706  0.12723906  0.00043748 -0.06539209
 -0.20330532 -0.29728731  0.10161476 -0.3024797   0.07153177  0.26603781
 -0.17171421 -0.03019694  0.00290262  0.03083542  0.05290126 -0.01780719
 -0.2148345  -0.23494342]
08/06/2020 02:35:31 PM  [INFO    ]  [-0.24488505 -0.40020771  0.26734085  0.06336104 -0.01561979 -0.19677161
  0.29498591 -0.00980197  0.04640455 -0.09566694 -0.12725217  0.3803441
 -0.31543093 -0.10960499 -0.05537333 -0.30819449  0.00613294  0.068314
  0.13851433  0.15245565 -0.16241355  0.28786917 -0.14767324 -0.39154034
 -0.30012324 -0.60976281  0.01604236 -0.12119899  0.04235395  0.12985458
  0.03750433  0.20356318]
peak memory: 2835.16 MiB, increment: 0.98 MiB<jupyter_text>Training the crypted models  Initialization of the model...<jupyter_code>logging.info("Model initialization")
model = encrypted_LR.LogisticRegressionHE(init_weight=weight,
                                          init_bias=bias,
                                          bob=BOB,
                                          context=context,
                                          verbose=VERBOSE,
                                          save_weight=SAVE_WEIGHT,
                                          max_epoch=EPOCH,
                                          learning_rate=LEARNING_RATE,
                                          momentum_rate=MOMENTUM_RATE,
                                          reg_para=REGULARIZATION_RATE,
                                          n_jobs = N_JOBS)<jupyter_output>08/06/2020 02:35:31 PM  [INFO    ]  Model initialization<jupyter_text>Warn Bob that Alice is starting the trainingAt this point Alice has received all she needs. The model is ready, Alice can start the training. While training its models, Alice will probably need to make more computations than allowed by the safety parameters (here around 8 multiplications are allowed before running out of scale). So in order to evaluate calculus circuit of arbitrary depth, Alice needs a "refresh" function, which reset this dept. To do so, Bob will listening to Alice, and when needed, Alice will sent a vector to Bob. Bob will deserialize it, decrypt it, re-encrypt it, and after serialization, send it back to Alice. Therefore, Alice will receive a freshly encrypted vector, which she will be able to use. She never saw any secret key nor unencrypted data.The idea is that the safety parameters allow Alice to perform one epoch. So every epoch, as we said, Alice needs to refresh the weights.<jupyter_code>BOB.transmission(b'start_fitting')
logging.info("Training starting")
timer=time.time()
model.fit(b_X, b_Y)
logging.info("Training done. " + str(round(time.time() - timer, 0)) + " seconds")<jupyter_output>08/06/2020 02:36:20 PM  [INFO    ]  Training starting
08/06/2020 02:36:20 PM  [INFO    ]  Data already serialized
08/06/2020 02:36:20 PM  [INFO    ]  Deserialization of the label for the future computations of the loss
08/06/2020 02:36:26 PM  [INFO    ]  Initialization of 8 workers
08/06/2020 02:36:34 PM  [INFO    ]  Initialization done for process Process-3. Len of data : 69
08/06/2020 02:36:34 PM  [INFO    ]  Initialization done for process Process-4. Len of data : 69
08/06/2020 02:36:35 PM  [INFO    ]  Initialization done for process Process-7. Len of data : 68
08/06/2020 02:36:35 PM  [INFO    ]  Initialization done for process Process-5. Len of data : 68
08/06/2020 02:36:35 PM  [INFO    ]  Initialization done for process Process-6. Len of data : 68
08/06/2020 02:36:35 PM  [INFO    ]  Initialization done for process Process-8. Len of data : 68
08/06/2020 02:36:36 PM  [INFO    ]  Initialization done for process Process-9. Len of data : 68
08/06/2020 02:36:36 PM  [INFO    ]  Initializ[...]<jupyter_text>Warn Bob that Alice finished the training, so Bob no longer needs to refresh vectors.<jupyter_code>BOB.transmission(b'STOP_REFRESH')<jupyter_output><empty_output><jupyter_text>Alice cannot exploit the loss, as it is encrypted. But she can sent it to Bob, to help him evaluating the model.<jupyter_code>ser_loss = [l.serialize() for l in model.loss_list]
ser_epoch = [i*VERBOSE for i in range(len(model.loss_list))]
BOB.transmission(pk.dumps((ser_epoch, ser_loss)))<jupyter_output><empty_output><jupyter_text>Now Alice will make her model available to Bob. To do so, she waits for new data, on which Bob wants predictions. When Alice receives a data, she uses her model to make a prediction (encrypted) and sent it back to Bob. Then she iterates the process, and wait for the next data to predict on.<jupyter_code>while True:
    b_data = BOB.reception()
    if b_data == b'STOP_PREDICT': break
    key, vector = pk.loads(b_data)
    prediction = model.predict(ts.ckks_vector_from(model.context, vector))
    BOB.transmission(pk.dumps((key, prediction.serialize())))<jupyter_output><empty_output><jupyter_text>When Bob finished to do his prediction, Alice stop waiting for data, and close the communication.<jupyter_code>BOB.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07018877486380337, 'fraction_numerical': 0.06220701887748638, 'mean_word_length': 3.8665228113440198, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '227532', 'license': 'MIT', 'n_tokens_mistral': 5122, 'n_tokens_neox': 4477, 'n_words': 1948, 'path': 'notebooks/1-ap-Alice.ipynb', 'repo_name': 'arthurPignet/privacy-preserving-titanic-challenge', 'size': '15805'}