<jupyter_start><jupyter_text>Encrypted Inference-Linear Regression Author:    <NAME> - Twitter - GithubEncrypted Inference is the process of performing inference with machine learning models such that model owner cannot observe the true input data nor can the data owner see the true model weights. The weights and data are encrypted by splitting them into shares and performiming computations according to a protocol. The general class of methods know as Secure Multi Party Computation (SMPC). Below figure depicts MPC for ML models for 2 parties.  In this example we use Virtual Machine to demonstrate performing inference using SMPC. That is the workers are present on the same PC. If you want to understand how to perform remotely check out duet tutorials. In this example, we train a Linear regression model in plaintext on Boston Housing Dataset. Then we use the model for performing encrypted inference on test data. This tutorial uses protocol Falcon for 3 parties and SPDZ for 3 and 5 parties. In SyMPC the computation between parties occurs using a orchestrator which describes how computations should take place.<jupyter_code>#External libraries
import pandas as pd
import numpy as np
import time
#Import torch
import torch
import torch.nn as nn
import torch.utils.data as data_utils
#Set a manual seed to maintain consistency
torch.manual_seed(0)<jupyter_output><empty_output><jupyter_text>Data Loading and Processing<jupyter_code>#Download Boston housing dataset
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/housing/housing.data
#Import dataset and add headers
dataset=pd.read_csv("housing.data",delim_whitespace=True,
                    names=["crim","zn","indus",
                           "chas","nox","rm",
                           "age","dis","rad",
                           "tax","ptratio","black",
                           "lstat","medv"])
#Visualize and look at columns and rows of dataset
dataset.head()
#Split data into features and target variables
features = dataset.drop("medv",axis=1)
targets = dataset["medv"]
#Normalize features
features = features.apply(
    lambda x: (x - x.mean()) / x.std()
)
features
targets
#Convert features and targets into torch tensors
features = torch.tensor(features.values.astype(np.float32)) 
targets = torch.tensor(targets.values.astype(np.float32))
# Arguments for training
batch_size = 16
epochs = 300
train_test_split = 0.8
lr = 0.001
#Split dataset into train and test
train_indices=int(len(features)*train_test_split)

train_x = features[:train_indices]
train_y = targets[:train_indices]

test_x = features[train_indices+1:]
test_y = targets[train_indices+1:]
#Divide dataset into batches
def get_batches(X, y):
    batches = []
    for index in range(0,len(train_x)+1,batch_size):
        batches.append((X[index:index+batch_size],y[index:index+batch_size]))
    
    return batches
train_batches=get_batches(train_x,train_y)<jupyter_output><empty_output><jupyter_text>Plaintext Training<jupyter_code>#Import syft
import syft as sy
sy.logger.remove()
#Define Linear regression model
class LinearSyNet(sy.Module):
    def __init__(self, torch_ref):
        super(LinearSyNet, self).__init__(torch_ref=torch_ref)
        self.fc1 = self.torch_ref.nn.Linear(13,1)

    def forward(self, x):
        x = self.fc1(x)
        return x
#Define model, loss function and optimizer
model = LinearSyNet(torch)
criterion = torch.nn.MSELoss(reduction='mean') 
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
#Training Loop
for epoch in range(epochs):
  running_loss = 0.0
  for index in range(0,len(train_batches)):
    # Clear gradient buffers because we don't want any gradient from previous epoch to carry forward, dont want to cummulate gradients
    optimizer.zero_grad()

    # get output from the model, given the inputs
    outputs = model(train_batches[index][0]).reshape([-1])

    # get loss for the predicted output
    loss = criterion(outputs,train_batches[index][1])
    running_loss += loss
    # get gradients w.r.t to parameters
    loss.backward()

    # update parameters
    optimizer.step()
    
  test_accuracy = criterion(model(test_x).reshape([-1]),test_y)
  if((epoch%50)==0):
     print(f"Epoch {epoch}/{epochs}  Running Loss : {running_loss.item()/batch_size} and test loss: {test_accuracy.item()}")<jupyter_output>Epoch 0/300  Running Loss : 978.0777587890625 and test loss: 367.94952392578125
Epoch 50/300  Running Loss : 52.76874542236328 and test loss: 108.14092254638672
Epoch 100/300  Running Loss : 43.0278434753418 and test loss: 47.971736907958984
Epoch 150/300  Running Loss : 41.08037185668945 and test loss: 30.054576873779297
Epoch 200/300  Running Loss : 40.123294830322266 and test loss: 23.0802059173584
Epoch 250/300  Running Loss : 39.57941436767578 and test loss: 20.64508628845215<jupyter_text>Plaintext Inference<jupyter_code>#Perform inference in plaintext
start_time=time.time()
plaintext_predictions = model(test_x).reshape([-1])
end_time=time.time()
#Calculate inference time and MSELoss
print("MSE Loss: ",criterion(plaintext_predictions,test_y).item())
print("Inference time: ",str(end_time-start_time),"s")<jupyter_output>MSE Loss:  20.227853775024414
Inference time:  0.0007698535919189453 s<jupyter_text>Encrypted Inference<jupyter_code>#SyMPC imports required for encrypted inference
import sympc
from sympc.session import Session
from sympc.session import SessionManager
from sympc.tensor import MPCTensor
from sympc.protocol import Falcon,FSS
def get_clients(n_parties):
  #Generate required number of syft clients and return them.

  parties=[]
  for index in range(n_parties): 
      parties.append(sy.VirtualMachine(name = "worker"+str(index)).get_root_client())

  return parties
def split_send(data,session):
    """Splits data into number of chunks equal to number of parties and distributes it to respective 
       parties.
    """
    data_pointers = []
    
    split_size = int(len(data)/len(session.parties))+1
    for index in range(0,len(session.parties)):
        ptr=data[index*split_size:index*split_size+split_size].share(session=session)
        data_pointers.append(ptr)
        
    return data_pointers
def inference(n_clients,protocol=None):
    
  # Get VM clients 
  parties=get_clients(n_clients)

  # Setup the session for the computation
  if(protocol):
     session = Session(parties = parties,protocol = protocol)
  else:
     session = Session(parties = parties)
        
  SessionManager.setup_mpc(session)

  #Split data and send data to clients
  pointers = split_send(test_x,session)

  #Encrypt model 
  mpc_model = model.share(session)

  #Encrypt test data
  #test_data=MPCTensor(secret=test_x, session = session)

  #Perform inference and measure time taken
  start_time = time.time()
    
  results = []
    
  for ptr in pointers:
     encrypted_results = mpc_model(ptr)
     plaintext_results = encrypted_results.reconstruct()
     results.append(plaintext_results)
        
  end_time = time.time()

  print(f"Time for inference: {end_time-start_time}s")
    
  predictions = torch.cat(results).reshape([-1])

  #Calculate Loss
  print("MSE Loss: ",criterion(predictions,test_y).item())
    
  return predictions
predictions=inference(3,Falcon("semi-honest"))<jupyter_output>Time for inference: 0.11805987358093262s
MSE Loss:  20.227657318115234<jupyter_text>We can see that the prediction values and mean squared error values are almost the same as final model. Small differences are due to precision loss.<jupyter_code>for index in range(0,10):
    print(f"Index {index}")
    print(f"Encrypted Prediction Output {predictions[index].item()}")
    print(f"Plaintext Prediction Output {plaintext_predictions[index].item()}")
    print(f"Expected Prediction: {test_y[index]}")
    print("\n")<jupyter_output>Index 0
Encrypted Prediction Output 2.636962890625
Plaintext Prediction Output 2.6367740631103516
Expected Prediction: 5.0


Index 1
Encrypted Prediction Output 4.957275390625
Plaintext Prediction Output 4.957102298736572
Expected Prediction: 11.899999618530273


Index 2
Encrypted Prediction Output 19.214553833007812
Plaintext Prediction Output 19.214506149291992
Expected Prediction: 27.899999618530273


Index 3
Encrypted Prediction Output 12.028610229492188
Plaintext Prediction Output 12.028535842895508
Expected Prediction: 17.200000762939453


Index 4
Encrypted Prediction Output 19.063995361328125
Plaintext Prediction Output 19.063936233520508
Expected Prediction: 27.5


Index 5
Encrypted Prediction Output 11.89825439453125
Plaintext Prediction Output 11.898107528686523
Expected Prediction: 15.0


Index 6
Encrypted Prediction Output 16.335174560546875
Plaintext Prediction Output 16.335037231445312
Expected Prediction: 17.200000762939453


Index 7
Encrypted Prediction Output -1.255981[...]<jupyter_text>Conclusion  Falcon can also provide a malicious security guarantee for an honest majority at the cost of higher inference time. Malicious security ensures that all the parties compute according to the protocol and do not deviate from protocol or tamper with shares.<jupyter_code>predictions=inference(3,Falcon("malicious"))<jupyter_output>Time for inference: 0.665564775466919s
MSE Loss:  20.227645874023438<jupyter_text>When we do not pass any protocol to session, SyMPC uses SPDZ and FSS protocol with semi-honest security type. SPDZ is used for multiplication and related operations (convolution,matmul,etc).Functional Secret Sharing (FSS) for other operations such as comparison, equality, maxpool, etc. FSS works only for 2 parties while SPDZ could extend to N parties. Linear regression uses only matmul which utilizes SPDZ protocol allowing us to run linear regression with several parties in this tutorial.<jupyter_code>predictions=inference(3)
predictions=inference(5)<jupyter_output>Time for inference: 1.31923508644104s
MSE Loss:  20.22768211364746
{'fraction_non_alphanumeric': 0.08271742431912524, 'fraction_numerical': 0.0744153082919915, 'mean_word_length': 4.883263847528291, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '476196', 'license': 'MIT', 'n_tokens_mistral': 3537, 'n_tokens_neox': 3004, 'n_words': 921, 'path': 'examples/Encrypted-Inference-LinearRegression.ipynb', 'repo_name': 'manel1874/SyMPC', 'size': '9888'}