<jupyter_start><jupyter_text>Encrypted Linear Regression In this tutorial you are going to see how you can run a linear regression model on **data distributed in a pool of workers** with **encrypted computations leveraged by Secured Multi-Party Computation**. For this demonstration we are going to use the classical Housing Prices dataset that is already available in the VirtualGrid set up by the Syft Sandbox.The idea for the implementation of the Encrypted Linear Regression algorithm in PySyft is based on the section 2 of [this paper](https://arxiv.org/abs/1901.09531) written by <NAME> of the Broad Institute of MIT and Harvard.**Author**: <NAME>. Github: [@andrelmfarias](https://github.com/andrelmfarias) | Twitter: [@andrelmfarias](https://twitter.com/andrelmfarias)  1. Preliminaries First, let's import PySyft and PyTorch and set up the Syft sandbox, which will create all the objects and tools we will need to run our simulation (Virtual Workers, VirtualGrid with datasets, etc...)<jupyter_code>import warnings
warnings.filterwarnings("ignore")
import torch
import syft as sy
sy.create_sandbox(globals(), verbose=False)<jupyter_output>WARNING:tf_encrypted:Falling back to insecure randomness since the required custom op could not be found for the installed version of TensorFlow. Fix this by compiling custom ops. Missing file was '/Users/andre.farias/venvs/pysyft/lib/python3.7/site-packages/tf_encrypted/operations/secure_random/secure_random_module_tf_1.14.0.so'<jupyter_text>You can see that we have several workers already set up:<jupyter_code>workers<jupyter_output><empty_output><jupyter_text>And each one has a chunk of the Housing Prices dataset:<jupyter_code>for worker in workers:
    print(worker.search(["#housing", "#data"]))<jupyter_output>[(Wrapper)>[PointerTensor | me:30189164229 -> bob:89440057589]
	Tags: #data #boston .. _boston_dataset: #boston_housing #housing 
	Shape: torch.Size([84, 13])
	Description: .. _boston_dataset:...]
[(Wrapper)>[PointerTensor | me:8249706715 -> theo:46905611490]
	Tags: #data #boston .. _boston_dataset: #boston_housing #housing 
	Shape: torch.Size([84, 13])
	Description: .. _boston_dataset:...]
[(Wrapper)>[PointerTensor | me:40521198768 -> jason:18847732952]
	Tags: #data #boston .. _boston_dataset: #boston_housing #housing 
	Shape: torch.Size([84, 13])
	Description: .. _boston_dataset:...]
[(Wrapper)>[PointerTensor | me:38604677687 -> alice:85568866295]
	Tags: #data #boston .. _boston_dataset: #boston_housing #housing 
	Shape: torch.Size([84, 13])
	Description: .. _boston_dataset:...]
[(Wrapper)>[PointerTensor | me:84600693994 -> andy:27396108387]
	Tags: #data #boston .. _boston_dataset: #boston_housing #housing 
	Shape: torch.Size([84, 13])
	Description: .. _boston_dataset:...]
[(Wrapper)[...]<jupyter_text>2. Encrypted Linear Regression with PySyft   2.1 Loading Housing Prices data from Virtual Grid Now we have our Syft environment set, let's load the data.Please note that in order to avoid overflow with the SMPC computations performed by the linear model, and to maintain its stability, **we need to scale the data in a such way that the magnitude of each coordinate average lies in the interval [0.1, 10]**.Usually that can be done without revealing the data or the averages, you only need to have an idea of the order of magnitude. For example, if one of the coordinate is the surface of the house and it is represented in m², you should scale it by dividing by 100, as we know the surfaces of houses have an order of magnitude close to 100 in average.After running the model and obtaining the main statistics, we can rescale them back if needed. The same can be done with predictions.In this tutorial I will be loading the data and scale them following this idea:<jupyter_code>scale_data = torch.Tensor([10., 10.,  10., 1., 1., 10., 100., 10., 10., 1000., 10., 1000., 10.])
scale_target = 100.0

housing_data = []
housing_targets = []
for worker in workers:
    housing_data.append(worker.search(["#housing", "#data"])[0] / scale_data.send(worker))
    housing_targets.append(worker.search(["#housing", "#target"])[0] / scale_target)<jupyter_output><empty_output><jupyter_text>2.2 Setting up 2 more Virtual workers: the crypto provider and the "honest but curious" worker In order to run the linear regression, we will need **two more workers**, a *crypto provider* and a *honest but curious* worker. Both are necessary to assure the security of the SMPC computations when we run the model in a pool with more than 3 workers.> *Note: the **honest but curious** worker is a legitimate participant in a communication protocol who will not deviate from the defined protocol but will attempt to learn all possible information from legitimately received messages.*<jupyter_code>crypto_prov = sy.VirtualWorker(hook, id="crypto_prov")
hbc_worker = sy.VirtualWorker(hook, id="hbc_worker")<jupyter_output><empty_output><jupyter_text>2.3 Running Encrypted Linear Regression with SMPC Now let's import the EncryptedLinearRegression from the linalg module of pysyft:<jupyter_code>from syft.frameworks.torch.linalg import EncryptedLinearRegression<jupyter_output><empty_output><jupyter_text>Let's train the model!!<jupyter_code>crypto_lr = EncryptedLinearRegression(crypto_provider=crypto_prov, hbc_worker=hbc_worker)
crypto_lr.fit(housing_data, housing_targets)<jupyter_output><empty_output><jupyter_text>We can display the results with the method `.summarize()`<jupyter_code>crypto_lr.summarize()<jupyter_output>====================================================
           SMPC Linear Regression Results
====================================================
                 value         stderr        p-value
----------------------------------------------------
coef1          -0.0108         0.0033         0.0011
coef2           0.0046         0.0014         0.0008
coef3           0.0020         0.0062         0.7452
coef4           0.0269         0.0086         0.0019
coef5          -0.1775         0.0382         0.0000
coef6           0.3807         0.0418         0.0000
coef7           0.0008         0.0132         0.9541
coef8          -0.1474         0.0200         0.0000
coef9           0.0305         0.0066         0.0000
coef10         -0.1228         0.0376         0.0012
coef11         -0.0953         0.0131         0.0000
coef12          0.0930         0.0269         0.0006
coef13         -0.0525         0.0051         0.0000
intercept       0.3645         0.0511         0.0000
----[...]<jupyter_text>**We can see that the EncryptedLinearRegression does not only give the coefficients and intercept values, but also their standard errors and the p-values!**  3. Comparing results with other linear regressors Now, in order to show the effectiveness of the EncryptedLinearRegression, let's compare it with the Linear Regression from other known libraries.  3.1 Sending data to local server for comparison purposes First, let's send the data to the local worker and transform the `torch.Tensor`s in `numpy.array`s<jupyter_code>import numpy as np

data_tensors = [x.copy().get() for x in housing_data] 
target_tensors = [y.copy().get() for y in housing_targets]

data_np = torch.cat(data_tensors, dim=0).numpy()
target_np = torch.cat(target_tensors, dim=0).numpy()<jupyter_output><empty_output><jupyter_text>3.2 Scikit-learn First let's compare the results with the sklearn's Linear Regression:<jupyter_code>from sklearn.linear_model import LinearRegression
lr = LinearRegression().fit(data_np, target_np.squeeze())<jupyter_output><empty_output><jupyter_text>Display the results:<jupyter_code>print("=" * 25)
print("Sklearn Linear Regression")
print("=" * 25)
for i, coef in enumerate(lr.coef_, 1):
    print(" coeff{:<3d}".format(i), "{:>14.4f}".format(coef))
print(" intercept:", "{:>12.4f}".format(lr.intercept_))
print("=" * 25)<jupyter_output>=========================
Sklearn Linear Regression
=========================
 coeff1          -0.0108
 coeff2           0.0046
 coeff3           0.0021
 coeff4           0.0269
 coeff5          -0.1777
 coeff6           0.3810
 coeff7           0.0007
 coeff8          -0.1476
 coeff9           0.0306
 coeff10         -0.1233
 coeff11         -0.0953
 coeff12          0.0931
 coeff13         -0.0525
 intercept:       0.3646
=========================<jupyter_text>**You can notice that the are results are pretty much the same!! The are some small differences, but they are never higher than 0.2% of the value computed by the sklearn model!!****For an ecrypted model that can compute linear regression coefficients without ever revealing the data, this is a huge achievement!**  3.3 Statsmodel API We can do the same using the Linear Regression from Statsmodel API, which also gives us the **standard errors** and **p-values** of the coefficients. We can then compare it with the results given by the EncryptedLinearRegression<jupyter_code>import statsmodels.api as sm
mod = sm.OLS(target_np.squeeze(), sm.add_constant(data_np), hasconst=True)
res = mod.fit()
print(res.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.741
Model:                            OLS   Adj. R-squared:                  0.734
Method:                 Least Squares   F-statistic:                     108.1
Date:                Fri, 11 Oct 2019   Prob (F-statistic):          6.72e-135
Time:                        15:46:06   Log-Likelihood:                 831.41
No. Observations:                 506   AIC:                            -1635.
Df Residuals:                     492   BIC:                            -1576.
Df Model:                          13                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]
{'fraction_non_alphanumeric': 0.1259222333000997, 'fraction_numerical': 0.058823529411764705, 'mean_word_length': 3.183069224353628, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 60, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '98624', 'license': 'Apache-2.0', 'n_tokens_mistral': 3261, 'n_tokens_neox': 2863, 'n_words': 979, 'path': 'examples/tutorials/advanced/Encrypted Linear Regression.ipynb', 'repo_name': 'richarajpal/PySyft', 'size': '10051'}