<jupyter_start><jupyter_text>Running FL with secure aggregation using homomorphic encryptionThis notebook will walk you through how to setup FL with homomorphic encryption (HE).  PrerequisitesBefore starting this notebook, please make yourself familiar with other FL notebooks in this repo.- (Optional) Look at the introduction Notebook for [Federated Learning with Clara Train SDK](FederatedLearning.ipynb).- (Optional) Look at [Client Notebook](Client.ipynb).- (Optional) Look at [Admin Notebook](Admin.ipynb).- Run [Provisioning Notebook](Provisioning.ipynb) and started the server.Make sure the project.yml used for provision contains these HE related settings:     homomorphic encryption    he:      lib: tenseal      config:        poly_modulus_degree: 8192        coeff_mod_bit_sizes: [60, 40, 40]        scale_bits: 40        scheme: CKKS        > **Note:** These settings are recommended and should work for most tasks but could be further optimized depending on your specific model architecture and machine learning task. See this [tutorial on the CKKS scheme](https://github.com/OpenMined/TenSEAL/blob/master/tutorials/Tutorial%202%20-%20Working%20with%20Approximate%20Numbers.ipynb) and [benchmarking](https://github.com/OpenMined/TenSEAL/blob/master/tutorials/Tutorial%203%20-%20Benchmarks.ipynb) for more information on different settings.  Dataset  Option 1 This notebook uses a sample dataset (i.e. a single image volume of the spleen dataset) provided in the package to train a small neural network for a few epochs. This single file is duplicated 32 times for the training set and 9 times for the validation set to mimic the full spleen dataset.  Option 2  You could do minor changes as recommended in the exercise section in the [Admin notebook](./Admin.ipynb) to train on the spleen segmentation task. The dataset used is Task09_Spleen.tar from the [Medical Segmentation Decathlon](http://medicaldecathlon.com/). Prior to running this notebook the data should be downloaded following the steps in [Data Download Notebook](../../Data_Download.ipynb).  Disclaimer  We will be training a small networks so that both clients can fit the model on 1 GPU. Training will run for a couple of epochs, in order to show the concepts, we are not targeting accuracy.  Lets get startedIn order to learn how FL works with homomorphic encryption (HE) in Clara Train SDK we will first give some background on what homomorphic encryption is and how the MMAR configurations need to be modified to enable it.   New HE componentsWe implemented secure aggregation during FL with homomorphic encryption using the [TenSEAL library](https://github.com/OpenMined/TenSEAL) by OpenMined, a convenient wrapper around [Microsoft SEAL](https://github.com/microsoft/SEAL). Both libraries are available as open-source and provide an implementation of ["Homomorphic encryption for arithmetic of approximate numbers"](https://eprint.iacr.org/2016/421.pdf), aka the "CKKS" scheme, which was proposed as a solution for [encrypted machine learning](https://en.wikipedia.org/wiki/Homomorphic_encryptionFourth-generation_FHE) and which we use for these FL experiments. The configuration files in `adminMMAR_HE` use the following new componets that are needed on top or instead of standard FL components used in Clara Train. See the [nvflare docs](https://docs.nvidia.com/clara/clara-train-sdk/federated-learning/apidocs/nvflare.html) for more details.  Client-side See `config_fed_client.json`: `HEModelEncryptor`A filter to the encrypt Shareable object that being sent to the server.```Args:    tenseal_context_file: tenseal context files containing encryption keys and parameters    encrypt_layers: if not specified (None), all layers are being encrypted;                    if list of variable/layer names, only specified variables are encrypted;                    if string containing regular expression (e.g. "conv"), only matched variables are being encrypted.    aggregation_weights: dictionary of client aggregation `{"client1": 1.0, "client2": 2.0, "client3": 3.0}`;                         defaults to a weight of 1.0 if not specified.    weigh_by_local_iter: If true, multiply client weights on first before encryption (default: `True` which is recommended for HE)```HE will increase the message sizes when encrypting the model updates of each client. One can choose to not encrypt all layers but specify which layers to encrypt, see argument `encrypt_layers`.To choose the layer names for a given model, one can use:<jupyter_code>from monai.networks.nets.unet import UNet

# use the same configuration as in adminMMAR_HE
net = UNet(
    dimensions=3,
    in_channels=1,
    out_channels=2,
    channels=[16, 32, 64, 128, 256],
    strides=[2, 2, 2, 2],
    num_res_units=2    
)

# here, we only print convolutional layers that we might want to encrypt
for key in net.state_dict().keys():
    if 'conv' in key:
        print(key)<jupyter_output>model.0.conv.unit0.conv.weight
model.0.conv.unit0.conv.bias
model.0.conv.unit0.adn.A.weight
model.0.conv.unit1.conv.weight
model.0.conv.unit1.conv.bias
model.0.conv.unit1.adn.A.weight
model.1.submodule.0.conv.unit0.conv.weight
model.1.submodule.0.conv.unit0.conv.bias
model.1.submodule.0.conv.unit0.adn.A.weight
model.1.submodule.0.conv.unit1.conv.weight
model.1.submodule.0.conv.unit1.conv.bias
model.1.submodule.0.conv.unit1.adn.A.weight
model.1.submodule.1.submodule.0.conv.unit0.conv.weight
model.1.submodule.1.submodule.0.conv.unit0.conv.bias
model.1.submodule.1.submodule.0.conv.unit0.adn.A.weight
model.1.submodule.1.submodule.0.conv.unit1.conv.weight
model.1.submodule.1.submodule.0.conv.unit1.conv.bias
model.1.submodule.1.submodule.0.conv.unit1.adn.A.weight
model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.weight
model.1.submodule.1.submodule.1.submodule.0.conv.unit0.conv.bias
model.1.submodule.1.submodule.1.submodule.0.conv.unit0.adn.A.weight
model.1.submodule.1.submodule.1.[...]<jupyter_text>Based on this output, our example `config_fed_client.json` chooses three layers to encrypt:```"outbound_filters": [  {    "path": "flare.experimental.homomorphic_encryption.he_model_encryptor.HEModelEncryptor",    "args": {      "encrypt_layers": [        "model.0.conv.unit0.conv.weight",        "model.1.submodule.1.submodule.1.submodule.2.0.conv.weight",        "model.2.1.conv.unit0.conv.weight"      ],      "aggregation_weights": {        "client1":  0.4,        "client2":  0.6      }    }  }```Using this setting, the client will see an output like this:```2021-05-14 18:23:53,547 - HEModelEncryptor - INFO - Running HE Encryption algorithm on 63 variables2021-05-14 18:23:53,547 - HEModelEncryptor - INFO - Encrypting vars 1 of 63: model.0.conv.unit0.conv.weight with 432 values2021-05-14 18:23:53,566 - HEModelEncryptor - INFO - Encrypting vars 41 of 63: model.1.submodule.1.submodule.1.submodule.2.0.conv.weight with 663552 valuesWARNING: The input does not fit in a single ciphertext, and some operations will be disabled.The following operations are disabled in this setup: matmul, matmul_plain, enc_matmul_plain, conv2d_im2col.If you need to use those operations, try increasing the poly_modulus parameter, to fit your input.2021-05-14 18:23:54,086 - HEModelEncryptor - INFO - Encrypting vars 62 of 63: model.2.1.conv.unit0.conv.weight with 108 values2021-05-14 18:23:54,089 - HEModelEncryptor - INFO - Encryption time for 664092 of 4806481 params (encrypted value range [-0.0016116723418235779, 0.0016116723418235779]) 0.541804313659668 seconds.```> **Note:** the warning from TenSEAL is expected and can be ignored as these more advanced operations are not used in the `FedAvg` algorithm.  `HEModelDecryptor`A filter to decrypt Shareable object, i.e. the updated global model received from the server.```Args:    tenseal_context_file: tenseal context files containing decryption keys and parameters```> **Note:** The tenseal_context_file for the client will be generated by the provision tool and is part of the startup kit, see [Provisioning](./Provisioning.ipynb).  `HEPTModelReaderWriter`This component is used as argument to `ClientTrainer` to reshape the decrypted parameter vectors to the local Pytorch model for training.  `HEEvalDecryptor`Filter to decrypt encrypted Shareable object (i.e. global model(s)) distributed during cross-site validation. Currently, only the global server models are encrypted. Locally best models are shared unencrypted.> **Note:** cross-site validation is optional and a client does not need to participate if not wanted.```Args:    tenseal_context_file: tenseal context files containing decryption keys and parameters    defaults to `False` for use during FL training```The client will see an output like this```2021-05-14 18:23:59,886 - HEModelDecryptor - INFO - Running HE Decryption algorithm 63 variables2021-05-14 18:23:59,886 - HEModelDecryptor - INFO - Decrypting vars 1 of 63: model.0.conv.unit0.conv.weight with 432 values2021-05-14 18:23:59,887 - HEModelDecryptor - INFO - Decrypting vars 41 of 63: model.1.submodule.1.submodule.1.submodule.2.0.conv.weight with 663552 values2021-05-14 18:23:59,976 - HEModelDecryptor - INFO - Decrypting vars 62 of 63: model.2.1.conv.unit0.conv.weight with 108 values2021-05-14 18:23:59,977 - HEModelDecryptor - INFO - Decryption time for 664092 of 664092 params 0.09076642990112305 seconds.```  Server-side See `config_fed_server.json`:  `HEInTimeAccumulateWeightedAggregator`This aggregator can perform federated averaging (i.e. the [`FedAvg`](http://proceedings.mlr.press/v54/mcmahan17a.html) algorithm) in encrypted space. The server doesn't have a key for decryption and only processes the encrypted values sent by the clients.)```Args:    exclude_vars: variable names that should be excluded from aggregation (use regular expression)    aggregation_weights: dictionary of client aggregation `{"client1": 1.0, "client2": 2.0, "client3": 3.0}`;                         defaults to a weight of 1.0 if not specified. Will be ignored if weigh_by_local_iter: False (default for HE)    weigh_by_local_iter: If true, multiply client weights on first in encryption space                         (default: `False` which is recommended for HE, first multiply happens in `HEModelEncryptor`)```  `HEPTFileModelPersistor`This model persistor is used to save the encrypted models on the server.  `HEModelShareableGenerator`ShareableGenerator converts between Shareable and Learnable objects generated with HE. It is used to update the global model weights using the averaged encrypted updates from the clients. The updated global stays encrypted.    ```Args:    tenseal_context_file: tenseal context files containing decryption keys and parameters``` The output on the server shows how the encrypted layer updates are being aggregated and the global models are updated and saved on the server and the best global model is being saved using `IntimeModelSelectionHandler`.```2021-05-14 18:25:08,354 - FederatedServer - INFO - received Project1_1e41cdaa-e454-432f-a709-fde535d7e03b_1 (55173069 Bytes, 71 seconds)2021-05-14 18:25:08,354 - IntimeModelSelectionHandler - INFO - validation metric 0.05224878713488579 from client client12021-05-14 18:25:08,354 - HEInTimeAccumulateWeightedAggregator - INFO - Adding contribution from client1.2021-05-14 18:25:08,453 - HEInTimeAccumulateWeightedAggregator - INFO - 3 of 63 layers are encrypted.2021-05-14 18:25:08,454 - HEInTimeAccumulateWeightedAggregator - INFO - Round 1 adding client1 time is 0.09885239601135254 seconds2021-05-14 18:25:13,519 - ServerCrossSiteValManager - INFO - Received best model from client12021-05-14 18:25:13,543 - ServerCrossSiteValManager - INFO - Client client1 requested models for cross site validation.2021-05-14 18:25:13,543 - ServerCrossSiteValManager - INFO - Sent 0 out of 3 models to client1. Will be asked to wait and retry.2021-05-14 18:25:16,626 - FederatedServer - INFO - received Project1_49ed5224-0962-4683-a20e-11fe16082a88_1 (55172936 Bytes, 79 seconds)2021-05-14 18:25:16,626 - IntimeModelSelectionHandler - INFO - validation metric 0.05224878713488579 from client client22021-05-14 18:25:16,626 - HEInTimeAccumulateWeightedAggregator - INFO - Adding contribution from client2.2021-05-14 18:25:16,738 - HEInTimeAccumulateWeightedAggregator - INFO - 3 of 63 layers are encrypted.2021-05-14 18:25:16,739 - HEInTimeAccumulateWeightedAggregator - INFO - Round 1 adding client2 time is 0.11211466789245605 seconds2021-05-14 18:25:16,739 - IntimeModelSelectionHandler - INFO - weighted validation metric 0.052248787134885792021-05-14 18:25:16,739 - IntimeModelSelectionHandler - INFO - new best validation metric at round 1: 0.052248787134885792021-05-14 18:25:16,739 - FederatedServer - INFO - > aggregating: 12021-05-14 18:25:16,740 - HEPTFileModelPersistor - INFO - Saving encrypted model on server...2021-05-14 18:25:16,740 - HEPTFileModelPersistor - INFO - 3 of 63 layers are encrypted.2021-05-14 18:25:16,774 - HEPTFileModelPersistor - INFO - Saved encrypted model at /claraDevDay/FL/project1/server/startup/../run_1/mmar_server/models/best_FL_global_model.pt2021-05-14 18:25:16,859 - HEInTimeAccumulateWeightedAggregator - INFO - Aggregated 2 contributions for round 1 time is 0.08475756645202637 seconds2021-05-14 18:25:16,859 - HEInTimeAccumulateWeightedAggregator - INFO - 3 of 63 layers are encrypted.2021-05-14 18:25:16,859 - HEModelShareableGenerator - INFO - HEModelShareableGenerator shareable_to_learnable...2021-05-14 18:25:16,859 - HEModelShareableGenerator - INFO - serialize encrypted model.0.conv.unit0.conv.weight2021-05-14 18:25:16,891 - HEModelShareableGenerator - INFO - serialize encrypted model.1.submodule.1.submodule.1.submodule.2.0.conv.weight2021-05-14 18:25:16,954 - HEModelShareableGenerator - INFO - serialize encrypted model.2.1.conv.unit0.conv.weight2021-05-14 18:25:16,954 - HEModelShareableGenerator - INFO - Updated global model 63 vars with 4806481 params in 0.0953524112701416 seconds2021-05-14 18:25:16,955 - HEPTFileModelPersistor - INFO - Saving encrypted model on server...2021-05-14 18:25:16,955 - HEPTFileModelPersistor - INFO - 3 of 63 layers are encrypted.```  Running FL experiment with HE  1 - Start server, and clients (if they are not already running)Open four terminals in JupyterLab.In the server terminal run:```cd /claraDevDay/FL/project1/server/startup./start.sh```  In the client1 terminal run:```cd /claraDevDay/FL/project1/client1/startup./start.sh```  In the client2 terminal run:```cd /claraDevDay/FL/project1/client2/startup./start.sh```    2 - Starting Admin ShellIn the admin terminal, if you haven't already started the admin console you should to admin folder in side your project and run```cd /claraDevDay/FL/project1/admin/startup./fl_admin.sh``` you should see```Admin Server: localhost on port 5000User Name: ````type `<EMAIL>` Admin Server: localhost on port 8003User Name: <EMAIL>Type ? to list commands; type "? cmdName" to show usage of a command.  3 - Check server/client statustype ```> check_status server```to see ```FL run number has not been set.FL server status: training not startedRegistered clients: 2 -------------------------------------------------------------------------------------------------| CLIENT NAME | TOKEN                                | LAST ACCEPTED ROUND | CONTRIBUTION COUNT |-------------------------------------------------------------------------------------------------| client1     | f735c245-ce35-4a08-89e0-0292bb053a9c |                     | 0                  || client2     | e36db52e-2624-4989-855a-28fa195f58e9 |                     | 0                  |-------------------------------------------------------------------------------------------------```To check on clients type ```> check_status client```to see ```instance:client1 : client name: client1 token: <PASSWORD>3bf-40c1-bc02-9be84d7c339f     status: training not startedinstance:client2 : client name: client2 token: <PASSWORD>     status: training not started```To check on folder structure ```> info```To see```Local Upload Source: /claraDevDay/FL/project1/admin/startup/../transferLocal Download Destination: /claraDevDay/FL/project1/admin/startup/../transferServer Upload Destination: /claraDevDay/FL/project1/server/startup/../transferServer Download Source: /claraDevDay/FL/project1/server/startup/../transfer  4- Upload and deploy the MMAR configurations for HE and set FL run numberFirst set a run number (Choose a different one if you don't want to overwrite previous results)```> set_run_number 1```Then, upload the HE MMAR and deploy to server and clients```> upload_folder ../../../adminMMAR_HE> deploy adminMMAR_HE server> deploy adminMMAR_HE client```  5 - Start TrainingNow you can start training by:1. `> start server`2. `> start client`You can check on the status of the training using:3. `> check_status client` or `> check_status server`  to see ```FL run number:1FL server status: training startedrun number:1    start round:0   max round:2     current round:0min_num_clients:2       max_num_clients:100Registered clients: 2 Total number of clients submitted models for current round: 0-------------------------------------------------------------------------------------------------| CLIENT NAME | TOKEN                                | LAST ACCEPTED ROUND | CONTRIBUTION COUNT |-------------------------------------------------------------------------------------------------| client1     | f735c245-ce35-4a08-89e0-0292bb053a9c |                     | 0                  || client2     | e36db52e-2624-4989-855a-28fa195f58e9 |                     | 0                  |-------------------------------------------------------------------------------------------------```4. get logs from server or clients using `cat server log.txt` or `cat client1 log.txt`  6 - Stop Training (if needed ) You could send signals to stop the training if you need to using:- `abort client`- `abort server`  7 - Cross-site validateOnce training is completed, you would like to get the validation matrices. See more information about cross-site validation the [Admin notebook](./Admin.ipynb).Run `validate all` to show the cross-site validation results. You could also run `validate source_site target_site` to see the performance of a certain model on a certain site.You should see something like ```> validate all{'client2': {'FL_global_model': {'validation': {'val_mean_dice': 0.11059250682592392, 'val_acc': 0.6894438906188668}}, 'client1': {'validation': {'val_mean_dice': 0.12774145603179932, 'val_acc': 0.7208915544895221}}, 'best_FL_global_model': {'validation': {'val_mean_dice': 0.08191516250371933, 'val_acc': 0.625326333805216}}, 'client2': {'validation': {'val_mean_dice': 0.12774255871772766, 'val_acc': 0.7208930085240025}}}, 'client1': {'FL_global_model': {'validation': {'val_mean_dice': 0.1105940118432045, 'val_acc': 0.6894455385246112}}, 'client1': {'validation': {'val_mean_dice': 0.12774255871772766, 'val_acc': 0.7208934608902853}}, 'best_FL_global_model': {'validation': {'val_mean_dice': 0.0819176733493805, 'val_acc': 0.62532652767648}}, 'client2': {'validation': {'val_mean_dice': 0.12774227559566498, 'val_acc': 0.720892297662701}}}}Done [23617 usecs] 2021-05-13 22:04:39.087597``` parsing this json and putting it in a table would look like  Client (val_mean_dice) |  FL_global_model | best_FL_global_model |  client1 |  client2:--- | :--- | :---: | :---: | --- client1  |       0.110594   |           0.081918 | 0.127743 | 0.127742client2  |       0.110593   |           0.081915 | 0.127741 | 0.127743  8 - Server securityTo illustrate that the server cannot decrypt the models saved on the server but the client can, we shall execute this small test script.Encrypted models can be loaded using `pickle`.<jupyter_code>import pickle

def load_enc_model(global_model_file):
    with open(global_model_file, "rb") as f:
        model = pickle.load(f)

    print("model:", list(model.keys()))
    
    encrypted_layers = model["he_encrypted_layers"] # holds a True/False boolean indicating whether the layer was encrypted
    model = model["model"]  # model state_dict holding the (partially) encrypted model weights

    count_encrypted_layers(encrypted_layers)    
    
    return model, encrypted_layers<jupyter_output><empty_output><jupyter_text>> **Note:** The server decryption is expected to fail because the current context of the tensor doesn't hold a secret_key (see output below).<jupyter_code>import numpy as np
import tenseal as ts
from nvflare.experimental.homomorphic_encryption.homomorphic_encrypt import count_encrypted_layers, load_tenseal_context

global_model_file = "/claraDevDay/FL/project1/server/run_1/mmar_server/models/best_FL_global_model.pt"
server_context_file = "/claraDevDay/FL/project1/server/startup/server_context.tenseal"
client_context_file = "/claraDevDay/FL/project1/client1/startup/client_context.tenseal"

# load the server and client TenSEAL context files
server_ts_ctx = load_tenseal_context(server_context_file)
client_ts_ctx = load_tenseal_context(client_context_file)

# load the global model saved on the server
model, encrypted_layers = load_enc_model(global_model_file)

# try decrypting the first encrypted layer
for layer_name in encrypted_layers:
    if encrypted_layers[layer_name]:
        print(f"{layer_name} is encrypted. Trying to decrypt...")
        print(type(model[layer_name]))
        
        try:
            # server can deserialize the bytes
            ckks_vector = ts.ckks_vector_from(server_ts_ctx, model[layer_name])

            # this is supposed to fail with the available server context as it doesn't hold a secret key!
            ckks_vector.decrypt()
        except Exception as e:
            print(f"Server decryption failed with: {e}!")
            pass
        
        # However, the client can decrypt using its own TenSEAL context
        ckks_vector = ts.ckks_vector_from(client_ts_ctx, model[layer_name])
        decrypted_params = ckks_vector.decrypt()
        
        print(f"Client decrypted parameters for {layer_name}:")
        np.set_printoptions(threshold=10) # don't show all values
        print(np.asarray(decrypted_params))
        
        break<jupyter_output>Loaded TenSEAL context from /claraDevDay/FL/project1/server/startup/server_context.tenseal
Loaded TenSEAL context from /claraDevDay/FL/project1/client1/startup/client_context.tenseal
model: ['model', 'train_conf', 'he_encrypted_layers']
3 of 63 layers are encrypted.
model.0.conv.unit0.conv.weight is encrypted. Trying to decrypt...
<class 'bytes'>
Server decryption failed with: the current context of the tensor doesn't hold a secret_key, please provide one as argument!
Client decrypted parameters for model.0.conv.unit0.conv.weight:
[ 0.07863831  0.15738822  0.12805362 ... -0.07288725  0.05486628
 -0.0626071 ]<jupyter_text>10 - Decrypting and validating the global modelsA client can decrypt the final global models using their TenSEAL context which olds the secret keys. First, decrypt the global model:<jupyter_code>import torch
from monai.networks.nets.unet import UNet

global_model_file = "/claraDevDay/FL/project1/server/run_1/mmar_server/models/best_FL_global_model.pt"
client_context_file = "/claraDevDay/FL/project1/client1/startup/client_context.tenseal"
dec_global_model_file = "/claraDevDay/FL/project1/client1/dec_best_FL_global_model.pt"

# load the client TenSEAL context files
client_ts_ctx = load_tenseal_context(client_context_file)

# load the global model saved on the server
model, encrypted_layers = load_enc_model(global_model_file)

# configure the model to get the model parameter shapes. Note, this is needed as the encrypted params are just vectors and don't hold the original shapes.
net = UNet(
  dimensions = 3,
  in_channels = 1,
  out_channels = 2,
  channels = [16, 32, 64, 128, 256],
  strides = [2, 2, 2, 2],
  num_res_units = 2    
)
dec_model = net.state_dict()

# decrypt encrypted layers and assign to the model
for layer_name in dec_model.keys():
    if encrypted_layers[layer_name]:
        # decrypt using client TenSEAL context
        ckks_vector = ts.ckks_vector_from(client_ts_ctx, model[layer_name])
        dec_params = ckks_vector.decrypt()
        print(f"[{np.min(dec_params)} to {np.max(dec_params)}]")
    else:
        # layer is not encrypted
        dec_params = model[layer_name]
    dec_model[layer_name] = torch.Tensor(np.reshape(dec_params, dec_model[layer_name].size()))
        

torch.save(dec_model, dec_global_model_file)
print(f"Saved decrypted model at {dec_global_model_file}")<jupyter_output>Loaded TenSEAL context from /claraDevDay/FL/project1/client1/startup/client_context.tenseal
model: ['model', 'train_conf', 'he_encrypted_layers']
3 of 63 layers are encrypted.
[-0.19118951273129894 to 0.19245263678326385]
[-0.025587799373183125 to 0.02559611276262976]
[-0.13731417125904205 to 0.13619750839840944]
Saved decrypted model at /claraDevDay/FL/project1/client1/dec_best_FL_global_model.pt<jupyter_text>Then, validate the decrypted global model:<jupyter_code>!python3 -u  -m medl.apps.evaluate \
    -m "/claraDevDay/FL/adminMMAR_HE" \
    -c "config/config_validation.json" \
    -e "config/environment.json" \
    --set \
    print_conf=True \
    use_gpu=True \
    multi_gpu=False \
    DATA_LIST_KEY="validation" \
    dont_load_ts_model=True \
    dont_load_ckpt_model=False \
    MMAR_CKPT="/claraDevDay/FL/project1/client1/dec_best_FL_global_model.pt" \
    MMAR_EVAL_OUTPUT_PATH="/claraDevDay/FL/project1/client1/eval"<jupyter_output>2021-05-18 18:52:04,150 - torch.distributed.nn.jit.instantiator - INFO - Created a temporary directory at /tmp/tmps8dahjst
2021-05-18 18:52:04,150 - torch.distributed.nn.jit.instantiator - INFO - Writing /tmp/tmps8dahjst/_remote_module_non_sriptable.py
========== Validate Config Result ===========
Use GPU:  True
Multi GPU:  False
Automatic Mixed Precision:  Enabled
Determinism Evaluation:  Disabled
cuDNN BenchMark:  False
CUDA Matmul Allow TF32:  True
cuDNN Allow TF32:  True
Model:  <class 'monai.networks.nets.unet.UNet'>
Dataset:  <class 'monai.data.dataset.Dataset'>
DataLoader:  <class 'monai.data.dataloader.DataLoader'>
Validate Transform #1: <class 'monai.transforms.io.dictionary.LoadImaged'>
Validate Transform #2: <class 'monai.transforms.spatial.dictionary.Spacingd'>
Validate Transform #3: <class 'monai.transforms.utility.dictionary.EnsureChannelFirstd'>
Validate Transform #4: <class 'monai.transforms.intensity.dictionary.ScaleIntensityRanged'>
Validate Transform #5: <class 'mona[...]
{'fraction_non_alphanumeric': 0.11453777710752526, 'fraction_numerical': 0.0773261951440205, 'mean_word_length': 3.9211502782931356, 'pattern_counts': {'":': 13, '<': 30, '<?xml version=': 0, '>': 47, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '234531', 'license': 'Apache-2.0', 'n_tokens_mistral': 9261, 'n_tokens_neox': 7955, 'n_words': 2427, 'path': 'PyTorch/NoteBooks/FL/Homomorphic_Encryption.ipynb', 'repo_name': 'visiont3lab/clara-train-examples', 'size': '26566'}