<jupyter_start><jupyter_text>Federated learning: Simple experiment using pretrained learning modelIn this notebook we provide a simple example of how to make an experiment of a federated environment with the help of this framework. We are going to use a popular dataset to start the experimentation in a federated environment. The framework provides some functions to load the [Emnist](https://www.nist.gov/itl/products-and-services/emnist-dataset) Digits dataset.<jupyter_code>import shfl

database = shfl.data_base.Emnist()
train_data, train_labels, test_data, test_labels = database.load_data()<jupyter_output><empty_output><jupyter_text>Let's inspect some properties of the loaded data.<jupyter_code>print(len(train_data))
print(len(test_data))
print(type(train_data[0]))
train_data[0].shape<jupyter_output><empty_output><jupyter_text>So, as we have seen, our dataset is composed of a set of matrices of 28 by 28. Before starting with the federated scenario, we can take a look at a sample in the training data.<jupyter_code>import matplotlib.pyplot as plt

plt.imshow(train_data[0])<jupyter_output><empty_output><jupyter_text>We are going to simulate a federated learning scenario with a set of client nodes containing private data, and a central server that will be responsible to coordinate the different clients. But, first of all, we have to simulate the data contained in every client. In order to do that, we are going to use the previously loaded dataset. The assumption in this example will be the data is distributed as a set of independent and identically distributed random variables, having every node approximately the same amount of data. There are a set of different possibilities in order to distribute the data. The distribution of the data is one of the factors that could impact more a federated algorithm. Therefore, the framework contains the implementation of some of the most common distributions that allow you to experiment different situations easily. In [Federated Sampling](./federated_learning_sampling.ipynb) you can dig into the options that the framework provides at the moment.<jupyter_code>iid_distribution = shfl.data_distribution.IidDataDistribution(database)
federated_data, test_data, test_label = iid_distribution.get_federated_data(num_nodes=20, percent=10)<jupyter_output><empty_output><jupyter_text>That's it! We have created federated data from the Emnist dataset using 20 nodes and 10 percent of the available data. This data is distributed to a set of data nodes in the form of private data. Let's learn a little more about the federated data.<jupyter_code>print(type(federated_data))
print(federated_data.num_nodes())
federated_data[0].private_data<jupyter_output><empty_output><jupyter_text>As we can see, private data in a node is not accesible directly but the framework provides mechanisms to use this data in a machine learning model. A federated learning algorithm is defined by a machine learning model locally deployed in each node that learns from the respective node’s private data and an aggregating mechanism to aggregate the different model parameters uploaded by the client nodes to a central node. In this example we will use a deep learning model using keras to build it. The framework provides classes to allow using Tensorflow (see [Basic Concepts Tensorflow](./federated_learning_basic_concepts_tensorflow.ipynb)) and Keras (see [Basic Concepts](./federated_learning_basic_concepts.ipynb)) models into a federated learning scenario, your job is only to create a function acting as model builder. Moreover, the framework provides classes to allow using pretrained Tensorflow and Keras models. In this example use a pretrained Keras learning model.<jupyter_code>import tensorflow as tf
#If you want execute in GPU, you must uncomment this two lines.
# physical_devices = tf.config.experimental.list_physical_devices('GPU')
# tf.config.experimental.set_memory_growth(physical_devices[0], True)

train_data = train_data.reshape(-1,28,28,1)

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu', strides=1, input_shape=(28, 28, 1)))
model.add(tf.keras.layers.MaxPooling2D(pool_size=2, strides=2, padding='valid'))
model.add(tf.keras.layers.Dropout(0.4))
model.add(tf.keras.layers.Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu', strides=1))
model.add(tf.keras.layers.MaxPooling2D(pool_size=2, strides=2, padding='valid'))
model.add(tf.keras.layers.Dropout(0.3))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation='relu'))
model.add(tf.keras.layers.Dropout(0.1))
model.add(tf.keras.layers.Dense(64, activation='relu'))
model.add(tf.keras.layers.Dense(10, activation='softmax'))

model.compile(optimizer="rmsprop", loss="categorical_crossentropy", metrics=["accuracy"])

model.fit(x=train_data, y=train_labels, batch_size=128, epochs=3, validation_split=0.2, 
                verbose=1, shuffle=False)
def model_builder():
    return shfl.model.DeepLearningModel(model=model)<jupyter_output><empty_output><jupyter_text>Now, the only piece missing is the aggregation operator. Nevertheless, the framework provides some aggregation operators that we can use. In the following piece of code we define the federated aggregation mechanism. Moreover, we define the federated goverment based on the keras learning model, the federated data and the aggregation mechanism.<jupyter_code>aggregator = shfl.federated_aggregator.FedAvgAggregator()
federated_government = shfl.federated_government.FederatedGovernment(model_builder, federated_data, aggregator)<jupyter_output><empty_output><jupyter_text>If you want to see all the aggregation operators you can check the following notebook [Federated Aggregation Operators](./federated_learning_basic_concepts_aggregation_operators.ipynb). Before running the algorithm, we want to apply a transformation to the data. A good practice is to define a federated operation that will ensure that the transformation is applied to the federated data in all the client nodes. We want to reshape the data, so we define the following FederatedTransformation.<jupyter_code>import numpy as np

class Reshape(shfl.private.FederatedTransformation):
    
    def apply(self, labeled_data):
        labeled_data.data = np.reshape(labeled_data.data, (labeled_data.data.shape[0], labeled_data.data.shape[1], labeled_data.data.shape[2],1))

class CastFloat(shfl.private.FederatedTransformation):
    
    def apply(self, labeled_data):
        labeled_data.data = labeled_data.data.astype(np.float32)
        
shfl.private.federated_operation.apply_federated_transformation(federated_data, Reshape())
shfl.private.federated_operation.apply_federated_transformation(federated_data, CastFloat())<jupyter_output><empty_output><jupyter_text>We are now ready to execute our federated learning algorithm.<jupyter_code>test_data = np.reshape(test_data, (test_data.shape[0], test_data.shape[1], test_data.shape[2],1))
test_data = test_data.astype(np.float32)
federated_government.run_rounds(2, test_data, test_label)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08021839563208735, 'fraction_numerical': 0.011059778804423912, 'mean_word_length': 6.543822597676875, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '81694', 'license': 'Apache-2.0', 'n_tokens_mistral': 2123, 'n_tokens_neox': 2012, 'n_words': 697, 'path': 'notebooks/federated_learning/federated_learning_basic_concepts_pretrained_model.ipynb', 'repo_name': 'joarreg/Sherpa.ai-Federated-Learning-Framework', 'size': '7143'}