<jupyter_start><jupyter_text>Federated Learning - MNIST Example  Populate remote GridNodes with labeled tensorsIn this notebbok, we will show how to populate a GridNode with labeled data, so it will be used later (link to second part) by people interested in train models.In particular, we will consider that two Data Owners (Alice & Bob) want to populate their nodes with some data from the well-known MNIST dataset.  0 - Previous setupComponents: - PyGrid Network      http://network:7000 - PyGrid Node Alice (http://alice:5000) - PyGrid Node Bob   (http://bob:5001)This tutorial assumes that these components are running in background. See [instructions](https://github.com/OpenMined/PyGrid/tree/dev/exampleshow-to-run-this-tutorial) for more details.  Import dependenciesHere we import core dependencies<jupyter_code>import syft as sy
from syft.grid.clients.data_centric_fl_client import DataCentricFLClient  # websocket client. It sends commands to the node servers

import torch
import torchvision
from torchvision import datasets, transforms

import requests<jupyter_output><empty_output><jupyter_text>Syft and client configurationNow we hook Torch and connect the clients to the servers<jupyter_code># address
alice_address = "http://alice:5000" 
bob_address   = "http://bob:5001"
hook = sy.TorchHook(torch)

# Connect direcly to grid nodes
compute_nodes = {}

compute_nodes["Alice"] = DataCentricFLClient(hook, alice_address)
compute_nodes["Bob"]   = DataCentricFLClient(hook, bob_address) 

# Check if they are connected
for key, value in compute_nodes.items(): 
    print("Is " + key + " connected?: " + str(value.ws.connected))<jupyter_output><empty_output><jupyter_text>1 - Load datasetDownload (and load) the MNIST dataset<jupyter_code>N_SAMPLES = 10000  # Number of samples
MNIST_PATH = './dataset'  # Path to save MNIST dataset

# Define a transformation.
transform = transforms.Compose([
                              transforms.ToTensor(),
                              transforms.Normalize((0.1307,), (0.3081,)),  #  mean and std 
                              ])

# Download and load MNIST dataset
trainset = datasets.MNIST(MNIST_PATH, download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=N_SAMPLES, shuffle=True)

dataiter = iter(trainloader)
images_train_mnist, labels_train_mnist = dataiter.next()  # Train images and their labels<jupyter_output><empty_output><jupyter_text>2 - Split datasetWe split our dataset ...<jupyter_code>images_train_mnist = torch.split(images_train_mnist, int(len(images_train_mnist) / len(compute_nodes)), dim=0 ) #tuple of chunks (dataset / number of nodes)
labels_train_mnist   = torch.split(labels_train_mnist, int(len(labels_train_mnist) / len(compute_nodes)), dim=0 )  #tuple of chunks (labels / number of nodes)<jupyter_output><empty_output><jupyter_text>... and we add tags to them so that we can search them later<jupyter_code>for index, _ in enumerate(compute_nodes):
        
    images_train_mnist[index]\
        .tag("#X", "#mnist", "#dataset")\
        .describe("The input datapoints to the MNIST dataset.") 
    
    
    labels_train_mnist[index]\
        .tag("#Y", "#mnist", "#dataset") \
        .describe("The input labels to the MNIST dataset.")<jupyter_output><empty_output><jupyter_text>3 - Sending our tensor to grid nodesWe can consider the previous steps as data preparation, i.e., in a more realistic scenario Alice and Bob would already have their data, so they just would need to load their tensors into their nodes.<jupyter_code>for index, key in enumerate(compute_nodes):
    
    print("Sending data to", key)
    
    images_train_mnist[index].send(compute_nodes[key], garbage_collect_data=False)
    labels_train_mnist[index].send(compute_nodes[key], garbage_collect_data=False)<jupyter_output><empty_output><jupyter_text>If everything is ok, tensors must be hosted in the nodes. GridNode have a specific endpoint to request what tensors are hosted. Let's check it!<jupyter_code>print("Alice's tags: ", requests.get(alice_address + "/data-centric/dataset-tags").json())
print("Bob's tags: ",   requests.get(bob_address   + "/data-centric/dataset-tags").json())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09851169383416017, 'fraction_numerical': 0.009685802031656036, 'mean_word_length': 4.744911804613297, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '155965', 'license': 'Apache-2.0', 'n_tokens_mistral': 1319, 'n_tokens_neox': 1254, 'n_words': 372, 'path': 'examples/data-centric/mnist/.ipynb_checkpoints/01-FL-mnist-populate-a-grid-node-checkpoint.ipynb', 'repo_name': 'hivecell-io/federated_learning', 'size': '4233'}