<jupyter_start><jupyter_text>Tensorcom Data ServerThis illustrates transmitting a training set via tensorcom. Here, we use a standard Torch `Dataloader` as a data source.<jupyter_code>import sys
import torch
from torchvision import datasets, transforms
import numpy as np
import tensorcom
loader = torch.utils.data.DataLoader(
    datasets.MNIST('.', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=32, shuffle=True)<jupyter_output><empty_output><jupyter_text>We use simple URLs with `zpub`, `zsub`, `zpush`, and `zpull` schemas for making ZMQ connections. There are also reverse versions `zr...`, which reverse the connect/bind schemes.Here we use a ZMQ PUB socket for distributing data. Such a socket will send data asynchronously, whether clients are connected or not.<jupyter_code>serve = tensorcom.Connection()
serve.connect("zpub://127.0.0.1:7888")<jupyter_output><empty_output><jupyter_text>In this sample library, all tensors are represented as NumPy arrays, so we have to convert the PyTorch tensors to NumPy before sending.For many application, sending floating point data in `float16` format is sufficient and potentially faster when networking is involved.<jupyter_code>for epoch in range(5):
    sys.stderr.write("{} ".format(epoch))
    for i, (xs, ys) in enumerate(loader):
        xs = np.array(xs).astype('float16')
        ys = np.array(ys).astype('int32')
        serve.send([xs, ys])<jupyter_output>0 1 2 3 4
{'fraction_non_alphanumeric': 0.10199004975124377, 'fraction_numerical': 0.021144278606965175, 'mean_word_length': 3.9968944099378882, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '385665', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 498, 'n_tokens_neox': 462, 'n_words': 156, 'path': 'docs/simple-server.ipynb', 'repo_name': 'Pexeso/tensorcom', 'size': '1608'}