We experiment with our knowledge transfer architecture and validate it performs well in several key areas. We make a comparison both with similar methods, as well as different configurations of a knowledge transfer network. We test a learning set-up where models spend one epoch learning local data, followed by one epoch of transferring knowledge from remote sources. Datasets used in our comparisons are CIFAR-10 [1]}, CelebA [2]} and FMNIST [3]}. Data is distributed non-i.i.d. between three agents. For CIFAR-10 and FMNIST, Local data comprises four out of the ten targets available in the datasets, and each of the remote source models is trained on three of the targets available. For CelebA, we focus on detecting smiling faces and distribute the data between the three agents without overlap.
To emulate a situation where a local model has minimal exposure to some targets and needs to acquire knowledge about these targets from a remote model, we add \(5\%\)  random data to the local data pool. This addition creates a slight overlap with training data to the remote models. For all our experiments and comparisons, we use Resnet-50 as our base model.
