We ran experiments on large-scale convolutional, ShuffleNet- [1]}, MobileNet- [2]}, and ResNet-style [3]} networks, using CIFAR-10 as local client data and both STL-10 and CIFAR-100 as auxiliary data. Of the auxiliary data, 80% is used for distillation and 20% for unsupervised pretraining. The pretraining is done by contrastive representation learning using the Adam optimizer with a learning rate of \(10^{-3}.\)
