Datasets. We used benchmark datasets with the same train/test splits as in previous works [1]}, [2]} which are MNIST [3]}, CIFAR-10, CIFAR-100 [4]}, a subset of EMNIST [5]} (EMNIST-L), Shakespeare [6]} as well as a synthetic dataset. The IID split is generated by randomly assigning datapoints to the devices. The Dirichlet distribution is used on the label ratios to ensure uneven label distributions among devices for non-IID splits as in [7]}. For example, in MNIST, 100 device experiments, each device has about 5 and 3 classes that consume \(80\%\)  of local data at Dirichlet parameter settings of \(0.6\)  and \(0.3\)  respectively. To generate unbalanced data, we sample the number of datapoints from a lognormal distribution. Controlling the variance of lognormal distribution gives unbalanced data. For instance, in CIFAR-10, 100 device experiments, balanced and unbalanced data settings have standard deviation of device sample size of 0 and \(0.3\)  respectively.
