Federated ensemble distillation (FedD), an often even more communication-efficient and accurate alternative to FedAVG, uses knowledge distillation to transfer knowledge from clients to server [1]}, [2]}, [3]}, [4]}. In FedD, clients and server share a public dataset auxiliary to the clients' private data. The clients communicate the output of their privately trained models on the public distillation dataset to the server, which uses the average of these outputs as supervision for the distillation data in training the global model. In comparison to federated averaging, federated ensemble distillation offers additional privacy, as direct white box attacks are not possible for example, and allows combining different model architectures, making it appealing in an Internet-of-Things ecosystem [5]}, [6]}, [7]}.
