HyBNN and FedHyBNN: (Federated) Hybrid Binary Neural Networks
Binary Neural Networks (BNNs), neural networks with weights and activations
constrained to -1(0) and +1, are an alternative to deep neural networks which
offer faster training, lower memory consumption and lightweight models, ideal
for use in resource constrained devices while being able to utilize the
architecture of their deep neural network counterpart. However, the input
binarization step used in BNNs causes a severe accuracy loss. In this paper, we
introduce a novel hybrid neural network architecture, Hybrid Binary Neural
Network (HyBNN), consisting of a task-independent, general, full-precision
variational autoencoder with a binary latent space and a task specific binary
neural network that is able to greatly limit the accuracy loss due to input
binarization by using the full precision variational autoencoder as a feature
extractor. We use it to combine the state-of-the-art accuracy of deep neural
networks with the much faster training time, quicker test-time inference and
power efficiency of binary neural networks. We show that our proposed system is
able to very significantly outperform a vanilla binary neural network with
input binarization. We also introduce FedHyBNN, a highly communication
efficient federated counterpart to HyBNN and demonstrate that it is able to
reach the same accuracy as its non-federated equivalent. We make our source
code, experimental parameters and models available at:
https://anonymous.4open.science/r/HyBNN.