GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithm
Binary neural networks (BNNs) show promising utilization in cost and
power-restricted domains such as edge devices and mobile systems. This is due
to its significantly less computation and storage demand, but at the cost of
degraded performance. To close the accuracy gap, in this paper we propose to
add a complementary activation function (AF) ahead of the sign based
binarization, and rely on the genetic algorithm (GA) to automatically search
for the ideal AFs. These AFs can help extract extra information from the input
data in the forward pass, while allowing improved gradient approximation in the
backward pass. Fifteen novel AFs are identified through our GA-based search,
while most of them show improved performance (up to 2.54% on ImageNet) when
testing on different datasets and network models. Our method offers a novel
approach for designing general and application-specific BNN architecture. Our
code is available at http://github.com/flying-Yan/GAAF.