The computational complexity of leveraging deep neural networks for extracting deep feature representations is a significant barrier to its widespread adoption, particularly for use in embedded devices. One particularly promising strategy to addressing the complexity issue is the notion of evolutionary synthesis of deep neural networks, which was demonstrated to successfully produce highly efficient deep neural networks while retaining modeling performance. Here, we further extend upon the evolutionary synthesis strategy for achieving efficient feature extraction via the introduction of a stress-induced evolutionary synthesis framework, where stress signals are imposed upon the synapses of a deep neural network during training to induce stress and steer the synthesis process towards the production of more efficient deep neural networks over successive generations and improved model fidelity at a greater efficiency. The proposed stress-induced evolutionary synthesis approach is evaluated on a variety of different deep neural network architectures (LeNet5, AlexNet, and YOLOv2) on different tasks (object classification and object detection) to synthesize efficient StressedNets over multiple generations. Experimental results demonstrate the efficacy of the proposed framework to synthesize StressedNets with significant improvement in network architecture efficiency (e.g., 40x for AlexNet and 33x for YOLOv2) and speed improvements (e.g., 5.5x inference speed-up for YOLOv2 on an Nvidia Tegra X1 mobile processor).
In the area of methods tackling memory complexity, Lecun @cite_36 addressed this issue in their seminal paper by proposing the optimal brain damage method where synapses were pruned based on their strengths. They utilized the second-derivative information to specify the neuron to be pruned and made a trade-off between the number of parameters and training error. They formulated a new error function and the effect of perturbing the parameter vector was analytically calculated during the training. The main framework can be summarized as follow, the initial network architecture is chosen and the network is trained to obtain a reasonable performance. Based on the second derivative the saliency value for each parameter is computed and at then end, the set of parameters with low-saliency value are removed from the network model. The proposed approach took advantage of information theory to select non-important parameters in the model to be removed. The neural networks can be considered as a non-linear mapping between inputs and outputs where the network parameters extract the knowledge, therefore, different information theoretic methods can be applied in this area.
@cite_36
2114766824
We have used information-theoretic ideas to derive a class of practical and nearly optimal schemes for adapting the size of a neural network. By removing unimportant weights from a network, several improvements can be expected: better generalization, fewer training examples required, and improved speed of learning and or classification. The basic idea is to use second-derivative information to make a tradeoff between network complexity and training set error. Experiments confirm the usefulness of the methods on a real-world application.