After the tremendous success of convolutional neural networks in image classification, object detection, speech recognition, etc., there is now rising demand for deployment of these compute-intensive ML models on tightly power constrained embedded and mobile systems at low cost as well as for pushing the throughput in data centers. This has triggered a wave of research towards specialized hardware accelerators. Their performance is often constrained by I O bandwidth and the energy consumption is dominated by I O transfers to off-chip memory. We introduce and evaluate a novel, hardware-friendly compression scheme for the feature maps present within convolutional neural networks. We show that an average compression ratio of 4.4× relative to uncompressed data and a gain of 60 over existing method can be achieved for ResNet-34 with a compression block requiring <300 bit of sequential cells and minimal combinational logic.
For this work, we build on a method known in the area of texture compression for GPUs, @cite_27 , fuse it with sparsity-focused compression methods, and evaluate the resulting compression algorithm on intermediate feature maps to show compression ratios of 4.4 and 2.8 for 8 ,bit AlexNet and SqueezeNet, respectively.
@cite_27
2516109628
State-of-the-art deep neural networks (DNNs) have hundreds of millions of connections and are both computationally and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources and power budgets. While custom hardware helps the computation, fetching weights from DRAM is two orders of magnitude more expensive than ALU operations, and dominates the required power. Previously proposed 'Deep Compression' makes it possible to fit large DNNs (AlexNet and VGGNet) fully in on-chip SRAM. This compression is achieved by pruning the redundant connections and having multiple connections share the same weight. We propose an energy efficient inference engine (EIE) that performs inference on this compressed network model and accelerates the resulting sparse matrix-vector multiplication with weight sharing. Going from DRAM to SRAM gives EIE 120 energy saving; Exploiting sparsity saves 10; Weight sharing gives 8; Skipping zero activations from ReLU saves another 3. Evaluated on nine DNN benchmarks, EIE is 189 and 13 faster when compared to CPU and GPU implementations of the same DNN without compression. EIE has a processing power of 102 GOPS working directly on a compressed network, corresponding to 3 TOPS on an uncompressed network, and processes FC layers of AlexNet at 1.88104 frames sec with a power dissipation of only 600mW. It is 24,000 and 3,400 more energy efficient than a CPU and GPU respectively. Compared with DaDianNao, EIE has 2.9, 19 and 3 better throughput, energy efficiency and area efficiency.