An Efficient OpenCL-Based FPGA Accelerator for MobileNet

Convolutional neural network is widely used in image processing and image recognition. In order to obtain higher accuracy, the computational complexity and the scale of model and parameters are increasing. FPGA has become a good choice because of its low power consumption and high flexibility. MobileNet replaces the standard convolution by depthwise convolution and pointwise convolution, which greatly reduces the computational complexity and parameters of the model in the case of less precision loss, so that it can be applied to the equipment with limited computing resources. In this paper, we propose an efficient OpenCL-based FPGA CNN accelerator to realize inferencing acceleration of MobileNet. We designed the convolution layer with modularization. We used pipeline to design a depth separable convolution parallel acceleration scheme, and made full use of the DSP resources of FPGA. This design finally achieved a good balance of hardware resources, processing speed and power consumption. The experiments show that the accelerator can reach the inferencing speed of 32.56ms, and the energy consumption is 20W. The speedup is 4x compared with CPU and energy efficiency is 3x compared with GPU.


Introduction
In recent years, with the expansion of dataset and the innovation of neural network structure, neural network has achieved higher accuracy and performance [1] [2]. However, in order to achieve higher accuracy, the neural network model becomes more and more complex, up to hundreds of layers; the amount of data used for training is also more and more huge, reaching billions (10gflops) [3]. In practical use, efficiency and cost are also factors that must be considered, so the requirements for the computing power and energy consumption of neural network are becoming higher and higher [4] [5].
FPGA has the characteristics of high flexibility, programmability and low power consumption. It is usually used to replace GPU as an efficient accelerator for deep neural network. [6] implemented the corresponding acceleration based on OpenCL by using efficient architecture and pipelined kernels to implement large CNN networks. [7] LACS reduces the negative impact on computing efficiency caused by invalid operations. Similar to the CPU computing mode, it reduces the amount of data and movement between the register and in-chip buffer BRAM. However, the size and number of large CNNs are still a factor influencing the inferencing speed. Researchers have also started to study the acceleration of small neural networks such as MobileNet [8], ShuffleNet [9], GhostNet [10] on the FPGA. [11] proposed redundancy reduced MobileNet, which saves the parameters in FPGA on-chip memory, the parameters are reduced and the accuracy is improved. [12]  proposed a scalable high performance DSC optimized CNN accelerator. They processed the computing tasks of different layers in one unit, used hierarchical storage to reduce the on-chip memory limit.
However, using hardware description language to develop FPGA accelerator is very complex. Programmers can use high-level language to develop without understanding hardware circuit. We designed a convolution layer with modularization, designed configurable kernel programs to speed up the different layers. We designed the convolution layer with modularization, and completed the modular implementation of depthwise convolution(DC) and pointwise convolution(PC). We used pipeline to design a depth separable convolution parallel acceleration scheme, and made full use of the DSP resources of FPGA. This design finally achieved a good balance of hardware resources, processing speed and power consumption. In the final experiment, we classified the pictures in ImageNet on Inspur_F10A. The FPGA design achieves 4x speedup if compared to CPU and 3x less energy consumption than GPU.

MobileNet model architecture
MobileNet is a lightweight CNN, which reduces the size and number of models, while ensuring the accuracy, so that it can be implemented under resource-constrained conditions. The main difference between MobileNet and VGG is that the standard convolution (SC) is replaced by the DSC. As shown in Figure 2, MobileNet decomposes a 3*3 SC layer into 3*3 DC and 1*1 PC, with BN batch normalization and ReLU activation processing behind each section [14].

MobileNet Performance Analysis
For standard convolution layers, there are N convolution cores of size K K D D M   , and the number of operations for each convolution core is W H D D  . The computation of standard convolution is: The computation of DSC includes two parts: the computation of DC and PC. The convolution kernel size of depth convolution kernel is K K D D M   , each convolution kernel needs W H D D  times multiplication and addition; PC has N convolution kernels of 1 1 M   , and each convolution kernel needs W H D D  times multiplication and addition. Therefore, the computation for DSC is as follows: Compared with the standard convolution and the depth separable convolution, the results are as follows: It can be seen that by using depth separable convolution, the number of parameters and computation of MobileNet can be reduced to 1/8 to 1/9 of the standard convolution (when using 3*3 convolution kernel).

OpenCL
OpenCL is a standard API and programming language for parallel operations on heterogeneous computing devices. It is a heterogeneous computing language based on C language and can run on accelerator such as CPU, GPU and FPGA. Compared with HDL hardware description language (Verilog, VHDL) FPGA design process, OpenCL programmers can develop high-performance applications in a higher level to design abstraction environment without understanding the hardware circuit details, which can make it put on the market timely. Figure 3 shows the development process on FPGA using OpenCL. In the process of FPGA development, it is divided into host program and kernel program. Kernel programs need to take full account of the parallel computing power of the FPGA. The computing task is written as a module into the kernel, which will be compiled as hardware logic circuits on the development board of the FPGA. The host program runs on the CPU and is responsible for assigning different kernel tasks to the FPGA according to the algorithm logic that is written.

Accelerator design
We use OpenCL to design mobile net based on FPGA, which is composed of a series of OpenCL kernel programs. According to the structure of MobileNet, we divide all the kernel modules into two types: Conv3D and Conv2D. Conv3D includes SC, PC and full join layer operation; Conv2D includes DC and ReLU. In the process of operation, we adjust the data to be transmitted, and use channel pipe to realize data transmission between the kernels. Through the reuse analysis of parameters and weights in DSC, we share memory data to reduce the transmission between on-chip storage and external storage, so as to reduce the time of memory access and data exchange and improve the efficiency of memory utilization.

Kernels design
3.1.1. Conv2D. In this group of kernels, the kernel performs DC (2D) and average pooled ReLU operations on the data in memory according to the configuration of the host program. The DC splits the convolution kernel of SC into single channel mode. We do not change the depth of the input feature image, but we do convolution for each channel, and then we can achieve the same number of channels of output feature image and input feature image. Because the dimension and channel number of the output feature map after DC are less than that of the standard convolution, the output feature information in the volume and operation will be less, so we need to use PC to increase and reduce the dimension of the feature map after DC, so as to obtain the same feature information as the SC.
We import the input feature image data from memory into the DC kernel, the output of each channel is obtained by multiplying and accumulating the input features and the same number of weights in the feature graph. In the process of DC, each different channel performs 2D calculation on the input image, and the calculation of each channel has no data exchange. We parallelize the convolution process of different channels to increase the utilization of FPGA on-chip resources and improve the computational efficiency. The figure shows the architecture of Conv2D kernel group.

Conv3D
. Different input channels are used to calculate the data of different layers, and different computing tasks are configured through the CPU. The pointwise convolution has high parallelism in convolution. Each different output channel and each pixel operation are independent. We make the operation of PC highly parallel, so as to speed up the model inferencing. In the kernel program of PC, we adjust the parallelism of PC by adjusting the number of convolution kernels that can be calculated at the same time. By using the loop expansion method, #pragma unroll in OpenCL, we optimize the original six layers 3D convolution kernel and change it into two layers, which reduces the number of loops and increases the efficiency of operation. The pseudo code of the kernel module is as follows: The calculation logic of standard convolution and pointwise convolution is similar. Setting the channel number of pointwise convolution to K is standard convolution. The same computing logic applies to the fully connected layer in this kind of kernel. After decomposing the weight matrix, we can convert the calculation of this layer into matrix multiplication and output the results separately. In the Conv3d kernel group, all three kernels can form the addition operation of one-dimensional vectors after logical decomposition. By adjusting the size of these one-dimensional vectors and the frequency of iteration, we can reuse the resources in memory, reduce data exchange, and increase the efficiency of model inferencing by increasing the resource utilization of the FPGA.

Data transfer design
In the process of inferencing on the FPGA, the in-chip memory will exchange data with the external memory several times, while frequent data exchange will affect the speed and cause a lot of energy consumption. We redesigned the kernels of read and write operations by adding data reuse and adjusting the logic of data access to achieve high-efficiency data transmission. We adjust the data structure that needs to be sent from the host to the device for operation, and use channel pipes for transmission, reducing the frequency of data transmission. FPGA has limited memory, so it needs to use external memory to save and exchange a large number of parameters. When the convolution Stride and the edge length of the convolution kernel K D are in the convolution operation, the convolution kernel will slide on the input feature map. There are ( ) parameters that can be reused, and we can store them as buffers to reuse data. The weight data of each layer in Conv2D is K D N  . The memory of the FPGA can be stored completely. We can convolute in different channels and extract the output of parameter calculation. Conv3D has a very large number of weight parameters that are reused. Three dimensional convolution has more channels than 2D convolution and a large number of parameters. We use buffered weight stream to batch import data, and make full use of each read data. Experimental results

Experiment setup
We use Inspur_F10A FPGA development board to complete the design. We use the Intel OpenCL Pro 19.1 SDK on the development board. Table 1 shows the hardware information.

Resource utilization analysis
The design is experimented with in Inspur_F10A platform, after summarizing the resources occupied by each module, the final resource usage is shown in Table 2. The same MobileNet neural network model is implemented on CPU and GPU respectively to compare with the accelerator we designed for FPGA. We do inferencing on CPU, GPU and FPGA respectively, and compare the acceleration effect and power consumption in different ways. As shown in Table 3, the order of speed from fast to slow is CPU (1x), FPGA (4x), GPU (13x). The speed of neural network running on the FPGA is better than that on the CPU, GPU consumes 3x as much energy to process a single image as the field programmer. Combining acceleration efficiency and energy consumption, the FPGA accelerator has better performance and lower energy consumption.  Energy Efficiency / 1x 3x

Conclusion
In this paper, MobileNet is studied and an efficient lightweight CNN accelerator is proposed. MobileNet is implemented for inferencing acceleration based on the OpenCL on the FPGA. We implement the depthwise separable convolution module and design a pipeline parallel acceleration scheme. Our experimental results show that the accelerator achieves 4x of speedup compared with CPU and 3x of energy efficiency and compared with GPU in the experiment of image classification in ImageNet. It makes full use of the DSP resources available in FPGA board, achieves a better balance in hardware resources, speed and power consumption, and fully demonstrates the performance and energy consumption advantages of FPGA in the implementation of lightweight neural network.