On-Sensor Binarized Fully Convolutional Neural Network with A Pixel Processor Array

This work presents a method to implement fully convolutional neural networks (FCNs) on Pixel Processor Array (PPA) sensors, and demonstrates coarse segmentation and object localisation tasks. We design and train binarized FCN for both binary weights and activations using batchnorm, group convolution, and learnable threshold for binarization, producing networks small enough to be embedded on the focal plane of the PPA, with limited local memory resources, and using parallel elementary add/subtract, shifting, and bit operations only. We demonstrate the first implementation of an FCN on a PPA device, performing three convolution layers entirely in the pixel-level processors. We use this architecture to demonstrate inference generating heat maps for object segmentation and localisation at over 280 FPS using the SCAMP-5 PPA vision chip.


Introduction
Fully convolutional neural networks (FCN) have been used across many modern computer vision tasks such as object detection [10], classification [15] and segmentation [19,26]. However, the deployment of deep FCN usually relies on powerful GPU/CPUs which are typically not present in emerging embedded edge devices, where cost and energy considerations dictate stringent limits on storage and computing resources. Despite this, there is an ever increasing demand for artificial intelligence on such edge devices. One promising approach to the edge computing hardware is represented by Pixel Processor Arrays (PPA). Unlike conventional vision systems, which consist of separate sensing and computing hardware, PPA devices are emerging vision architectures, integrating sensing, storage, and computing on a single silicon chip ( Fig.1) [4,14]. Such integration optimises data movements in the system, promising high performance and low-power consumption, but requires careful algorithm implementation, in order to efficiently utilise the hardware resources available in an on-sensor Text Text (a) (b) Figure 1: (a) Hardware architecture of the Pixel Processor Array (PPA) used in this work. The SCAMP-5 camera is based on a PPA vision chip with 256×256 Processing Elements (PE), which are simple programmable processor cores where parallel image processing is conducted by directly operating on analogue signals (e.g. electric current from photodetector PIX, which is proportional to the light intensity) within analogue registers (AREG) and bit operations within digital registers (DREG). (b) The ultra-compressed sensing capabilities with ×32,000 data reduction and a low operation power on PPA using the proposed FCN for 2D localisation.
computing device. This paper demonstrates how to implement and deploy binarized FCNs on PPA hardware. Networks with binary weights and activations are difficult to train and usually suffer from performance drop when compared to their floating-point equivalent. Use of batch normalisation has been proposed to avoid gradient explosion and train binarized neural networks successfully [30]. In this work we introduce batch normalisation into binarized FCNs to improve the training efficiency and the performance of inference on PPA arrays. We implement a purely binary convolutional network containing both binarized weights and activations. The use of binary activations alleviates accumulative errors introduced by approximate computations used to perform image convolutions upon PPA hardware devices [12]. This error mitigation allows our approach to perform deeper networks than previous work [2,3,21] wholly upon the focal plane without encountering an increasing loss of accuracy that would occur otherwise. Furthermore, it is noted that the implementation of the batch normalisation, the sign activation function, and learnable activation threshold for binarized activations is equivalent to adding a bias matrix to the layer activations, which significantly simplifies the inference process on sensor. With this binarized FCN, the inference calculation process can be implemented entirely with efficient add/subtract, threshold, and shifting operations for all layers. This scheme specifically benefits PPA computing devices such as the one shown in Fig. 1 because it matches the simple instruction set of these devices, and reduces the impact of calculation errors caused by noise accumulation when using analogue registers (AREG) for storage and calculations, especially for activations. For experiments, we train binarized FCN and deploy it on the PPA for object localisation and coarse segmentation. On-Sensor Computing and the SCAMP-5d Vision System: The concept of on-sensor computing originates from emerging novel circuit designs that enable direct signal processing on the sensing chip [33]. The SCAMP vision sensor [6] used in this work is based on a PPA concept implemented using mixed-signal analog/digital datapath (Fig. 1), other devices such as SONY IMX500 1 or Aistorm Mantis 2 integrate sensing and computing resources in a single device using alternative strategies. Our work takes advantages of the SCAMP analog/digital PPA to efficiently implement a binarized FCN. As shown in Fig. 1, each sensing element converts light into analogue signals that are immediately processed on the focal plane. Unlike the hardware architecture of standard computer vision systems, the PPA does not involve Analogue-Digital-Conversion (ADC) after sensing. Instead it directly operates on analogue electric currents, in physical proximity to the image sensor pixel circuits, accelerating the signal processing and avoiding the bottleneck of ADC and data transmission process to external processor units. However, such analogue processing introduces errors into the computation and data is prone to noise and temporal decay [12]. Fig. 1b shows the data extraction capabilities of the PPA based on the proposed FCN, effectively reducing images of thousands of pixels to specific contextual information within a tiny number of bits. The SCAMP vision sensor has been demonstrated in several applications in the field of robotics [17,18,22,28] and computer vision [1,5,27]. Contributions: The main contributions of this work are: 1) We propose, train, and demonstrate the use of a purely binarized network (both binary weights and activations) specifically for PPAs. This approach of binary activations addresses the accumulation of analogue computing errors and value saturation after each layer, thus enabling deeper networks while maintaining performance. 2) We present the first implementation of an FCN architecture for PPAs. Our approach uses group convolutional layers [31], and stores hundreds of convolutional filter weights upon the focal plane of the PPA. 3) Unlike earlier work, we apply batch normalisation during training and utilise this to learn bias parameters to be applied during inference on the PPA device. 4) We provide the first demonstration of object localisation and coarse segmentation tasks on a PPA, with previous works being only concerned with classification tasks. In addition, we created a simulated environment and datasets (to be available after publication) to support PPA developers and researchers for idea validation off and on-sensor.

Related Work
Recent work for PPAs has concentrated on CNNs and demonstrated on classification tasks [2,3,21]. However we found no previous PPA work on FCNs which are important for further tasks like localisation and segmentation. The research on CNN implementation and inference within PPA was pioneered by Bose et al. [2] where a CNN with a single convolutional layer was implemented upon the PPA array and a fully-connected layer upon its controller chip. Their work performs 16-bit image convolution operations using 4×4 DREG "Super Pixel" blocks and demonstrates live digit classification based on MNIST dataset at speed of around 200 frames per second (FPS). In their work, the ternary {-1, 0, 1} kernel filters are stored in the program memory, and are effectively encoded in the instructions/operations sent to the PPA array, performing convolutions sequentially. To fully take advantage of PPA's parallel computing characteristics and further improve the CNN inference efficiency, Bose et al. [3] proposed the idea of in-pixel weight storage, where the network's weights are directly stored within the registers of the PPA's processing elements. This enabled both parallel computation of multiple convolutions, and implementation of a fully connected layer upon the PPA array resulting in a ×22 faster CNN inference (4464 FPS) on the same digit recognition task. Based on these two works, Liu et al. [21] proposed a high-speed lightweight neural network using BinaryConnect [8] with a new method for computing convolutions upon the PPA, allowing for varying convolutional stride. Their work demonstrated four different classification tasks with frame rates ranging from 2,000 to 17,500 FPS with different stride setups. Based on their network, a direct servo control using real-time CNN inference results [23] and a simulated robot tracking from a drone [24] with on-sensor CNN computing results are presented. Moreover, AnalogNet [32] implements convolution with value approximation for multiplication on the focal plane using 3 kernels and fully-connected layer on the micro-controller based on the MNIST. AUKE [11] is developed to automatically generate convolution kernel code on the PPA by Debrunner et al. Our work advances state-of-theart in neural network implementation on PPA. We propose a new FCN network architecture suitable for PPA implementation, including three convolutional layers with binary weights and binary activations. The network utilises group convolution and is trained using batch normalisation, and enables new segmentation-related applications.

Method
Neural network architectures for PPAs must be carefully designed taking into account model size, architecture, and the feasibility of exploiting the PPA's parallel computation and onsensor storage. This is essential due to the limited on-sensor resources compared to standard computer hardware which may have access to powerful GPU/CPUs. This section attempts to find a balance between the neural network performance and its efficient implementation on the PPA.

CNN with Binary Weights and Activations
BinaryConnect [8] trains neural network with binary weights, however, it is not a fully binary neural network with floating-point neuron activations. The neural network in our work is based on Binarized CNN [9], with binary weights and neuron activations, which can be stored and processed with bit-wise operations. Compared to BinaryConnect, Binarized CNN reduces the intermediate memory storage for activations and replaces most arithmetic operations with bit-wise operations. Such fully binarized networks are thus highly suitable for PPAs due to their small memory footprint. For the forward propagation, we take different strategies to binarise the weights and activations to simplify the binarization process. All the weights are binarized with a deterministic function Eq.1 where w r is floating-point weights and w b is the binarized weights. In terms of activations, we train channel-wise learnable thresholds α to binarise the activations to obtain more informative binary feature maps, inspired by work [25]. Additional coefficients, introduced by channel-wise thresholds, have low impact on the implementation efficiency on the PPA. In Eq.2, α is the trainable thresholds for binarization of each channel, a r is the real-valued activations and a b is the binarized activations. During the training process, using standard backpropagation and stochastic gradient descent, the gradients are calculated with the floatingpoint weights. The weights and activations are only binarized during forward pass. In our work, the Binarized CNN is trained on a PC machine and the CNN inference process is implemented on the SCAMP-5d vision system.
The training process for batch norm parameters can be seen from [20] Algorithm 1, in which ε is used to avoid a zero denominator and the main scaling and shifting parameters γ and β for batch norm are learned during the training process. Then the batch norm can be applied to manipulate activations [20]. In Fig. 2   forward propagation process: Considering activation function tanh does not change the sign of inputs, we have: In Equation 6, where β , γ, and α are all trainable parameters that can be obtained directly after training. Thus the 'bias' B can be calculated used these parameters offline, before implementing it on the PPA. During the inference process, the batch norm and activation reduces to a bias term, as shown in Equation 5 B on Y. Hence, the inference process on the PPA can be simplified as shown in Fig. 2b. 3.2 FCN architecture on sensor FCN is a CNN architecture providing pixel-level classification, targeting image segmentation [26]. This paper proposes a 3-conv layer FCN that can be implemented on a PPA sensor. This work extends from previous CNN classifications, that were done using fullyconnected output layers [3,21]. In this paper, FCN is used for heat map generation by adding one convolutional layer with 128 filters and replacing the final fully-connected layer   with a convolutional layer of kernel size 1×1. Fig. 3 shows the overall FCN architecture with configurations for each layer. Fig. 6 illustrates the first convolution layer, generating 16 binary feature maps. The second layer adopts group image convolution [7] of 8 on the input 16 feature maps to make a trade off between convolution computation complexity and network performance, where each of 64 outputs is generated by adding two intermediate feature maps (Fig. 9). These 64 binary feature maps from the second convolutional layer are stored in 4 DREG. The third layer then generates the final heat map representing the prediction probability distribution, taking these 64 binary feature maps and combining them within an AREG. Each binary feature map being multiplied by an associated weight of -1/1.

FCN deployment on the PPA
This section gives the implementation detail of the binarized FCN on the PPA sensor hardware. First Layer: 16 binary filters are replicated to fill a DREG (Fig. 7) for parallel convolution purpose [3,21]. Fig. 5 shows the image convolution process on the PPA. The image convolution on the PPA can be decomposed as 'multiplications', shifting and addition and the convolution result is obtained by performing shifting and addition process for 16 times with a stride = 1. Then the pre-calculated bias B is plotted into 4×4 grids on a AREG and is subtracted from the feature map (seen in Fig. 6 ). Then the output binary image is obtained by binarizing feature map after subtracting B. In this layer, tanh is used as the activation function. When implementing inference on sensor, the tanh activation function is transformed into binarisation with a sign function, with offset computed from batch norm parameters as can be seen from Eq. 4. This layer shares some similarity with work [21] including the image resize, replication, and image convolution but with extra activation binarization process. Second Layer: The key to the second layer is to implement the group convolution with 16 feature maps as inputs and 64 as outputs. By dividing them into eight groups, thus there are 128 binary filters need to be stored on sensor. The layout of filters directly affects the inference efficiency. We design a storage structure for filters in first (Fig. 7) and second layer. Fig. 8 illustrates the layout of these filters within one DREG. As can be seen, each time to perform a convolution, the corresponding kernel filters are activated in parallel, shifted, and  replicated to fill each 64×64 block in 256×256 PEs. This filter storage structure can also extend to store more filters following the similar way to fill all PEs with 16 filters. More details can be seen from Section 6. In Fig. 9, to implement second convolution layer with 8 groups, the input 16 binary feature maps are first transformed by switching position of adjacent maps. This is followed by convolution with associated 32 filters for these 32 feature maps. Then 16 gray-scale feature maps are obtained by adding each two of 32 maps. By performing convolution for another 96 filters, 64 gray-scale feature maps can be derived. The bias matrix is subtracted and then after binarization, 64 binary feature maps are generated. Third Layer: In this layer, as shown in Fig. 3, 64 1-bit filters are plotted on a DREG, followed by 'multiplication' with the 1-bit feature maps from the previous layer. After 1 × 1 convolution, these 64 feature maps in 4 DREG is relocated to 1 AREG after 2 × 2 maxpooling. The summation of these extracted feature can be obtained by shifting and adding into    Fig. 4). Unlike in the previous two layers, the activation function for this layer is ReLU to generate a gray-scale feature map as the final prediction result of the network.

SCAMP-5 Inference, Experiments, and Evaluation
This section demonstrates the application of the proposed network architecture to coarse segmentation and object 2D localisation from a bird's eye view. We implement the FCN algorithm on the SCAMP vision system hardware (Fig. 10). We set up a realistic environment in Webots 3 [29] robot simulator (Fig. 11) for data collection and the validation of FCN deployment on sensor. Training, testing and validation datasets are collected by repeatedly taking images from a flying drone equipped with a simulated "SCAMP" and then validation images are sent to the PPA hardware for inference. Binarized FCN is trained offline based on these datasets with the method proposed in Section 3. The whole neural network for both coarse segmentation and localisation is performed on sensor. Fig. 11 shows the samples of collected datasets and their annotations for segmentation of road and grass. To validate the performance of the proposed network on different tasks, a Task  IoU  Road segmentation on simulation (Computer) 74.0% Road segmentation on sensor (PPA) 69.3% Grass segmentation on simulation (Computer) 76.6% Grass segmentation on sensor (PPA) 72.9% Table 1: Intersection over Union (IoU) performance comparison between simulation on computer and execution on-sensor for coarse segmentation. Figure 13: Training process comparison for grass segmentation with and without batch norm.   road and grass coarse segmentation is explored in this section. As shown in Fig. 11, we directly use the road/grass shape as the ground truth for coarse segmentation. Notice that the trees and grass areas often share similar gray-scale levels with the road, making coarse segmentation unfeasible by simply using binary thresholding. Tab. 1 shows the Intersection over Union (IoU) performance comparison between FCN inference on simulation and on sensor. Specifically, IoU is measured here by counting the number of intersected pixels over the number of united pixels of the predictions and groundtruth. In addition, Fig. 13 compares the FCN training process for segmentation task between using and not using batch norm, which shows the binarized FCN dose not converge without batch norm, justifying its use here. Some of the results can be seen from Fig. 12. Tab. 1 compares the experimental results on sensor and its counterpart baseline on computer with identical neural networks and validation images.

Object Detection
We also implemented an object detection task, based on the heat map. As for the object localisation, rather than using the vehicle segmentation image as the ground truth for training, we use Gaussian position distribution (Fig. 11) as the ground truth since the probability distribution is adequate to represent the object 2D localisation. For the validation, a distance threshold is set from 0 to 63 to count the number of predictions with a distance to the groundtruth that falls into this threshold. A zero distance means a perfect prediction. The final localisation is obtained by the weighted sum of all the possible positions. After the test, within a distance of 10 pixels, the vehicle localisation accuracy for simulation and SCAMP is around 88% and 83% respectively (Fig. 14). Tab. 2 shows the FCN performance in terms of time, power consumption and model size.

Pupil detection
Considering the light weight and low-power consumption of the PPA chip and the increasing popularity of virtual and augmented reality (VR/AR) based on the eye movement, it is promising to mount the PPA chip to a wearable device such as glasses in the near future, hence we explore the pupil detection with the proposed binarized FCN based on the public dataset of eye images: TEyeD [16]. Fig. 15 shows some of the training images and their annotations and Fig. 16 shows results comparison between SCAMP and simulation inference, where the accuracy curve is plotted according to the Euclidean distance between simulation/scamp inference results and the groundtruth. Within a distance of 10 pixels, the localisation accuracy for simulation and scamp is around 88% and 83% respectively. More experimental results can be seen from supplementary materials. Notice that there is around 5% -6% performance gap for the experiment on sensor compared to the simulation. This is due to noise in the convolution operation performed on AREG because of the inherent non-idealities of analog computation [13] and some random bit-flipping errors observed in DREG when performing massively parallel shifting and replications. Mitigation of these issues requires further software or hardware solutions. In this work, we tried to find a balance between network complexity and viability for deployment upon the available PPA prototype hardware. Pixel-wise accurate segmentation, with a quality equal to one that can be obtained using a CPU/GPUs hardware, using embedded low-power SCAMP-5d vision system, is still a challenging task with current hardware and neural network architecture.

Conclusion and Future Work
On-sensor computing is important for embedded and low-lag, low-power vision systems. Due to their compactness and computational advantages, binary FCNs are increasingly appealing. In this paper, we propose and implement a method that demonstrates carrying out an inference with a binarized FCN on an on-sensor computing device. In contrast to previous works that have mainly focused on classification with a fully-connected layer, we, for the first time, exploit the FCN architecture design, implementation method, and inference on sensor. We validate, using a real pixel processor array (PPA) hardware, on the visual competences of region segmentation and target object localisation with a latency of 3.5 milliseconds for each inference. With the development of future generation of on-sensor devices in terms of image resolution, manufacturing techniques, and local memory capacity, we believe our proposed binarized FCN can be extended with extra layers for more challenging applications.
Researches will be able to baseline against our work here and benefit from the code and data we will release after publication.