Uncertainty is a very important feature of the intelligence and helps the brain become a flexible, creative and powerful intelligent system. The crossbar-based neuromorphic computing chips, in which the computing is mainly performed by analog circuits, have the uncertainty and can be used to imitate the brain. However, most of the current deep neural networks have not taken the uncertainty of the neuromorphic computing chip into consideration. Therefore, their performances on the neuromorphic computing chips are not as good as on the original platforms (CPUs/GPUs). In this work, we proposed the uncertainty adaptation training scheme (UATS) that tells the uncertainty to the neural network in the training process. The experimental results show that the neural networks can achieve comparable inference performances on the uncertain neuromorphic computing chip compared to the results on the original platforms, and much better than the performances without this training scheme. Uncertainty reasoning is the essence of human thinking activities and a key aspect of the intelligence. There are two kind of uncertainties in intelligent systems. One is the fuzziness, the other is the stochasticity. The fuzziness helps the brain deal with the real world efficiently by ignoring the enormous redundant information. When we try to distinguish a cat or a dog, we do not need to know the expressions and the number of the legs. Although such information can be easily captured by our visual system with a glance, it will be ignored for efficiency. The stochasticity endows the brain the creativity and enables us not always failed in an unfamiliar field. Our decisions may change when we do not sure. These characteristics are not available in most existing artificial intelligence (AI) systems, such as a classifier based on a deep neural network (DNN). The 32-bit or 64-bit floating numbers are used to describe the weights and activations. While some researchers found that the 8-bit integer is enough for many applications Banner et al. (2018) ; . Moreover, after the training procedure, the results will be the same no matter how many times it performs, although the margin is very small and the answer is wrong. There are some methods to address these issues, such as the network quantization and the Bayesian network. In addition, the neuromorphic computing chip has provide a hardware approach to supplement the missing uncertainty in DNN. In recent years, the emerging nanotechnology device and crossbar structure based neuromorphic computing chips have developed a lot Fuller et al. (2019) ; ; Yao et al. (2017) . The Ohms law and Kirchhoffs law make the crossbar structure very efficient when doing the vectormatrix multiplication (VMM), and the emerging nanoscale nonvolatile memory (NVM) device at each cross point provides additional storage capability (Figure 1 ). The crossbar holds the devices conductances as memory in peacetime, and performs the computing function when applied voltages. The so-called computing in memory (CIM) architecture can relieve the memory bottleneck, which is the most serious problem in the von Neumann architecture, and make the neuromorphic computing chips more energy and area efficiency. Therefore, the neuromorphic computing has become a promising approach to realize the AI applications, which is full of VMMs and great memory requirement. Besides the energy and area efficiency, the uncertainty is also an important and intrinsic feature of the neuromorphic computing chips and is not well utilized. Figure 1 : The crossbar structure. V is the applied voltage that correspond to the input x, G is the conductance of devices that correspond to the weight W, I is the output current, which can indicates the output y according to the Ohms law and Kirchhoffs law. The uncertainty in the neuromorphic computing chips comes from two aspects. The fuzziness is mainly caused by the analog to digital converters (ADCs) and the stochasticity is mainly induced by the NVM devices. According to the Kirchhoffs law, the VMM result is indicated as the summarization of the currents, which is an analog output. It is necessary to use the ADC to convert the analog currents to digital voltages for data transferring. The function of ADC is similar as the activation quantization in the network. The stochasticity of the NVM device is due to the intrinsic physical mechanism Zhao et al. (2017) ; Lin et al. (2018) . The random movement of the particles in the device makes the conductance varied. The output current will be different even applying the same voltage. The stochasticity of the device is usually simulated as a non-ideal factor that makes the network perform worse Prezioso et al. (2015) ; Ambrogio et al. (2018); Tang et al. (2017) . In this work, we proposed a training scheme that utilizes the stochasticity to improve the performance of the neuromorphic computing chips. The uncertainty is very important in the intelligent system. The Bayesian network is a very useful method to build an uncertain neural network. However, it usually requires that the distribution of each weight is controllable. This is hard to be realized by the neuromorphic computing chip due to the distribution is determined by the devices. Although there may be some methods to manipulate the conductance distribution of the device, it is not as convenient as UATS, which has no additional circuit required. We have tried a series of distributions to model the device stochasticity besides the Gaussian distribution, such as the Laplacian distribution, the uniform distribution, and the asymmetrical distributions, such as the lognormal distribution, the asymmetric Laplacian distribution, and the Bernoulli distribution for devices that have two stable states or the random telegraph noise (RTN). Although the modeled behavior of the device from different distributions is significantly different, the performance of network using each type of distribution with the same mean and variance is similar. It is because the VMM transform the individual distribution of each device to a summarization of a large number of random parameters.p The computation intension of UATS may be a little strong due to the requirement of a large number of random numbers. There are some methods to reduce the requirement of random numbers. Such as samples the weight for every input or every batch instead of the every VMM and using the uncertainty model of VMM results instead of the weights. The simulation speed can be accelerated and achieve similar results.
A training method that can make deep learning algorithms work better on neuromorphic computing chips with uncertainty
{"src": "scitldr-AIC_train", "paper_id": "Byekm0VtwS"}