TECHNICAL FIELD

The present disclosure generally relates to the field of artificial intelligence (AI) accelerators and machine learning (ML) and particularly to a system which reduces memory requirements of neural networks.

BACKGROUND

Neural networks are widely applied in the field of machine learning (ML) and artificial intelligence (AI). Neural networks usually consist of several layers. Each layer of a neural network consists of one or more cells (also called unit or neuron). Each cell's output (yi,jin equation (1)) is a function of the inputs to the cell as follows:
yi,j=ƒ(Σl[Wi,j,l×xi,j,l]+biasi,j)  (1)

in which, yi,j=the output of cell i of the layer j, Wi,j,l=the weight of input l of cell i of layer j, biasi,j=the bias of cell i of layer j, and ƒ(.)=an activation function (i.e., sigmoid function).

Hardware implementation of inference neural networks on a device requires receiving a set of inputs and reading a set of coefficients from the memory to calculate the outputs. As the number of cells or layers of a neural network increases (i.e., deep neural networks) the number of the weights and biases (Wi,j,land biasi,j) increases, which demands a large memory to store them. The storage of the coefficients increases system memory area and power, and the uploading cycles increase the dynamic power consumption of the system. For instance, in a system-on-chip (SoC) with a finite system memory and processing capacity, a deep neural network is processed in batches, which require frequent uploading of the coefficients that increases the dynamic power consumption, slows down processing an inference neural network, and reduces the system's throughputs. Therefore, techniques are required to reduce the necessary memory and uploading cycles.

SUMMARY

Novel methods and systems are disclosed to reduce the memory storage of neural networks by encoding the optimized coefficients of a trained neural network including weights and biases into a smaller number of encoded parameters and decoding them during the inference neural network. The present embodiment eliminates the need for saving the original optimized coefficients in the memory. Instead, the encoded parameters, which occupy less memory than the original optimized coefficients, are saved. The disclosed embodiment is comprised of a neural network coefficient decoder (NNCD), a genetic algorithm-based encoding system (GAbES), a coefficient encoding method (CEM), and a genetic algorithm-based neural network encoding/decoding (GANCED) system.

The encoding and decoding procedures of the disclosed embodiment both utilize the neural network coefficient decoder (NNCD). The NNCD of the disclosed embodiment comprises of (1) a feedback loop unit and (2) a pseudo random number generator. The pseudo random number generator is seeded by the feedback loop unit repeatedly for several cycles. The feedback loop unit's output (the seed of the pseudo random number generator) of the disclosed embodiment is designed to be a function of some parameters received as the input by the NNCD and the previous output of the pseudo random number generator. Therefore, given some input parameters, the NNCD of the present embodiment is capable of producing a greater number of outputs due to its feedback loop. Also, the outputs of the disclosed NNCD are pseudo independent numbers because they are produced by a pseudo random number generator. Therefore, by choosing the input parameters of the NNCD such that the outputs of the NNCD are good estimations of the optimized coefficients of a neural network, the enclosed embodiment eliminates the need for storing the original optimized coefficients. Instead, it is enough to store the parameters of the NNCD. One of the features of the disclosed embodiment that distinguishes it from the prior art is that since the outputs of the NNCD are pseudo independent, the neural network coefficients can be encoded to some smaller number of parameters regardless of their dependency. In other words, the disclosed embodiment allows a sequence of N independent (orthogonal) or dependent real numbers be estimated using R parameters where R<N with an acceptable accuracy.

To choose the right values for parameters of the NNCD which generates the desired outputs, the disclosed embodiment applies an optimization model which minimizes a measure of the error between the outputs of the NNCD and the optimized coefficients of the neural network with respect to the parameters of the NNCD. The genetic algorithm-based encoding system (GAbES) of the present embodiment applies an evolutionary algorithm so called genetic algorithm to solve the aforementioned optimization model and hence conducts the task of encoding by finding the correct values of the input parameters of the NNCD which provides high accuracy estimations of the optimized coefficients. Accordingly, the disclosed GAbES comprises of (1) a set of encoding engines each utilizes a genetic algorithm to map independent, original real numbers into a smaller number of encoded parameters and (2) a central unit which receives said sequence of the original, real numbers, distributes the task of the encoding among the encoding engines, collects the encoded parameters from the encoding engines, and outputs the decoded parameters.

The coefficient encoding method (CEM) of the present embodiment conducts repetitive encoding and decoding tasks with retraining of the neural network to improve the accuracy of the encoding. Since application of the estimated coefficients instead of the original optimized coefficients introduces some minor error to the inference neural network, the CEM of the disclosed embodiment iteratively applies the GAbES to partially encode the coefficients of one or more layers of the neural network and conducts retraining of the rest of the coefficients which were not yet encoded while considering the encoded coefficients non-trainable and equal to their estimated values. Also, the GAbES and the NNCD works with numbers in a preset range. Hence the CEM of the present embodiment also scales the original optimized coefficients to be in the preset range of the GAbES and rescales the outputs of the NNCD to the original range of the optimized coefficients to be appropriate for the inference neural network. Accordingly, the disclosed CEM comprises of (1) an initial training of a neural network, (2) selecting some of the optimized coefficients for encoding, (3) scaling the selected coefficients into a range acceptable by the GAbES if necessary, (4) encoding the selected optimized coefficients into a shorter length sequence of parameters applying the GAbES, (5) simulating the NNCD to predict the estimations of the scaled encoded coefficients, (6) rescaling the estimations of the encoded coefficients if necessary, (7) retraining the neural network for the rest of the coefficients which have not yet been encoded considering the already encoded coefficients as non-trainable constants equal to their estimated values, (8) checking if there is any coefficients remained to be encoded, and (9) storing the encoded parameters in a memory.

One advantage of the disclosed embodiment is that the NNCD design is totally hardware friendly and all of the relevant operations can be done by bitwise operations like bitwise AND, bitwise inclusive OR, and bitwise exclusive OR (XOR), which facilities incorporating one or more NNCD in an AI accelerator with little or no overhead. Hence, the genetic algorithm-based neural network encoding/decoding (GANCED) of the disclosed embodiment is a system for speeding up the neural network coefficient encoding and decoding tasks applying actual NNCD circuits which replaces the simulation of the NNCD in the firmware (i.e., using a central processing unit, CPU). Accordingly, the disclosed GANCED comprises of (1) an input/output unit, (2) a processor block which comprises of one or several processors, (3) a memory block, and (4) a neural network coefficient decoder block (NNCD block) comprises of one or several NNCDs. The required data, information and flow controls are received by the input/output unit of the GANCED. The processor block of the GANCED performs the computation and data processing operations required for running and training neural networks, reading and writing the memory, running a genetic algorithm, scaling, and any other operations required for the present embodiment to work correctly. In fact, the GANCED conducts the steps of the CEM while the processor block is responsible for all the tasks but simulating the NNCD. Instead, the NNCD block are utilized for the decoding task which allows parallel computation and helps speeding up the process.

The present embodiment does not rely on any relation between the coefficients. This feature distinguishes the disclosed embodiment from the prior arts. There are different methods devised to reduce the memory usage of neural networks. A recent non-patent publication studied the prior art methods including compression through quantization, network pruning, designing structural matrix, and tensor decomposition. Cheng, Y, Wang, D., Zhou, P., and Zhang, T, “a survey of model compression and acceleration for deep neural networks.”, IEEE Signal Processing Magazine, Special Issue on Deep Learning for Image Understanding,2020, pp. 1-10. One prior art method is to compress the coefficients before their storage, yet as the size of a network increases the compression complexity and circuit size increases, which may tradeoff the benefits. Tensorized network is another method. In this method, the coefficients of a neural network are represented as an N-way tensor and then decomposed using CANDECOMP/PARAFAC factorization, tucker decomposition, or tensor train (TT) decomposition or other variations. Tensor factorization is mainly useful for over-parametrized neural networks or, where there are dependencies between the coefficients that can be eliminated through factorization. Designing structural matrix is another technique, but its constraints may negatively affect the accuracy since it introduces bias to the network. In addition, finding an appropriate structural matrix is difficult. Network pruning is another technique in which the connections whose weights are close to zero are removed from the network. Most of the prior art methods use the dependencies of the coefficients or take advantage of other commonalities among coefficients like classifying, averaging, etc. In contrast, this discloser does not rely on any relation between the coefficients, which distinguish the present embodiment from the prior arts. Furthermore, the disclosed embodiment may be applied in conjunction with one or some other dimensionality reduction or compression techniques, where applicable.

