BACKGROUND OF THE INVENTION

Deep computing frameworks, such as Convolutional Neural Networks (CNNs), have been used in many application areas, including pattern recognition, signal processing, time series analysis, and the like. CNNs require large amounts of computation involving a usually large number of parameters both during training and when the fully-trained network are deployed in the field. CNNs are deployed in mobile and embedded systems that interact with the real world. However, efficiency of CNNs that require such large amounts of computation and data may be limited by the power (e.g., battery), memory access bandwidth, and communication cost.

General-purpose processors may be programmable to perform complex calculations. However such processors may consume more power and perform operations at a lower speed. Graphical computing unit (GPU) may be configured to run faster than general-purpose processors; however, higher power consumption may be required. It would be helpful to have a method and system that satisfy the requirements for reduced latency and low power consumption.

SUMMARY OF THE INVENTION

Systems and methods are provided for data management so as to reduce memory accessing times and improve efficiency of computation. The computation and memory access pattern associated with CCNs are generally conducive to an optimized chip design. Therefore, the systems and methods can include implementing application of a CNN on a single chip. The data management may include arranging a plurality of items such as neural network parameters and input feature map in storage with respect to regions and slices and reducing amount of traffic between a main memory and such a chip having a RAM and a computing unit capable of performing parallel operations. According to one aspect of the present invention, a system for data management is provided. In practice, the method comprises: storing the plurality of items in a contiguous space within the memory, executing an instruction containing an address and a size that together identify the contiguous space to transmit the plurality of items from the main memory to a random-access memory (RAM) on a chip, and the chip includes a computing unit comprising a plurality of multipliers; and instructing the computing unit on the chip to: retrieve multiple of the plurality of items from the RAM; and perform a plurality of parallel operations using the plurality of multipliers with the multiple items to yield output data.

In some embodiments, the method further comprises determining the address in the main memory and the size before storing the plurality of items into the contiguous space. In some embodiments, the method further comprises issuing an instruction that allows the computing unit to obtain information regarding how the plurality of items are arranged in the contiguous space. In some cases, the method further comprises receiving information regarding a neural network wherein the plurality of groups correspond to a plurality of layers of the neural network and the plurality of items include parameters of the neural network, and where the plurality of parallel operations correspond to multiplications associated with one of the plurality of layers. In some cases, the information regarding the neural network includes, for one of the plurality of layers, a combination of a number of filters, a number of channels, and a filter size. In some cases, the method may further comprise: for the one layer, determining a number of slices based on the number of channels, that the contiguous space is divided into a plurality of regions, each region being contiguous; and dividing an area within one of the plurality of regions into at least the number of slices, each slice being contiguous, that the storing includes arranging items classified into one of the plurality of groups across the number of slices such that all items stored at the same offset from starting points of the number of slices are used for the parallel operations. In some cases, the method may further comprise: for the one layer, determining a number of slices based on the filter size, that the contiguous space is divided into a plurality of regions, each region being contiguous; and dividing an area within one of the plurality of regions into at least the number of slices, each slice being contiguous, that the storing includes arranging the items across the number of slices such that all items stored at the same offset from starting points of the number of slices are used for the parallel operations.

In some embodiments, the contiguous space in the main memory for storing the data is divided into a plurality of regions, each region being contiguous, and that the storing further comprises arranging items classified into multiple of the plurality of groups in the same region to reduce unused space in the contiguous space. In some cases, the method further comprises dividing an area within one of the plurality of regions into a plurality of slices, each slice being contiguous, the area is allocated to the items classified into the one group, the storing further comprises arranging the items classified into the one group across the plurality of slices, and the instructing includes performing multiple of the plurality of parallel operations using multiple items stored at the same offset from starting points of the plurality of slices. In some embodiments, a second contiguous space in the RAM having the substantially same size as the contiguous space in the main memory is divided into regions and slices in the same manner as the contiguous space, and the plurality of items are arranged in the second contiguous space in substantially the same manner as in the contiguous space, further comprising issuing an instruction that allows the computing to obtain information regarding, for one of the plurality of groups, a starting point of an area in the second contiguous space where items classified into the one group are stored and a starting point of each of a plurality of slices in the area.

In another aspect of the invention, a system for arranging data to accelerate deep computing is provided. The system comprises: one or more processors; and one or more memories having instructions stored thereon which when executed by the one or more processors cause the one or more processors to perform: storing the plurality of items in a contiguous space within a main memory, issuing an instruction to transmit the plurality of items from the main memory to a random-access memory (RAM) on a chip, wherein the instruction contains an address and a size that together identify the contiguous space, and wherein the chip includes a computing unit comprising a plurality of multipliers; and instructing the computing unit on the chip to: retrieve multiple of the plurality of items from the RAM; and perform a plurality of parallel operations using the plurality of multipliers with the multiple items to yield output data.

In another aspect, a data management method is provided. The method comprises: receiving, with aid of one or more processors, data regarding a plurality of objects, each containing a group of three-dimensional numerical arrays; allocating a space in a main memory to the plurality of objects, wherein the space includes a plurality of regions; assigning an area within one of the plurality of regions to one of the plurality of objects; determining a number of slices for the one object based on a size of the group and dimensions of the three-dimensional numerical arrays contained in the one object; dividing the area into at least the number of slices for the one object; and storing numerical items in the three-dimensional arrays contained in the one object across the number of slices such that at least one numerical item is stored in each of the number of slices. In some embodiments, the method further comprises: determining a starting address and a size of the space before storing the numbers into the area.

In some embodiments, the method further comprises issuing an instruction to transmit content of the space in the main memory to a second space in a random-access memory (RAM) on a chip, wherein the instruction includes a starting address and a size of the space, and wherein the chip includes a computing unit comprising a plurality of multipliers. In some cases, the second space in the RAM has substantially the same size as the space in the main memory and is divided into regions and slices in substantially the same manner as the space in the main memory, and the transmitting includes arranging the content in the second space in substantially the same manner as in the space in the main memory. In some cases, the method further comprises issuing an instruction to: retrieve multiple numerical items from the same offset from a plurality of predetermined addresses in the RAM; and perform a plurality of parallel operations using the plurality of multipliers with the multiple numerical items to yield output data. In some cases, the plurality of predetermined addresses is beginnings of a plurality of the slices. In some cases, the plurality of objects corresponds to a plurality of layers of a neural network and each three-dimensional numerical array corresponds to a filter. In some cases, the method further includes assigning a second area within the one region to a second one of the plurality of objects.

In a separate yet related aspect, a non-transitory computer-readable storage medium with instructions stored thereon that is provided. The instructions when executed by a computing system, cause the computing system to perform a method of arranging data to accelerate deep computing, the method comprising: receiving, with aid of one or more processors, data regarding a plurality of objects, each containing a group of three-dimensional numerical arrays; allocating a space in a main memory to the plurality of objects, wherein the space includes a plurality of regions; assigning an area within one of the plurality of regions to one of the plurality of objects; determining a number of slices for the one object based on a size of the group and dimensions of the three-dimensional numerical arrays contained in the one object; dividing the area into at least the number of slices for the one object; and storing numerical items in the three-dimensional arrays contained in the one object across the number of slices such that at least one numerical item is stored in each of the number of slices.

In another aspect, an integrated circuit on a chip is provided. The integrated circuit on a chip comprises: a random-access memory (RAM) configured to store a set of items in a contiguous space; and a computing unit, comprising a plurality of multipliers configured to perform parallel operations in response to one or more instructions; and a plurality of multiplexers, wherein a subset of the plurality of multiplexers are connected with [input pins of] the plurality of multipliers, wherein the subset of multiplexers are programmed to transmit a subset of the set of items to the plurality of multipliers for the parallel operations using one offset from a plurality of predetermined addresses. In some cases, the chip utilized in the method includes a computing unit comprising a plurality of multipliers.

In some embodiments, the parallel operations are performed for multiple rounds to form an iterative process, and the parallel operations in a current round lead to a set of values to be processed in a next round of the iterative process. In some embodiments, the subset of multiplexers included in the chip receive a first set of control signals based on one of the one or more instructions, and wherein the first set of control signals determines one of a plurality of predetermined routes for transmitting the subset of items to the plurality of multipliers.

In some embodiments, the computing unit further comprises a plurality of adders connected with the plurality of multipliers. In some cases, the computing unit further comprises a second plurality of adders connected with the plurality of adders. In some cases, the plurality of adders is accumulators. In some cases, the computing unit further comprises a second subset of the plurality of multiplexers are connected with the plurality of adders, and wherein the second subset of multiplexers are configured to receive a second set of control signals based on one of the one or more instructions, and wherein the second subset of control signals determines one of a plurality of predetermined options for enabling accumulation in one or more of the plurality of adders. In some cases, a second subset of the plurality of multiplexers are programmed to receive items produced by the plurality of adders and transmit them to the RAM, the second subset of multiplexers are configured to receive a second set of control signals based on one of the one or more instructions, and the second set of control signals determines one of a second plurality of predetermine routes for transmitting items produced by the plurality of adders to a second offset from a second plurality of predetermined addresses in the RAM. In some embodiments, the computing unit further comprises a plurality of rectifiers connected with the plurality of adders. In some cases, a second subset of the plurality of multiplexers are connected with the plurality of rectifiers, the second subset of multiplexers are configured to receive a second set of control signals based on one of the one or more instructions, and the second set of control signals determines one of a plurality of predetermined options for enabling one or more of the plurality of rectifiers. In some cases, when the second set of control signals have predetermined values, at least one of the plurality of rectifiers is always disabled or periodically disabled. In some cases, a second subset of the plurality of multiplexers are programmed to receive items produced by the plurality of rectifiers and transmit them to the RAM, and the second subset of multiplexers are configured to receive a second set of control signals based on one of the one or more instructions, and the second set of control signals further determines one of a second plurality of predetermine routes for transmitting items produced by the rectifiers to a second offset from a second plurality of predetermined addresses in the RAM. In some cases, the RAM includes a plurality of regions, and the first set of control signals selects one of the plurality regions where the subset of items is stored. In some cases, the one region includes a plurality of consecutive, equal-sized slices, and the plurality of predetermined addresses corresponds to beginnings of the plurality of slices. In some cases, the subset of multiplexers are configured to receive a first set of control signals based on one of the one or more instructions, and the first set of control signals determines a value of the offset in the current round. In some cases, the value of the offset in the current round is incremented by a fixed amount, such as four in the next round.

In some embodiments, the chip further comprises a functional unit for expanding or contracting the set of values yielded in the current round. In some embodiments, the subset of the set of items processed by the chip corresponds to a plurality of pixels of an input feature map, and wherein the parallel operations performed in the current round on the chip corresponds to all multiplications required in applying one parameter of all filters for all channels in one layer of a convolutional neural network to the plurality of pixels of the input feature map. In some embodiments, the size of the plurality of multipliers is a power of two, such as 128. In some embodiments, the chip further comprises an instruction decoder to decode the one or more instructions and generate control signals.

It shall be understood that different aspects of the invention can be appreciated individually, collectively, or in combination with each other. Various aspects of the invention described herein may be applied to any of the particular applications set forth below.

Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in this art from the following detailed description, wherein only exemplary embodiments of the present disclosure are shown and described, simply by way of illustration of the best mode contemplated for carrying out the present disclosure. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference.

