BACKGROUND

In a typical neural network, a standard computation is a dot product between input values (activations) and weight values. A typical way for an integrated circuit to compute these weight values is to use multiply-accumulate (MAC) circuits that repeatedly perform the multiplication of an input value by a weight value, adds that to an existing partial dot product, and stores the new partial dot product. However, this requires numerous clock cycles, as each term in the dot product computed by a MAC uses a separate cycle. In addition, the storage of each intermediate term requires the use of memory (contributing to both slowing down of the computation and use of resources for the read/write operations). Accordingly, techniques for parallelization without massively expanding the surface area of the circuit are required.

BRIEF SUMMARY

Some embodiments provide an integrated circuit (IC) for implementing a machine-trained network (e.g., a neural network). The IC of some embodiments includes a set of input processing circuits, a neural network computation fabric that can be configured to apply a neural network to an input value, and a microprocessor (e.g., for controlling the input processing circuits and configuring the neural network computation fabric). The neural network computation fabric of some embodiments includes (i) a set of cores that compute dot products of input values and corresponding weight values and (ii) a channel that aggregates these dot products and performs post-processing operations (as well as performs other operations), in order to compute the outputs of neural network computation nodes.

In some embodiments, at startup of the IC, the microprocessor loads neural network configuration data (e.g., weight values, scale and bias parameters, etc.) from off-chip storage and generates instructions for the neural network computation fabric to write the neural network parameters to memory. In addition, microprocessor loads the neural network program instructions for the computation fabric to its own memory. These instructions are applied by the computation fabric to input data (e.g., images, audio clips, etc.) in order to execute the neural network. The instructions include, e.g., the memory locations to which input values are written, configuration data specifying how to compute specific neural network nodes, etc. Upon receiving input data (e.g., from a sensor on a device that incorporates the IC), the microprocessor provides neural network program instructions to the computation fabric. Once the final output of the neural network is computed, the fabric provides this output back to the microprocessor, so that the microprocessor (or other circuitry on the device) can evaluate this output and perform any actions based on the output.

The microprocessor executes a controller, in some embodiments, that provides the neural network instructions to the computation fabric. Some embodiments provide these instructions to the computation fabric incrementally. For instance, in some embodiments, the system controller on the microprocessor initially loads the instructions for the first layer (or a first portion of the first layer) of the neural network, then waits for a signal from the fabric indicating that these instructions have been completed. Once the first portion of the network is completed by the fabric, the system controller provides the fabric with the instructions for the second portion (either a second portion of the first layer, or the second layer of the network), and so on until the network has been fully executed.

As mentioned, the neural network computation fabric includes numerous cores as well as a global channel that connects the cores, with the various data processing circuits configured by the hierarchical set of control circuits. These data processing circuits operate to compute neural network operations in an efficient, low-power manner, according to the configuration data provided by the control circuits.

A typical neural network operates in layers, with each layer including numerous nodes. Examples of neural networks include feed-forward neural networks, regulatory feedback networks, radial basis function networks, recurrent networks, etc. In convolutional neural networks (a type of feed-forward network), a majority of the layers include computation nodes with both a linear function followed by a non-linear activation function (applied to the result of the linear function). The linear function is a dot product of input values (either the initial inputs based on the input data for the first layer, or outputs of the previous layer for subsequent layers) and predetermined (trained) weight values, along with bias (addition) and scale (multiplication) terms, which are also predetermined based on training. As such, for convolutional neural networks, the dot products are the primary computation that uses the most circuit resources.

The neural network computation circuit of some embodiments computes numerous neural network nodes simultaneously, with the computation for one node spread across multiple cores (and subsequently the global channel). That is, each of several cores of the computation fabric computes a partial dot product from a subset of the input values and weight values for the node. In some embodiments, a set of input values are used as the input to multiple nodes in a layer, so a core simultaneously computes the dot products of these input values with multiple sets of weight values. Similarly, a set of weight values (referred to as a filter, or filter slice when that filter is divided across multiple cores) are used as the weights for numerous nodes with different sets of input values, so in some embodiments the cores load sets of weight values once and then compute dot products of these weight values with numerous different sets of input values.

For a dot product computed across more than one core, these multiple cores compute partial dot products and provide these partial dot products to the global channel. In the simplest case, all of the partial dot products for a given computation node are computed in the same clock cycle and provided at the same time to the global channel. In some cases, however (e.g., for dot products with a very large number of terms), each core computes more than one partial dot product, requiring multiple clock cycles. Based on configuration data specifying which outputs from the cores are to be added together (and whether multiple partial dot products are required from the same core), the global channel aggregates these partial dot products to compute the complete dot product for each node, then applies various post-processing functions (e.g., the bias, scale, and non-linear activation functions) to compute the output of each node.

In some embodiments, each segment of the global channel includes (in addition to the cluster controllers) a dot product bus, a set of post-processing circuits, and an output bus. The dot product bus, in some embodiments, includes a number of independent dot product bus lanes that each receives partial dot products from the cores, aggregates these dot products together, and provides the aggregated dot products to the post-processing circuits. In some embodiments, configuration data from the cluster controllers specifies to which post-processing unit each aggregated dot product is sent. Each lane of the dot product bus spans all of the channel segments, each of which aggregates the partial dot products from its own cores. These aggregated values are then aggregated together by additional circuits of the dot product bus lane, and configuration data specifies whether to pass the aggregated dot products in one direction of the segment or the other, or whether that segment is the final aggregator for the dot product (in which case that aggregated dot product is provided to a post-processing unit in that segment).

In some embodiments, each segment includes the same number of post-processing units as dot product bus lanes, with each post-processing unit receiving the output of a different dot product bus lane as its primary input. The post-processing units, as mentioned, perform the non-dot product functions of the neural network nodes. For a typical computation node of a convolutional (or fully-connected) layer, this includes a bias factor, a scaling factor, and a non-linear activation function. In some embodiments, the outputs of the linear function are quantized or truncated to a particular number of bits (e.g., 4 bits). Using a small, fixed number of bits for the outputs of each computation node allows for (i) power and resource savings by enabling smaller computations and (ii) certainty in the scheduling of computations (i.e., by knowing that all input values will be within a particular range) that enables further power and resource savings in design.

The non-linear activation function, in some embodiments, is implemented as a lookup table rather than a hardwired function. This enables the IC to execute different neural networks that use different activation functions and, in some embodiments, allows for different activation functions to be used in different layers of the neural network. For instance, in addition to common activation functions such as the Rectified Linear Unit (RELU), periodic activation functions, etc. are possible. In some embodiments, the lookup table circuit receives a truth-table (e.g., a 4-bit to 4-bit or 5-bit to 4-bit mapping) as configuration data in order to implement the non-linear activation function.

In addition to these operations, in some embodiments the post-processing units include additional circuitry for (i) performing additional dot product operations if required and (ii) performing operations for neural network computation nodes that do not use dot products. The post-processing units of some embodiments each have the ability to combine dot products from two separate cycles if a dot product is too large to be computed in a single cycle across the cores of the computation fabric. In addition, if a particular filter slice needs to be split across multiple cores (because of too many non-zero weight values, as described in greater detail below), the post-processing unit is configured to account for that. Furthermore, some embodiments enable dot products to be double the size of the standard quantized output (e.g., 8-bit rather than 4-bit) by using dot products from multiple cycles and bit-shifting the first set of input data.

Neural network computation nodes that do not use dot products include, for example, pooling layers of convolutional networks (e.g., average pooling and max pooling layers) as well as nodes that perform element-wise operations. In some of these embodiments, the cores provide input values directly to the post-processing units without computing dot products, and the post-processing units are configured to perform the appropriate operations on these inputs.

The output bus carries the computation node outputs from the post-processing units back to the cores, to be stored in the memory of the core and used as inputs for the next layer of neural network computation nodes. In some embodiments, the output values may be computed by post-processing units in one cluster but carried to a core in another cluster to be stored. For efficiency, the compiler of some embodiments (a software program that generates the configuration data for enabling the IC to execute a particular neural network) attempts to optimize the location of the post-processing unit for each computation node output relative to the cores used to compute the constituent partial dot products for that computation node and the destination core for the output value.

As mentioned, the cores compute partial dot products in parallel that are provided to the dot product bus of the local channel segment. In some embodiments, the cores include memory that stores the weight values and input values, an input buffer into which input values are loaded for the partial dot product computations, a set of weight value buffers into which weight values are loaded for the partial dot product computations, a controller or set of controller circuits for loading the input values and weight values from memory into the respective buffers, and a set of partial dot product computation circuits.

In some embodiments, the number of partial dot product computation circuits in each core is equal to (i) the number of weight value buffers in the core (or half of the number of weight value buffers, if primary and secondary buffers are used), (ii) the number of independent lanes of the dot product bus, and (iii) the number of post-processing units of each segment of the global channel. Thus, for a typical neural network computation node, the partial dot products computed by the partial dot product computation circuits having a particular index are aggregated by the dot product bus lane with the same index and that aggregated dot product is provided for post-processing to one of the post-processing units with the same index (i.e., the post-processing unit with that index in one of the channel segments).

As mentioned, each core includes one input value buffer and numerous weight value buffers in some embodiments. In convolutional neural networks, a set of input values are used as the inputs to a number of different nodes, and each layer has numerous different filters (sets of weight values). For parallelization within the cores, such a set of input values are loaded into the input value buffers of the cores used to compute the dot product for a node (by computing partial dot products that are then aggregated), while the weight values for each of these different nodes are loaded into the weight value buffers (for a node computed across multiple cores, the weight values for a given node are loaded into the weight value buffers with the same index in each of the cores). The partial dot product computation circuits corresponding to these weight value buffers then simultaneously compute the partial dot product computations for the loaded input values.

In some embodiments, the weight values for each layer of the network are ternary values (e.g., each weight is either zero, a positive value, or the negation of the positive value), with at least a fixed percentage (e.g., 75%) of the weight values being zero. Using ternary weight values allows for the weights to be treated as {0, 1, −1} (with a potential multiplication by the actual positive weight value in the post-processor), such that the multiplication of each input value by its weight value can be handled with a simple circuit that avoids actual multiplier circuits. The remainder of the partial dot product computation in a core can then be computed with an adder tree. The sparsity requirement allows for some embodiments to reduce the size of the partial dot product computation circuits by mapping each of a first number (e.g., 144) input values to a second number (e.g., 36) of dot product inputs, such that each input value with a non-zero corresponding weight value is mapped to a different one of the dot product inputs.

Specifically, in some embodiments, the partial dot product computation circuits include at least two sets of wires for each input value, with each of the sets of wires for a given input value providing that input value to two different dot product inputs. With a guarantee of at least 75% weight sparsity (i.e., at least 75% of the weight values for any set of input values are zero), the number of dot product inputs is set at 25% (or slightly more than 25%, for redundancy) of the number of input values provided to the circuit. In some embodiments, the weight sparsity is guaranteed by the training algorithm used to train the weights to perform a specific purpose, and the neural network computation circuit is adaptable for any set of weights that meets the guarantee. In some cases, individual partial dot product computations will not meet the sparsity requirement, and solutions for handling these situations are described below.

In addition to storing discrete weight values (e.g., 0, 1, and −1), in some embodiments the input values (which, other than for the first layer, are output values of previous computation nodes) are discrete values (e.g., 4-bit values). As such, for the simplest type of dot product computation, the neural network computation circuit of some embodiments places the following restrictions on the computation: (i) all of the input values should be of the specified discrete size, (ii) the maximum number of input values is the size of the input buffer multiplied by the number of cores, (iii) all of the weight values are either 0, +a, or −a (and thus can be stored as 0, 1, and −1), and a large enough percentage of the weight values are 0 that the input values with non-zero weight values can each map to a different multiplexer input.

However, some embodiments include solutions for cases when each of these restrictions is broken (at least in certain specific ways). As mentioned above, the post-processing units of some embodiments include dot product input processing circuits for performing additional operations on the dot product inputs if needed. Specifically, some embodiments include a register that can be configured to store a dot product received by the post-processing unit on one clock cycle. The register outputs this stored value to an adder that receives both (i) the register output when enabled and (ii) the input from the dot product bus. The post-processing units also include a configurable bit shift circuit that operates on the dot product bus input prior to the adder in some embodiments.

In addition, the input value buffer and each of the weight value buffers in each core actually include both primary and secondary buffers. Each primary input value buffer can hold the specified number of input values for the partial dot product computation circuits (e.g.,144), with the secondary input value buffer able to hold the same number. Similarly, each primary weight value buffer can hold the specified number of weight values (which is the number of inputs to which the input values are reduced) for the partial dot product computation circuits (e.g.,36), with the secondary weight value buffers able to hold the same number.

In a typical dot product computation that meets the requirements listed above (i.e., sparsity, ternary weights, input values of a specified size, and less than the maximum number of dot product components), the secondary buffers are not used. However, this circuitry also enables more complex dot products to be computed if needed, thereby increasing the size and types of neural networks that can be executed by the neural network computation circuit.

Specifically, some embodiments allow up to twice the number of dot product components that can be computed in a single cycle across all of the cores. Such a computation uses the secondary input value buffer as well as the secondary weight value buffers, and uses two (or more) clock cycles of the neural network computation circuit. Weight values for the computation are loaded into the primary and secondary weight value buffers of each core used, while the input values are loaded into the primary and secondary input value buffers. In a typical convolutional neural network layer, each of the nodes has the same dimensions, and thus all of the filters will use both the primary and secondary buffers. In the first clock cycle, the partial dot product computation circuits in the cores and the dot product bus compute the dot products of the input values and weight values from the primary buffers, and the results are stored in the registers of the post-processing units. In a subsequent clock cycle, the dot product computation circuits compute the dot products of the input values and weight values from the secondary buffers, and the stored result is released from the register to be added to this secondary dot product. The results are the full dot products for the computation nodes.

In addition, some embodiments enable dot product computations with input values up to twice the bit length of the specified discrete size. For instance, if the neural network computation circuit is designed for a typical computation that uses 4-bit inputs (and thus 4-bit outputs), some embodiments allow for 8-bit input values (and outputs). To compute this using the hardware designed for the specified discrete size, some embodiments load the least significant bits (e.g., the least significant 4 bits) of the input values into the primary activation buffers in each core used and load the most significant bits (e.g., the most significant 4 bits) of the input values into the secondary activation buffers of these cores (or vice versa). The weight values for each filter are loaded into the primary weight value buffers, with the secondary weight value buffers are unused. Then, in the first clock cycle, the dot product computation circuits compute the dot products of the weight values with the input values from the primary input value buffer. These initial dot products are stored in the post-processing unit registers, as for the dot products with a large number of components. In a subsequent clock cycle, the dot product computation circuits compute the dot product of the same weight values with the input values from the secondary input value buffer. The dot product for the most significant bits is bit shifted by the post-processing unit (e.g., 4 bits in the case of 8-bit input values on 4-bit hardware) before being added to the dot product for the least significant bits (and before being stored in the register if the dot product for the most significant bits is computed first).

Furthermore, some embodiments have solutions if a particular filter slice (i.e., the weights assigned to a particular weight value buffer in a particular core) has too many non-zero values (i.e., the network is not locally sparse enough for the hardware). That is, if the number of non-zero weight values is greater than the number of multiplexer inputs to the dot product computation, then the filter slice is not sparse enough. In addition, the wiring of each input value in the input value buffer to two multiplexers will not accommodate certain configurations of non-zero weight values (which is why the neural network computation circuit of some embodiments includes a number of multiplexers that is slightly more than one-fourth the number of input values in the buffer, with some of the input values wired to three multiplexers).

The input values (and their corresponding weight values), in some embodiments, cannot be easily moved around to different cores because (i) the input values are used for numerous simultaneous dot products and (ii) the input values are efficiently carried in contiguous blocks to the cores by the output bus, with the ordering of these input values dependent on the post-processing unit indices that computed them as output values in the previous layer.

As such, the compiler of some embodiments is responsible for dividing the filter slice into two slices while maintaining efficient computation by the neural network computation circuit. The compiler, in some embodiments, is a software application that is responsible for receiving a trained neural network and converting that network into instructions for loading the weight values onto the neural network computation circuit and instructions for the circuit to execute. The compiler assigns each layer to a particular number of cores, assigns the different filter slices to different weight value buffer/partial dot product computation circuit/dot product bus lane/post-processing unit indices, specifies which segment will perform post-processing on each output value, etc. Because the computations of a previously-trained network are not dynamic (that is, the computations performed are not dependent on the results of previous computations), the entire execution of a network can be statically scheduled by the compiler.

When a filter slice is not sparse enough for each input value with a nonzero weight value to be mapped to a different multiplexer input, different embodiments use different techniques to divide the filter. Some embodiments move a portion of the weight values to the secondary weight value buffer in the core. For instance, the odd weight values could be moved to the secondary weight value buffer, such that all of the odd values are 0 in the primary weight value buffer and all of the even values are 0 in the secondary weight value buffer. In other embodiments, the compiler does not blindly move half of the weight values to the secondary weight value buffer, but instead moves a subset of the weight values while ensuring that all of the input values with nonzero weights can be mapped to different input multiplexers. In this scenario, the computation of the node incurs an extra clock cycle of latency, as the dot product is computed similarly to the manner described above. That is, a dot product is computed between the primary weight value buffer and the input value buffer, stored in the post-processing unit register, then added to a dot product computed in a subsequent clock cycle between the secondary weight value buffer and the input value buffer (with different mappings of input values to multiplexers for each computation).

However, some embodiments also provide for the ability to split filters without incurring the latency cost. In addition, dot products with a large number of components or with large input values already use the secondary buffers, and thus require a different solution for such filters. In some embodiments, each post-processing unit (or the majority of the post-processing units) receives the output of two different dot product bus lanes. One of these is the output of the dot product bus lane with the same index as the post-processing unit, and one of these is the output of a different lane. In some embodiments, the first half of the dot product bus lanes are only wired to a single post-processing unit in each channel segment (those with the same index), with the secondary inputs all coming from the second half of the lanes. This allows for optimal packing of output values from the post-processing units when filter splits are required.

Using this scheme, when a filter slice needs to be split, a subset of the weight values are assigned to a different input value buffer (i.e., an input value buffer with a different index), with the removed weight values again replaced by zeros. This different input value buffer is selected such that the same post-processing unit receives both of the dot product bus computations from (i.e., as its primary and secondary inputs). The post-processing unit adds these two dot products together before performing the post-processing operations (bias, scaler, etc.). In addition, if a filter split is required for a large dot product or a dot product with large input values, the post-processing unit performs this aggregation before performing the bit shifting and/or storing the initial dot product in the register. However, if a particular dot product index is used for a split filter, then the post-processing unit with that index will not receive a useful primary input. However, that post-processing unit can have its primary input disabled and its secondary input enabled (so long as that filter does not also need to be split). In addition, the use of secondary inputs can be rippled to additional post-processing units if needed, so as to pack the output values next to each other on the output bus (allowing for easier writes to memory in the cores).

Some embodiments also use similar techniques to enable larger numbers of possible weight values, while using the same hardware designed for ternary weights. To illustrate these techniques, the example of quinary weights (e.g., −2, −1, 0, 1, 2) will be used. As an optimal solution if the weights in a filter are sparse enough, then quinary weights can be used without incurring any latency or cost. As noted, each input value in the input value buffer is wired to two different multiplexers. In certain cases, both of the multiplexers that receive input values with weights of −2 or 2 select those inputs, and the weight of 1 or −1 is used for both of these inputs (so that the input value is effectively multiplied by 2 or −2).

However, in many cases the compiler will not be able to find such a solution if the weight values are not sparse enough (i.e., the second multiplexers for all of the input values with +2/−2 weights will not be free to select those input values). In these cases, however, the compiler can use the filter splitting techniques. Rather than move half of the input values to a different input value buffer (either the secondary buffer or a buffer with a different index), the entire filter is duplicated. Weight values of +2/−2 are represented by +1/−1 in both filters, while weight values of +1/−1 are represented by +1/−1 in the first filter and 0 in the second filter. Some embodiments use either of the filter splitting techniques, depending on the requirements of the network (i.e., either using the secondary weight value buffer and incurring a latency penalty or moving the second half of the filter to a weight value buffer with a different index). For example, if the number of filters is large and all of the filters use quinary weights, then using the secondary weight value buffers and incurring the latency penalty may be preferred.

The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description and the Drawings is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description and the Drawings, but rather are to be defined by the appended claims, because the claimed subject matters can be embodied in other specific forms without departing from the spirit of the subject matters.

