BACKGROUND

This specification relates to localized vector processing units that can be used to perform a variety of computations associated with dimensional arrays of data which can generally be referred to as vectors.

Vector processing units can be used for computations associated with technology fields such as numerical simulations, graphics processing, gaming console design, supercomputing, and machine learning computations for Deep Neural Networks (“DNN”) layers.

In general, neural networks are machine learning models that employ one or more layers of models to generate an output, e.g., a classification, for a received input. A neural network having multiple layers can be used to compute inferences by processing the input through each of the layers of the neural network.

SUMMARY

As compared to features of conventional vector processing units (VPUs), this specification describes a VPU configured to partition computations into: a) an example single instruction multiple data (SIMD) VPU having increased flexibility, increased memory bandwidth requirements, and fairly low computational density; b) a matrix unit (MXU) with lower flexibility, low memory bandwidth requirements, and high computational density; and c) a low memory-bandwidth cross-lane unit (XU) for performing certain operations that might not fit into the SIMD paradigm, but also might not have the computational density of MXU computational operations. In general, at least the contrast between the computational features of a) and b), provide for an enhanced SIMD processor design architecture relative to current/conventional SIMD processors. In some implementations, the described VPU is an example Von-Neumann SIMD VPU.

In general, one innovative aspect of the subject matter described in this specification can be embodied in a vector processing unit, including, one or more processor units that are each configured to perform arithmetic operations associated with vectorized computations for a multi-dimensional data array; and a vector memory in data communication with each of the one or more processor units. The vector memory includes memory banks configured to store data used by each of the one or more processor units to perform the arithmetic operations. The one or more processor units and the vector memory are tightly coupled within an area of the vector processing unit such that data communications can be exchanged at a high bandwidth based on the placement of respective processor units relative to one another and based on the placement of the vector memory relative to each processor unit.

In some implementations, the vector processing unit couples to a matrix operation unit configured to receive at least two operands from a particular processor unit, the at least two operands being used by the matrix operation unit to perform operations associated with vectorized computations for the multi-dimensional data array. In some implementations, the vector processing unit further includes a first data serializer coupled to the particular processor unit, the first data serializer being configured to serialize output data corresponding to one or more operands provided by the particular processor unit and received by the matrix operation unit. In some implementations, the vector processing unit further includes a second data serializer coupled to the particular processor unit, the second data serializer being configured to serialize an output data provided by the particular processor unit and received by at least one of: the matrix operation unit, a cross-lane unit, or a reduction and permute unit.

In some implementations, each of the one or more processor units include a plurality of processing resources and the plurality of processing resources include at least one of a first arithmetic logic unit, a second arithmetic logic unit, a multi-dimensional register, or a function processor unit. In some implementations, the vector memory is configured to load data associated with a particular memory bank to respective processor units, and wherein the data is used by a particular resource of the respective processor units. In some implementations, the vector processing unit further includes a crossbar connector intermediate the one or more processor units and the vector memory, the crossbar connector being configured to provide data associated with a vector memory bank to a particular resource of the plurality of processing resources of a particular processor unit.

In some implementations, the vector processing unit further includes a random number generator in data communication with a resource of a particular processor unit, the random number generator being configured to periodically generate a number that can be used as an operand for at least one operation performed by the particular processor unit. In some implementations, the vector processing unit provides a primary processing lane and includes multiple processor units that each respectively form a processor sub-lane within the vector processing unit. In some implementations, each processor sub-lane is dynamically configured on a per-access basis to access a particular memory bank of the vector memory to retrieve data used to perform one or more arithmetic operations associated with vectorized computations for the multi-dimensional data array.

Another innovative aspect of the subject matter described in this specification can be embodied in a computing system having a vector processing unit, the computing system including, processor units that each include a first arithmetic logic unit configured to perform a plurality of arithmetic operations; a vector memory in data communication with each of the one or more processor units, the vector memory including memory banks configured to store data used by each of the one or more processor units to perform the arithmetic operations; and a matrix operation unit configured to receive at least two operands from a particular processor unit, the at least two operands being used by the matrix operation unit to perform operations associated with vectorized computations.

The one or more processor units and the vector memory are tightly coupled within an area of the vector processing unit such that data communications can be exchanged at a first bandwidth based on a first distance between at least one processor unit and the vector memory. The vector processing unit and the matrix operation unit are coupled such that data communications can be exchanged at a second bandwidth based on a second distance between at least one processor unit and the matrix operation unit. The first distance is less than the second distance and the first bandwidth is greater than the second bandwidth.

In some implementations, the computing system further includes a first data serializer coupled to the particular processor unit, the first data serializer being configured to serialize output data corresponding to one or more operands provided by the particular processor unit and received by the matrix operation unit. In some implementations, the computing system further includes a second data serializer coupled to the particular processor unit, the second data serializer being configured to serialize output data provided by the particular processor unit and received by at least one of: the matrix operation unit, a cross-lane unit, or a reduction and permute unit. In some implementations, each of the one or more processor units further include a plurality of processing resources comprising at least one of a second arithmetic logic unit, a multi-dimensional register, or a function processor unit.

In some implementations, the vector memory is configured to load data associated with a particular memory bank to respective processor units, and wherein the data is used by a particular resource of the respective processor units. In some implementations, the computing system further includes a crossbar connector intermediate the one or more processor units and the vector memory, the crossbar connector being configured to provide data associated with a vector memory bank to a particular resource of the plurality of processing resources of a particular processor unit. In some implementations, the computing system further includes a random number generator in data communication with a resource of a particular processor unit, the random number generator being configured to periodically generate a number that can be used as an operand for at least one operation performed by the particular processor unit. In some implementations, the computing system further includes a data path that extends between the vector memory and the matrix operation unit, the data path enabling data communications associated with direct memory access operations that occur between the vector memory and at least the matrix operation unit.

Another innovative aspect of the subject matter described in this specification can be embodied in a computer-implemented method in a computing system having a vector processing unit. The method includes, providing, by a vector memory, data for performing one or more arithmetic operations, the vector memory including memory banks for storing respective sets of data, receiving, by one or more processor units, data from a particular memory bank of the vector memory, the data being used by the one or more processor units to perform one or more arithmetic operations associated with vectorized computations; and receiving, by a matrix operation unit, at least two operands from a particular processor unit, the at least two operands being used by the matrix operation unit to perform operations associated with vectorized computations. The one or more processor units and the vector memory are tightly coupled within an area of the vector processing unit such that data communications occur at a first bandwidth based on a first distance between at least one processor unit and the vector memory. The vector processing unit and the matrix operation unit are coupled such that data communications occur at a second bandwidth based on a second distance between at least one processor unit and the matrix operation unit. The first distance is less than the second distance and the first bandwidth is greater than the second bandwidth.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. Using a vector processing unit that includes highly localized data storage and computational resources can provide increased data throughput relative to current vector processors. The described vector memory and processing unit architecture enables localized high bandwidth data processing and arithmetic operations associated with vector elements of an example matrix-vector processor. Hence, computational efficiency associated with vector arithmetic operations can be enhanced and accelerated based on use of vector processing resources that are disposed within a circuit die in a tightly coupled arrangement.

Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation cause the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

