CROSS-REFERENCE TO RELATED APPLICATIONS

This is the first filing related to the disclosed technology. At the time of filing, there are no related patents or applications.

FIELD OF THE INVENTION

The present disclosure relates generally to the field of optimization of memory access in heterogeneous parallel architectures, and more particularly to compile-time methods and systems for optimizing access to constant memory in heterogeneous parallel architectures.

BACKGROUND

Current computing architectures used, e.g., in artificial intelligence, data science, and other applications commonly take advantage of one or more central processing units (CPUs), typically having multiple parallel cores, as well as special-purpose processing units, such as graphics processing units (GPUs), digital signal processors (DSPs), tensor processing units (TPUs), field-programmable gate arrays (FPGAs) and other types of processors, which may include tens or even hundreds of special-purpose processors, all of which may run in parallel. Parallel architectures of this sort, in which there may be numerous different kinds of processors, are referred to as heterogeneous parallel architectures.

Developing software that runs on heterogeneous parallel architectures has been challenging. The various processors that are used may each be based on different models of computation, and there may be different types of memory, and so on. Typically, vendors who produced special-purpose processors that are used in heterogeneous parallel systems also provided specialized software development tools, such as compilers, libraries, specialized assemblers, and other tools for working with their hardware. As a result, it was often necessary to rewrite significant blocks of code to move from one heterogeneous parallel computing architecture to another.

OpenCL is one attempt to address these difficulties by providing a common and open framework for writing programs that execute across these heterogeneous parallel computing platforms. OpenCL provides a computing and memory model that provides a standardized interface for parallel computing, including task- and data-based parallelism.

One type of memory used in the OpenCL memory model is referred to as “constant memory.” Constant memory stores constants that are used in OpenCL programs, and cannot be written to by the parallel kernels that use these constants. The constants that are stored in constant memory are often arrays of values, accessed using an access index that varies to access the various elements of the array. Constant memory in OpenCL is usually a part of global memory, which may be accessed by all the parallel kernels of a program, and to which access is often relatively slow. In some architectures, the special-purpose processing units may include special on-chip constant memory and/or a constant cache to speed access to constants. However, these features are not available on all architectures, and although most special-purpose architectures include a variety of types of memory which may provide much faster access than global memory, these other types of fast memory are not generally used for storing constants.

SUMMARY

Advantageously, the present disclosure provides an optimizer for use in a compiler to permit constants to be stored in a variety of types of memory in a heterogeneous parallel architecture, including fast and medium-speed memories that would not typically be used for storing constants. Additionally, the disclosed technology permits constants to be stored and accessed from on-chip memory in special-purpose processing units that do not include on-chip constant memory or a constant cache. This can result in significant improvements in the memory performance of heterogeneous parallel computers when accessing constants. Although the disclosure gives examples targeting OpenCL, it will be understood that the disclosed technology can also be used to improve the memory access speed for constants in other heterogeneous parallel architectures.

In accordance with one aspect of the present disclosure, the technology is implemented in an apparatus including a processor, a memory coupled to the processor, and a compiler residing in the memory and executed by the processor. The compiler includes a constant memory optimizer that includes a scope determination module configured to determine a scope of a constant in code for a heterogeneous parallel computer. The constant memory optimizer further includes a code transformation module configured to transform the code to pass the constant as an implicit pointer argument into a kernel in the code that references the constant if the scope of the constant is a program scope or an outermost scope of the kernel.

In some implementations, the constant memory optimizer further includes an access pattern determination module configured to determine an access pattern of the constant in the kernel, and a memory type determination module configured to select a type of memory to be used for the constant based on the access pattern. In some implementations, the code transformation module is further configured to transform the code to use the selected type of memory for the constant

In accordance with other aspects of the present disclosure, a method of optimizing access to constants in a heterogeneous parallel computer is provided. The method includes determining a scope of a constant in code for the heterogeneous parallel computer using a scope determination module and transforming the code to pass the constant as an implicit pointer argument to a kernel in the code that references the constant if the scope of the constant is a program scope or an outermost scope of the kernel using a code transformation module. The method also includes determining an access pattern of the constant in the kernel using an access pattern determination module and selecting a type of memory to be used for the constant based on the access pattern using a memory type determination module. Transforming the code also includes transforming the code to use the selected type of memory for the constant.

In some implementations of the method, in a first case in which the scope of the constant is the program scope or the outermost scope of the kernel, determining the access pattern of the constant includes determining a size of the constant. In these selecting the type of memory to be used for the constant may include: selecting a fast memory for the constant in a “small-size” sub-case if the size of the constant is sufficiently small for placement in the fast memory; selecting a medium-speed memory for the constant in a “program scope known access range” sub-case if the size of the constant is too large for placement in the fast memory and if the access pattern includes a known access index range at compile-time; and selecting a normal type of memory used for constants in a “default” sub-case if neither the small-size sub-case nor the program scope known access range sub-case applies.

In some implementations of the method, in a second case in which the constant is a constant pointer argument, two sub-cases may be used for selecting the type of memory to be used for the constant. These two sub-cases include: selecting a fast memory or a medium-speed memory in a “constant pointer known access” sub-case, in which the fast memory is selected for a known access index range of the constant if the access pattern includes the known access index range at compile-time and a size of the known access index range is sufficiently small for placement in the fast memory, and in which the medium-speed memory is selected for the known access index range of the constant if the access pattern includes a known access index range at compile-time and the size of the known access index range is too large for placement in the fast memory; and selecting a normal type of memory used for constant values in a “constant pointer unknown access” sub-case if the access pattern does not include a known access index range at compile-time.

In accordance with still further aspects of the disclosure, the disclosed technology may be implemented as instructions encoded on a computer-readable medium. When these instructions are executed by a processor, they cause the processor to perform operations, including causing the processor to determine a scope of a constant in code for a heterogeneous parallel computer and determine a size and an access pattern of the constant in a kernel in the code. The instructions also cause the processor to select a type of memory to be used for the constant based on the size and access pattern.

In a first case, in which the scope of the constant is the program scope or the outermost scope of the kernel, the instructions cause the processor to select the type of memory according to several sub-cases. If the size of the constant is sufficiently small for placement in the fast memory, then the processor selects a fast memory for the constant in a “small-size” sub-case. If the size of the constant is too large for placement in the fast memory and if the access pattern includes a known access index range at compile-time, then the processor selects a medium-speed memory for the constant in a “program scope known access range” sub-case. Finally, if neither the small-size sub-case nor the program scope known access range sub-case applies, then the processor selects a normal type of memory used for constants in a “default” sub-case.

In a second case, in which the constant is a constant pointer argument, the size of the constant will not be known, so the instructions cause the processor to select the type of memory for the constant according to different sub-cases. The instructions cause the processor to select a fast memory or a medium-speed memory in a “constant pointer known access” sub-case, in which the fast memory is selected for a known access index range of the constant if the access pattern includes the known access index range at compile-time and a size of the known access index range is sufficiently small for placement in the fast memory. The medium-speed memory is selected for the known access index range of the constant if the access pattern includes the known access index range at compile-time and the size of the known access index range is too large for placement in the fast memory. In a “constant pointer unknown access” sub-case, the instructions cause the processor to select a normal type of memory used for constant values if the access pattern does not include a known access index range at compile-time.

Finally, the instructions cause the processor to transform the code to pass the constant as an implicit pointer argument to a kernel in the code that references the constant if the scope of the constant is a program scope or an outermost scope of the kernel and to use the selected type of memory for the constant.

It will be understood that in any of these aspects of the disclosure, the heterogeneous parallel computer may support OpenCL, and transforming the code may include transforming code expressed in an OpenCL intermediate representation, such as SPIR-V (see below) and/or LLVM IR (see below). Further, in any of these aspects, a “fast” memory may include registers in a compute device, and a “medium-speed” memory may include on-chip memory in a compute device, such a shared or local memory.

