TECHNICAL FIELD

Disclosed embodiments of memory access commands with near-memory address generation pertain generally to volatile and non-volatile computer memory and, more particularly, to computer-implemented logic for accessing data stored in computer memory.

BACKGROUND

Computer memory is a fundamental component of virtually every computer including personal computers, tablet computers, smartphones, server computers, and other computing devices such as, for example, printers and internet connected devices. Computer memory is typically used in computers to store data and processor-executable instructions (machine code) that operate on the data.

While the term “memory” is often used to refer to volatile computer memory such as, for example, dynamic random-access memory (or just “DRAM” for brevity) that takes the form of integrated circuit chips with metal-oxide semiconductor memory cells, computer memory can also be non-volatile. A non-exhaustive list of volatile and non-volatile computer technologies includes DRAM, static random-access memory (or just “SRAM” for brevity), non-volatile dual inline memory module (or just “NVDIMM” for brevity), flash memory, embedded DRAM, scratchpad memory, etc.

The computer memory of a computer is typically connected to a hardware “host” of the computer via a set of wire(s) or conductor(s) of the computer. The hardware host may encompass a memory controller and a central processing unit (or just “CPU” for brevity), a graphics processing unit (or just GPU for brevity), or other hardware data processor. The connection(s) between computer memory and the host typically encompasses an interface for command/address information (sometimes referred to as a “command/address bus”) and an interface for data information (sometimes referred to as a “data bus”).

Recently, the computing capabilities and the speed of processors (e.g., CPUs and GPUs) have been increasing faster than the bandwidth of the data bus connecting computer memory to the host. As a result, the data bus has become a bottleneck in computing throughput as the processor idly waits for data to be transferred to and from computer memory via the data bus.

Processor-in-memory (or just “PIM” for brevity) solutions provide data processing capabilities that are near the computer memory (e.g., on the same die). With PIM solutions, since data processing capabilities are near to the computer memory, computation can be performed on data accessed from computer memory without having to transfer all of the raw data over the data bus to the host. For example, a possible PIM solution may involve adding simple vector compute elements to each internal memory module of computer memory such as to each DRAM bank or sub-array. Then, with these additional elements, the host can trigger computation at all internal memory modules without requiring transfer of data across the DRAM interface across the data bus.

PIM solutions can conserve the consumption of bandwidth of the data bus by data transfers between computer memory and the host and improve throughput of compute workloads, especially for the growing category of data-limited compute workloads such as, for example, deep artificial neural network machine learning workloads, large-scale graph analytic workloads, sparse computations in machine learning workloads, among other types of compute-intensive workloads that can reach the bandwidth limit of the data bus with data transfers between the computer memory and the host in the absence of memory-local PIM units.

As but one example, a common operation in machine learning is a sparse embedding lookup. This operation can dominate execution time of some workloads. Such an operation can sparsely read large embeddings containing hundreds of elements out of a large embedding table containing millions of embedding entries stored in computer memory and accumulate them together. Because of the limited data reuse, such an operation can be highly sensitive to the bandwidth of the data bus.

Without using a PIM solution, a sparse embedding lookup operation might require the host issuing a set of memory load requests, one for each of a number of memory modules (e.g., DRAM banks). Each memory load request causes data to be transferred from a respective memory module to the host across the data bus. Since the memory modules may share the command/address bus and the data bus (e.g., DRAM banks on the same memory channel), access to the different memory modules may need to be serialized resulting in full utilization of both the command/address and data buses. In contrast, if a PIM solution is used, the accumulation can be computed in memory-local PIM units without requiring the transfer of some data on the data bus.

However, as PIM solutions can alleviate some or all of the data bus bottleneck for certain workloads, a bottleneck at the command/address bus can become a new bottleneck. For example, a bottleneck at the command/address bus can arise with data-limited compute workloads that have sparse or irregular memory access patterns such as those found with machine learning workloads and other workloads. Since the command/address bus is shared by the memory modules, each memory load command still needs to be sent across the command/address bus.

A possible solution to the command/address bus bottleneck is to increase the bandwidth of the command/address bus by increasing the number of memory channels or by increasing the number of command/address wires per memory channel. However, this increases the per-unit manufacturing cost and increases power consumption. Further, for workloads having regular memory access patterns, the extra command/address bus bandwidth may go unutilized.

Disclosed embodiments address this and other issues.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art, or are well-understood, routine or conventional, merely by virtue of their inclusion in this section.

