A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> This paper explores the benefits and limitations of in-storage processing on current Solid-State Disk (SSD) architectures. While disk-based in-storage processing has not been widely adopted, due to the characteristics of hard disks, modern SSDs provide high performance on concurrent random writes, and have powerful processors, memory, and multiple I/O channels to flash memory, enabling in-storage processing with almost no hardware changes. In addition, offloading I/O tasks allows a host system to fully utilize devices' internal parallelism without knowing the details of their hardware configurations. To leverage the enhanced data processing capabilities of modern SSDs, we introduce the Smart SSD model, which pairs in-device processing with a powerful host system capable of handling data-oriented tasks without modifying operating system code. By isolating the data traffic within the device, this model promises low energy consumption, high parallelism, low host memory footprint and better performance. To demonstrate these capabilities, we constructed a prototype implementing this model on a real SATA-based SSD. Our system uses an object-based protocol for low-level communication with the host, and extends the Hadoop MapReduce framework to support a Smart SSD. Our experiments show that total energy consumption is reduced by 50% due to the low-power processing inside a Smart SSD. Moreover, a system with a Smart SSD can outperform host-side processing by a factor of two or three by efficiently utilizing internal parallelism when applications have light trafic to the device DRAM under the current architecture. <s> BIB001 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> While Processing-in-Memory has been investigated for decades, it has not been embraced commercially. A number of emerging technologies have renewed interest in this topic. In particular, the emergence of 3D stacking and the imminent release of Micron's Hybrid Memory Cube device have made it more practical to move computation near memory. However, the literature is missing a detailed analysis of a killer application that can leverage a Near Data Computing (NDC) architecture. This paper focuses on in-memory MapReduce workloads that are commercially important and are especially suitable for NDC because of their embarrassing parallelism and largely localized memory accesses. The NDC architecture incorporates several simple processing cores on a separate, non-memory die in a 3D-stacked memory package; these cores can perform Map operations with efficient memory access and without hitting the bandwidth wall. This paper describes and evaluates a number of key elements necessary in realizing efficient NDC operation: (i) low-EPI cores, (ii) long daisy chains of memory devices, (iii) the dynamic activation of cores and SerDes links. Compared to a baseline that is heavily optimized for MapReduce execution, the NDC design yields up to 15X reduction in execution time and 18X reduction in system energy. <s> BIB002 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> As computation becomes increasingly limited by data movement and energy consumption, exploiting locality throughout the memory hierarchy becomes critical to continued performance scaling. Moving computation closer to memory presents an opportunity to reduce both energy and data movement overheads. We explore the use of 3D die stacking to move memory-intensive computations closer to memory. This approach to processing in memory addresses some drawbacks of prior research on in-memory computing and is commercially viable in the foreseeable future. Because 3D stacking provides increased bandwidth, we study throughput-oriented computing using programmable GPU compute units across a broad range of benchmarks, including graph and HPC applications. We also introduce a methodology for rapid design space exploration by analytically predicting performance and energy of in-memory processors based on metrics obtained from execution on today's GPU hardware. Our results show that, on average, viable PIM configurations show moderate performance losses (27%) in return for significant energy efficiency improvements (76\% reduction in EDP) relative to a representative mainstream GPU at 22nm technology. At 16nm technology, on average, viable PIM configurations are performance competitive with a representative mainstream GPU (7% speedup) and provide even greater energy efficiency improvements (85\% reduction in EDP). <s> BIB003 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> We explore the potential of making programmability a central feature of the SSD interface. Our prototype system, called Willow, allows programmers to augment and extend the semantics of an SSD with application-specific features without compromising file system protections. The SSD Apps running on Willow give applications low-latency, high-bandwidth access to the SSD's contents while reducing the load that IO processing places on the host processor. The programming model for SSD Apps provides great flexibility, supports the concurrent execution of multiple SSD Apps in Willow, and supports the execution of trusted code in Willow. ::: ::: We demonstrate the effectiveness and flexibility of Willow by implementing six SSD Apps and measuring their performance. We find that defining SSD semantics in software is easy and beneficial, and that Willow makes it feasible for a wide range of IO-intensive applications to benefit from a customized SSD interface. <s> BIB004 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> As CPU core counts continue to increase, the gap between compute power and available memory bandwidth has widened. A larger and deeper cache hierarchy benefits locality-friendly computation, but offers limited improvement to irregular, data intensive applications. In this work we explore a novel approach to accelerating these applications through in-memory data restructuring. Unlike other proposed processing-in-memory architectures, the rearrangement hardware performs data reduction, not compute offload. Using a custom FPGA emulator, we quantitatively evaluate performance and energy benefits of near-memory hardware structures that dynamically restructure in-memory data to cache-friendly layout, minimizing wasted memory bandwidth. Our results on representative irregular benchmarks using the Micron Hybrid Memory Cube memory model show speedup, bandwidth savings, and energy reduction. We present an API for the near-memory accelerator and describe the interaction between the CPU and the rearrangement hardware with application examples. The merits of an SRAM vs. a DRAM scratchpad buffer for rearranged data are explored. <s> BIB005 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> Many studies point to the difficulty of scaling existing computer architectures to meet the needs of an exascale system (i.e., capable of executing $10^{18} $ floating-point operations per second), consuming no more than 20 MW in power, by around the year 2020. This paper outlines a new architecture, the Active Memory Cube, which reduces the energy of computation significantly by performing computation in the memory module, rather than moving data through large memory hierarchies to the processor core. The architecture leverages a commercially demonstrated 3D memory stack called the Hybrid Memory Cube, placing sophisticated computational elements on the logic layer below its stack of dynamic random-access memory (DRAM) dies. The paper also describes an Active Memory Cube tuned to the requirements of a scientific exascale system. The computational elements have a vector architecture and are capable of performing a comprehensive set of floating-point and integer instructions, predicated operations, and gather-scatter accesses across memory in the Cube. The paper outlines the software infrastructure used to develop applications and to evaluate the architecture, and describes results of experiments on application kernels, along with performance and power projections. <s> BIB006 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> Processing -in-memory (PIM) is rapidly rising as a viable solution for the memory wall crisis, rebounding from its unsuc­ cessful attempts in I990s due to practicality concerns, which are alleviated with recent advances in 3D stacking technolo­ gies. However, it is still challenging to integrate the PIM architectures with existing systems in a seamless manner due to two common characteristics: unconventional progr amming models for in-memor y computation units and lack of ability to utilize large on- chip caches. In this paper, we propose a new PIM architecture that (I) does not change the existing sequential progr amming mod­ el s and (2) aut omatically decides whet her to exe cut e PIM operations in memory or processor s de pending on the local­ ity of data. The key idea is to implement simple in-memory computation using compute- capable memory commands and use specialized instructions, which we call PIM- enabled in­ structions, to invo ke in-memory computation. This allows PIM operations to be interoperable with existing program­ ming models, cache coherence protocols, and virtual memo ry mechanisms with no modification. In addition, we introduce a simple har dware struct ure that monitors the locality of data acce ssed by a PIM-en abled instruction at runtime to adap ­ tively execut e the instruction at the host processor (instead of in memory) when the instruction can benefit from large <s> BIB007 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> The continuous growth of main memory size allows modern data systems to process entire large scale datasets in memory. The increase in memory capacity, however, is not matched by proportional decrease in memory latency, causing a mismatch for in-memory processing. As a result, data movement through the memory hierarchy is now one of the main performance bottlenecks for main memory data systems. Database systems researchers have proposed several innovative solutions to minimize data movement and to make data access patterns hardware-aware. Nevertheless, all relevant rows and columns for a given query have to be moved through the memory hierarchy; hence, movement of large data sets is on the critical path. In this paper, we present JAFAR, a Near-Data Processing (NDP) accelerator for pushing selects down to memory in modern column-stores. JAFAR implements the select operator and allows only qualifying data to travel up the memory hierarchy. Through a detailed simulation of JAFAR hardware we show that it has the potential to provide 9x improvement for selects in column-stores. In addition, we discuss both hardware and software challenges for using NDP in database systems as well as opportunities for further NDP accelerators to boost additional relational operators. <s> BIB008 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> The explosion of digital data and the ever-growing need for fast data analysis have made in-memory big-data processing in computer systems increasingly important. In particular, large-scale graph processing is gaining attention due to its broad applicability from social science to machine learning. However, scalable hardware design that can efficiently process large graphs in main memory is still an open problem. Ideally, cost-effective and scalable graph processing systems can be realized by building a system whose performance increases proportionally with the sizes of graphs that can be stored in the system, which is extremely challenging in conventional systems due to severe memory bandwidth limitations. In this work, we argue that the conventional concept of processing-in-memory (PIM) can be a viable solution to achieve such an objective. The key modern enabler for PIM is the recent advancement of the 3D integration technology that facilitates stacking logic and memory dies in a single package, which was not available when the PIM concept was originally examined. In order to take advantage of such a new technology to enable memory-capacity-proportional performance, we design a programmable PIM accelerator for large-scale graph processing called Tesseract. Tesseract is composed of (1) a new hardware architecture that fully utilizes the available memory bandwidth, (2) an efficient method of communication between different memory partitions, and (3) a programming interface that reflects and exploits the unique hardware design. It also includes two hardware prefetchers specialized for memory access patterns of graph processing, which operate based on the hints provided by our programming model. Our comprehensive evaluations using five state-of-the-art graph processing workloads with large real-world graphs show that the proposed architecture improves average system performance by a factor of ten and achieves 87% average energy reduction over conventional systems. <s> BIB009 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> Emerging highly-parallel and big data applications have renewed the research interest in Processing-in-Memory (PIM) architectures. However, moving powerful processing unit into the CMOS-incompatible DRAM chips is not cost-effective for large capacity memory. In this work, we observe that Non-Volatile Memory is often naturally incorporated with basic logics like Data Comparison Write or Flip-n-Write modules that are essential for cell SET/REST operation. In contrast to other conventional PIM or Near Data Computing (NDC) architectures, ProPRAM, as a typical Active NVM, abandons the design approach of moving accelerators or customized processors into memory devices, but begins with exploiting the existing resources inside the memory chips to accelerate the key non-compute-intensive functions for emerging big data applications. With slight hardware and architectural modification, we succeed to expose the transparent peripheral logics to the application layer through instruction set extension and exploit them for in-field bulk data processing with limited hardware cost. Compared to conventional CPU-centric systems, ProPRAM achieves an excellent optimization on energy-efficiency (15×) for important data-intensive micro-benchmarks and kernels. <s> BIB010 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> Main memory bandwidth is a critical bottleneck for modern GPU systems due to limited off-chip pin bandwidth. 3D-stacked memory architectures provide a promising opportunity to significantly alleviate this bottleneck by directly connecting a logic layer to the DRAM layers with high bandwidth connections. Recent work has shown promising potential performance benefits from an architecture that connects multiple such 3D-stacked memories and offloads bandwidth-intensive computations to a GPU in each of the logic layers. An unsolved key challenge in such a system is how to enable computation offloading and data mapping to multiple 3D-stacked memories without burdening the programmer such that any application can transparently benefit from near-data processing capabilities in the logic layer. Our paper develops two new mechanisms to address this key challenge. First, a compiler-based technique that automatically identifies code to offload to a logic-layer GPU based on a simple cost-benefit analysis. Second, a software/hardware cooperative mechanism that predicts which memory pages will be accessed by offloaded code, and places those pages in the memory stack closest to the offloaded code, to minimize off-chip bandwidth consumption. We call the combination of these two programmer-transparent mechanisms TOM: Transparent Offloading and Mapping. Our extensive evaluations across a variety of modern memory-intensive GPU workloads show that, without requiring any program modification, TOM significantly improves performance (by 30% on average, and up to 76%) compared to a baseline GPU system that cannot offload computation to 3D-stacked memories. <s> BIB011 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> Pointer chasing is a fundamental operation, used by many important data-intensive applications (e.g., databases, key-value stores, graph processing workloads) to traverse linked data structures. This operation is both memory bound and latency sensitive, as it (1) exhibits irregular access patterns that cause frequent cache and TLB misses, and (2) requires the data from every memory access to be sent back to the CPU to determine the next pointer to access. Our goal is to accelerate pointer chasing by performing it inside main memory, thereby avoiding inefficient and high-latency data transfers between main memory and the CPU. To this end, we propose the In-Memory PoInter Chasing Accelerator (IMPICA), which leverages the logic layer within 3D-stacked memory for linked data structure traversal. <s> BIB012 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> The increasing demand for extracting value out of ever-growing data poses an ongoing challenge to system designers, a task only made trickier by the end of Dennard scaling. As the performance density of traditional CPU-centric architectures stagnates, advancing compute capabilities necessitates novel architectural approaches. Near-memory processing (NMP) architectures are reemerging as promising candidates to improve computing efficiency through tight coupling of logic and memory. NMP architectures are especially fitting for data analytics, as they provide immense bandwidth to memory-resident data and dramatically reduce data movement, the main source of energy consumption. Modern data analytics operators are optimized for CPU execution and hence rely on large caches and employ random memory accesses. In the context of NMP, such random accesses result in wasteful DRAM row buffer activations that account for a significant fraction of the total memory access energy. In addition, utilizing NMP's ample bandwidth with fine-grained random accesses requires complex hardware that cannot be accommodated under NMP's tight area and power constraints. Our thesis is that efficient NMP calls for an algorithm-hardware co-design that favors algorithms with sequential accesses to enable simple hardware that accesses memory in streams. We introduce an instance of such a co-designed NMP architecture for data analytics, the Mondrian Data Engine. Compared to a CPU-centric and a baseline NMP system, the Mondrian Data Engine improves the performance of basic data analytics operators by up to 49x and 5x, and efficiency by up to 28x and 5x, respectively. <s> BIB013 </s> A Review of Near-Memory Computing Architectures: Opportunities and Challenges <s> A. Programmable Unit <s> Modern data center solid state drives (SSDs) integrate multiple general-purpose embedded cores to manage ash translation layer, garbage collection, wear-leveling, and etc., to improve the performance and the reliability of SSDs. As the performance of these cores steadily improves there are opportunities to repurpose these cores to perform application driven computations on stored data, with the aim of reducing the communication between the host processor and the SSD. Reducing host-SSD bandwidth demand cuts down the I/O time which is a bottleneck for many applications operating on large data sets. However, the embedded core performance is still significantly lower than the host processor, as generally wimpy embedded cores are used within SSD for cost effective reasons. So there is a trade-o between the computation overhead associated with near SSD processing and the reduction in communication overhead to the host system. In this work, we design a set of application programming interfaces (APIs) that can be used by the host application to offload a data intensive task to the SSD processor. We describe how these APIs can be implemented by simple modifications to the existing Non-Volatile Memory Express (NVMe) command interface between the host and the SSD processor. We then quantify the computation versus communication tradeoffs for near storage computing using applications from two important domains, namely data analytics and data integration. Using a fully functional SSD evaluation platform we perform design space exploration of our proposed approach by varying the bandwidth and computation capabilities of the SSD processor. We evaluate static and dynamic approaches for dividing the work between the host and SSD processor, and show that our design may improve the performance by up to 20% when compared to processing at the host processor only, and 6when compared to processing at the SSD processor only. CCS CONCEPTS • Computer systems organization → Secondary storage organization; Distributed architectures; Firmware; <s> BIB014
NDC (2014) Pugsley et al. BIB002 proposed a near-memory computing (NDC) architecture for MapReduce workloads, in which a central host processor with many energy efficient cores is connected to many daisy-chained 3D-stacked memory devices with simple cores in their logic layer; these cores perform Map operations. Reduce operations, however, are executed on the central host processor because it requires random access to data. TOP-PIM (2014) Zhang et al. BIB003 proposed GPUaccelerated architecture. It consists of a main APU (GPU BIB005 2015 + CPU) interconnected with high-speed serial links with multiple 3D-memory modules. APU allows code portability and easier programmability. The kernels analyzed span across a huge domain from graph processing to fluid and structural dynamics. AMC (2015) Nair et al. BIB006 developed active memory cube (AMC). AMC is built upon the HMC adding several processing elements referred as lanes in the logic layer. The host processor and each AMC are interconnected over a bidirectional link following a daisy-chain topology. For each AMC there are 32 4-slices lanes. Each lane has is own register file, a load/store unit performing read and write operations to the memory contained in the same AMC, and a computational unit.The communications between AMCs should be coordinated by the host processor. PIM-enabled (2015) Ahn et al. BIB007 created NMC architectures with the same ISA as the host. They tried to leverage the existing programming model so that the conventional architectures can exploit the PIM concept without changing their programming interface. They implemented it by adding a computation unit composed of computation logic and an SRAM operand buffer, both on host side and in the logic layer of the Hybrid Memory Cube (HMC). TOM (2016) Hsieh et al. BIB011 proposed an NMC architecture consisting of main GPU interconnected via highspeed serial links with multiple 3D-stacked memories. Each memory device hosts multiple streaming multiprocessor (SMs) in the logic layer. In line with the philosophy of near-memory computing, blocks of code are offloaded to the SMs in order to reduce off-chip traffic and reduce power consumption due to data movement. BIB013 analyzed data analytics oeprators which are not a fit for NMC. They made a case for optimizing traditional algorithms for NMC system for sequential memory access rather than cache locality. JAFAR (2015) Xi et al. BIB008 proposed JAFAR, an NMC accelerator that allows to process filtering operations near the main memory. Thus only relevant data will be pushed up in caches, causing a significant reduction in data movement. TESSERACT (2015) Ahn et al. BIB009 focus on graph processing applications. Their architecture consists of one host processor with his own memory and a HMC, with an out-oforder processor mapped to each vault. These cores can see only their own local data partition, but they can communicate to each other using a message passing method. The host processor as well has access to the entire address space of the HMC. To exploit the high available memory bandwidth in the systems, they developed prefetching mechanisms. IMPICA (2016) Hsieh et al. BIB012 accelerated the pointer chasing operations which are ubiquitous in data structures. They proposed adding specialized units that decouple address generation from memory access in the logic layer of 3D stacked memory. These units traverse through the linked data structure in memory and return only the final node found to the CPU. They also proposed completely decoupling the page table of IMPICA from that of CPU to avoid virtual memory related issues. Smart SSD (2013) Kang et al. BIB001 proposed a model to harness the processing power of the SSD using an objectbased communication protocol. They implemented the Smart SSD features in the firmware of a Samsung SSD and modified the Hadoop core and MapReduce framework to use tasklets as a map or a reduce function. To evaluate the prototype, they used a micro-benchmark and log analysis application on both a device and a host. They found that under the current SSD architecture, excessive memory accesses will make the task execution slower than in the host due to the high memory latency and low processing power. WILLOW (2014) Seshadri et al. BIB004 proposed a prototype system called Willow, which has processing units called storage processor units (SPU). Each SPU runs a small operating system that manages and enforces security. On the host-side, the Willow driver creates and manages a set of objects that allow the OS and applications to communicate with SPUs. The programmable functionality is provided in the form of SSD Apps. Willow allows programmers to augment and extend the semantics of an SSD with application-specific features without compromising file system protection. The programming model for SSD Apps provides great flexibility, supports the concurrent execution of multiple SSD Apps in Willow, and supports the execution of trusted code in Willow. ProPRAM (2015) Wang et al. BIB010 observed that NVM is often naturally incorporated with basic logic like data comparison write or flip-n-write module, and exploited the existing resources inside memory chips to accelerate the key noncompute intensive functions of emerging big data applications. SUMMARIZER (2017) Koo et al. BIB014 designed APIs that can be used by the host application to offload a dataintensive task to the inherent ARM based cores inside a SSD processor. Using a fully functional SSD evaluation platform they performed design space exploration of the proposed approach. They evaluate static and dynamic approaches for dividing the work between the host and SSD processor.