Survey of Memory Management Techniques for HPC and Cloud Computing <s> A. TOWARDS EXASCALE COMPUTING <s> Over the last 20 years, the open-source community has provided more and more software on which the world’s high-performance computing systems depend for performance and productivity. The community has invested millions of dollars and years of effort to build key components. However, although the investments in these separate software elements have been tremendously valuable, a great deal of productivity has also been lost because of the lack of planning, coordination, and key integration of technologies necessary to make them work together smoothly and efficiently, both within individual petascale systems and between different systems. It seems clear that this completely uncoordinated development model will not provide the software needed to support the unprecedented parallelism required for peta/ exascale computation on millions of cores, or the flexibility required to exploit new hardware models and features, such as transactional memory, speculative execution, and graphics processing units. This report describes the work of the community to prepare for the challenges of exascale computing, ultimately combing their efforts in a coordinated International Exascale Software Project. <s> BIB001 </s> Survey of Memory Management Techniques for HPC and Cloud Computing <s> A. TOWARDS EXASCALE COMPUTING <s> Commodity clusters revolutionized high-performance computing when they first appeared two decades ago. As scale and complexity have grown, new challenges in reliability and systemic resilience, energy efficiency and optimization and software complexity have emerged that suggest the need for re-evaluation of current approaches. This paper reviews the state of the art and reflects on some of the challenges likely to be faced when building trans-petascale computing systems, using insights and perspectives drawn from operational experience and community debates. <s> BIB002
In the last decade, various researchers have worked to overcome Exascale challenges. The request for more complex computational resources was indicated in the International Exascale Software Project roadmap BIB001 . Increasing the number of cores per node is a traditional way to increase computational performance. The roadmap indicated the need for a wider use of heterogeneous nodes that combine streambased cores with traditional cores based on load/store. Also, it was noticed that in addition to increasing the complexity of the computational resources, the heterogeneous architecture complicates the programming models and languages in order to provide the efficient resources sharing (e.g. the memory bus). The study of data locality was appointed as one of the directions of the system software development since it can have a much significant impact on performance when used in systems with a complex memory hierarchy. Machines were expected to have deep and non-coherent memory hierarchies. The locality-aware scheduling in highly NUMA memory architectures would have demonstrated how to use them. By 2013, such runtime systems should have been developed. A dynamic runtime management system for all types of resources, such as cores, bandwidth, logical and physical memory or storage (i.e., data replication management, coherency and consistency, layout changes more suitable for specific cores/accelerators), was announced to be completed by 2019 year. The importance of new memory management solutions has also been described in the report on crosscutting technologies for computing at the exascale level  . The memory management was seen as a significant challenge for exascale science applications due to deeper, complex memory hierarchies and relatively smaller capacities. A complex memory hierarchy expected at the exascale requires tools and interfaces to manage and control memory for runtime systems, and to provide information to assist the compiler in optimising memory management. A promising way to deal with memory complexity issues was to base it on dynamic, latencytolerant approaches. The following two main challenges were highlighted. On-Chip Memory: Given the importance of minimising unnecessary data movement, the ability to allow software control of data movement for performance-critical kernels would be useful. Methods that can coexist with conventional, automatically managed caches are important because an incremental porting path is required. Global Addressing: Message passing probably can continue to be used for internode communication in exascale systems, but it is not practical for interprocessor communication at billionway parallelism. Global memory addressing is preferred over cache-coherence for managing fine-grained computation and data movement on-chip  . According to the survey looking at the challenges of the high-performance computing scaling challenges BIB002 , the complexity of an application and system software will continue to grow in several dimensions. In the specific case of memory management, it is expected that both homogeneous and heterogeneous designs would probably have multiple levels of memory; these hierarchical memories would affect the complexity of the software. The challenge proposed to the software developer is to find the most efficient way to utilise the different memory types, as well as to minimise the movement of data within the hierarchy, due to the NUMA nature of the hierarchy and the different types of memory. Moreover, there are clear similarities in HPC and Cloud Computing for what concerns different architectural and software issues, such as component rightsizing and optimisation, energy management and efficiency, and programming efficiency. The recent crosscut report of exascale requirements reviews held by Advanced Scientific Computing Research program  highlighted the extreme-scale systems that will be delivered in 8 to 10 years, and that will offer unprecedented compute power and challenging parallelism. Relatively slow interconnects between nodes become the main challenge for exascale machines that will contain millions of heterogeneous cores with deep memory hierarchies. The report discusses memory issues that significantly complicate software development, such as the need for explicit control of data movement, depth and types of memory and burst buffers. At the same time, planning and distribution policies are also complicated by the need to support extra functionality in modern workflows, especially data analysis requirements: real-time, pseudo-real time, collaborative planning, variable work requirements, and allocations based on other resources, such as disk and memory. In real simulations in nuclear astrophysics, deep memory hierarchies were expected to present significant challenges because large, multidimensional data structures are ubiquitous in astrophysical implementations. So the way the data structure is organised in memory is essential to ensure effective memory accessibility (for example, for vectorisation) and memory allocation. Problems with memory allocation include the placement of data structures in certain levels of the memory hierarchy to ensure the best locality. It was noted that the nuclear physics community needs a portable way without direct compiler support to effectively manage the memory layout and the memory allocation. Memory management libraries were proposed as probably the best method. Fully ceiding memory management solutions in runtime systems were found ineffective since the requirements for layout and placement are physically motivated and can depend on time.