Survey of Memory Management Techniques for HPC and Cloud Computing <s> D. PARTITIONS <s> Linux-based operating systems and runtimes (OS/Rs) have emerged as the environments of choice for the majority of HPC systems. While Linux-based OS/Rs have advantages such as extensive feature sets and developer familiarity, these features come at the cost of additional system overhead. In contrast to Linux, there is a substantial history of work in the HPC community focused on lightweight OS/Rs that provide scalable and consistent performance for HPC applications, but lack many of the features offered by commodity OS/Rs. In this paper, we propose to bridge the gap between LWKs and commodity OS/Rs by selectively providing a lightweight memory subsystem for HPC applications in a commodity OS/R where concurrently executing a diverse range of workloads is commonplace. Our system HPMMAP provides lightweight memory performance transparently to HPC applications by bypassing Linux's memory management layer. Using HPMMAP, HPC applications achieve consistent performance while the same local compute nodes execute competing workloads likely to be found in HPC clusters and “in-situ” workload deployments. Our approach is dynamically configurable at runtime, and requires no resources when not in use. We show that HPMMAP can decrease variance and reduce application runtime by up to 50 percent when executing a co-located competing commodity workload. <s> BIB001 </s> Survey of Memory Management Techniques for HPC and Cloud Computing <s> D. PARTITIONS <s> Exascale systems are expected to feature hundreds of thousands of compute nodes with hundreds of hardware threads and complex memory hierarchies with a mix of on-package and persistent memory modules. In this context, the Argo project is developing a new operating system for exascale machines. Targeting production workloads using workflows or coupled codes, we improve the Linux kernel on several fronts. We extendthe memory management of Linux to be able to subdivide NUMA memory nodes, allowing better resource partitioning among processes running on the same node. We also add support for memory-mapped access tonode-local, PCIe-attached NVRAM devices and introduce a new scheduling class targeted at parallel runtimes supporting user-level load balancing. These features are unified into compute containers, a containerization approach focused on providing modern HPC applications with dynamic control over a wide range of kernel interfaces. To keep our approach compatible with industrial containerization products, we also identifycontentions points for the adoption of containers in HPC settings. Each NodeOS feature is evaluated by using a set of parallel benchmarks, miniapps, and coupled applications consisting of simulation and data analysis components, running on a modern NUMA platform. We observe out-of-the-box performance improvements easily matching, and often exceeding, those observed with expert-optimized configurations on standard OS kernels. Our lightweight approach to resource management retains the many benefits of a full OS kernel that application programmers have learned to depend on, at the same time providing a set of extensions that can be freely mixed and matched to best benefit particular application components. <s> BIB002
Adapting the Linux MM in order to subdivide NUMA memory nodes,thus improving the partitioning of resources among processes running on the same node, was done by BIB002 . They extend the Linux kernel with additional resource control capabilities based on control groups (cgroups) to maximise resource utilisation with HPC workloads. Linux lacks control over physical memory location, making it non-effective for deeply heterogeneous hierarchical memory. To overcome this limitation, physical NUMA nodes were partitioned into arbitrary logical blocks at a user-defined granularity and they were then presented to the rest of the system as separate NUMA nodes. High Performance Memory Mapping and Allocation Platform based on bypassing Linux memory management presented in BIB001 allow to dynamically partition a node physical memory and independently manage partitions in a separate and isolated resource management layer. HPC applications running on a consolidated platform use a specialised lightweight memory management framework explicitly designed around the requirements of HPC applications.