Two-phase I O is a well-known strategy for implementing collective MPI-IO functions. It redistributes I O requests among the calling processes into a form that minimizes the file access costs. As modern parallel computers continue to grow into the exascale era, the communication cost of such request redistribution can quickly overwhelm collective I O performance. This effect has been observed from parallel jobs that run on multiple compute nodes with a high count of MPI processes on each node. To reduce the communication cost, we present a new design for collective I O by adding an extra communication layer that performs request aggregation among processes within the same compute nodes. This approach can significantly reduce inter-node communication congestion when redistributing the I O requests. We evaluate the performance and compare with the original two-phase I O on a Cray XC40 parallel computer with Intel KNL processors. Using I O patterns from two large-scale production applications and an I O benchmark, we show the performance improvement of up to 29 times when running 16384 MPI processes on 256 compute nodes.
Many MPI libraries such as MPICH @cite_26 and OpenMPI @cite_3 adopt ROMIO as the implementation for MPI-IO functions. For parallel jobs running multiple MPI processes per compute nodes, ROMIO selects one aggregator per node in its default settings. The selection of I O aggregators is implemented at file open time. If the underneath file system is a Lustre, the default setting changes to select the number of I O aggregators equal to the Lustre file striping count. This strategy produces a ono-to-one mapping between the aggregators and the file servers (OSTs) which avoids any possible file lock conflicts and achieves the best I O performance @cite_21 . When the aggregate file access region is larger than the number of aggregators times the file stripe size, the two-phase I O is carried out in multiple rounds.
@cite_26 @cite_21 @cite_3
2151436730 2104486653 2174300520 2567936601
Client-side file caching has long been recognized as a file system enhancement to reduce the amount of data transfer between application processes and I O servers. However, caching also introduces cache coherence problems when a file is simultaneously accessed by multiple processes. Existing coherence controls tend to treat the client processes independently and ignore the aggregate I O access pattern. This causes a serious performance degradation for parallel I O applications. In this paper we discuss our new implementation and present an extended performance evaluation on GPFS and Lustre parallel file systems. In addition to comparing our methods to traditional approaches, we examine the performance of MPI-IO caching under direct I O mode to bypass the underlying file system cache. We also investigate the performance impact of two file domain partitioning methods to MPI collective I O operations: one which creates a balanced workload and the other which aligns accesses to the file system stripe size. In our experiments, alignment results in better performance by reducing file lock contention. When the cache page size is set to a multiple of the stripe size, MPI-IO caching inherits the same advantage and produces significantly improved I O bandwidth. We discuss the issues involved in implementing MPI-IO portably on multiple machines and file systems and also achieving high performance. One way to implement MPI-IO portably is to implement it on top of the basic Unix I O functions (open, lseek, read, write, and close), which are themselves portable. We argue that this approach has limitations in both functionality and performance. We instead advocate an implementation approach that combines a large portion of portable code and a small portion of code that is optimized separately for different machines and file systems. We have used such an approach to develop a high-performance, portable MPI-IO implementation, called ROMIO. In addition to basic I O functionality, we consider the issues of supporting other MPI-IO features, such as 64-bit file sizes, noncontiguous accesses, collective I O, asynchronous I O, consistency and atomicity semantics, user-supplied hints, shared file pointers, portable data representation, and file preallocation. We describe how we implemented each of these features on various machines and file systems. The machines we consider are the HP Exemplar, IBM SP, Intel Paragon, NEC SX-4, SGI Origin2000, and networks of workstations; and the file systems we consider are HP HFS, IBM PIOFS, Intel PFS, NEC SFS, SGI XFS, NFS, and any general Unix file system (UFS). We also present our thoughts on how a file system can be designed to better support MPI-IO. We provide a list of features desired from a file system that would help in implementing MPI-IO correctly and with high performance. The I O access patterns of parallel programs often consist of accesses to a large number of small, noncontiguous pieces of data. If an application's I O needs are met by making many small, distinct I O requests, however, the I O performance degrades drastically. To avoid this problem, MPI-IO allows users to access a noncontiguous data set with a single I O function call. This feature provides MPI-IO implementations an opportunity to optimize data access. We describe how our MPI-IO implementation, ROMIO, delivers high performance in the presence of noncontiguous requests. We explain in detail the two key optimizations ROMIO performs: data sieving for noncontiguous requests from one process and collective I O for noncontiguous requests from multiple processes. We describe how one can implement these optimizations portably on multiple machines and file systems, control their memory requirements, and also achieve high performance. We demonstrate the performance and portability with performance results for three applications--an astrophysics-application template (DIST3D), the NAS BTIO benchmark, and an unstructured code (UNSTRUC)--on five different parallel machines: HP Exemplar, IBM SP, Intel Paragon, NEC SX-4, and SGI Origin2000. ROMIO is a high-performance, portable implementation of MPI-IO (the I O chapter in MPI-2). This document describes how to install and use ROMIO version 1.0.0 on the following machines: IBM SP; Intel Paragon; HP Convex Exemplar; SGI Origin 2000, Challenge, and Power Challenge; and networks of workstations (Sun4, Solaris, IBM, DEC, SGI, HP, FreeBSD, and Linux).