Dataset storage, exchange, and access play a critical role in scientific applications. For such purposes netCDF serves as a portable and efficient file format and programming interface, which is popular in numerous scientific application domains. However, the original interface does not provide an efficient mechanism for parallel data storage and access. In this work, we present a new parallel interface for writing and reading netCDF datasets. This interface is derived with minimum changes from the serial netCDF interface but defines semantics for parallel access and is tailored for high performance. The underlying parallel I O is achieved through MPI-IO, allowing for dramatic performance gains through the use of collective I O optimizations. We compare the implementation strategies with HDF5 and analyze both. Our tests indicate programming convenience and significant I O performance improvement with this parallel netCDF interface.
MPI-IO is a parallel I O interface specified in the MPI-2 standard. It is implemented and used on a wide range of platforms. The most popular implementation, ROMIO @cite_13 is implemented portably on top of an abstract I O device layer @cite_1 @cite_22 that enables portability to new underlying I O systems. One of the most important features in ROMIO is collective I O operations, which adopt a two-phase I O strategy @cite_11 @cite_20 @cite_18 @cite_2 and improve the parallel I O performance by significantly reducing the number of I O requests that would otherwise result in many small, noncontiguous I O requests. However, MPI-IO reads and writes data in a raw format without providing any functionality to effectively manage the associated metadata. Nor does it guarantee data portability, thereby making it inconvenient for scientists to organize, transfer, and share their application data.
@cite_18 @cite_22 @cite_1 @cite_2 @cite_13 @cite_20 @cite_11
2082416889 2104486653 2111925167 2174300520 2567936601 2108155100 2083200599
A number of applications on parallel computers deal with very large data sets that cannot fit in main memory. In such applications, data must be stored in files on disks and fetched into memory during program execution. Parallel programs with large out-of-core arrays stored in files must read write smaller sections of the arrays from to files. In this article, we describe a method for accessing sections of out-of-core arrays efficiently. Our method, the extended two-phase method, uses collective l O: Processors cooperate to combine several l O requests into fewer larger granularity requests, to reorder requests so that the file is accessed in proper sequence, and to eliminate simultaneous l O requests for the same data. In addition, the l O workload is divided among processors dynamically, depending on the access requests. We present performance results obtained from two real out-of-core parallel applications - matrix multiplication and a Laplace's equation solver - and several synthetic access patterns, all on the Intel Touchstone Delta. These results indicate that the extended two-phase method significantly outperformed a direct (noncollective) method for accessing out-of-core array sections. We discuss the issues involved in implementing MPI-IO portably on multiple machines and file systems and also achieving high performance. One way to implement MPI-IO portably is to implement it on top of the basic Unix I O functions (open, lseek, read, write, and close), which are themselves portable. We argue that this approach has limitations in both functionality and performance. We instead advocate an implementation approach that combines a large portion of portable code and a small portion of code that is optimized separately for different machines and file systems. We have used such an approach to develop a high-performance, portable MPI-IO implementation, called ROMIO. In addition to basic I O functionality, we consider the issues of supporting other MPI-IO features, such as 64-bit file sizes, noncontiguous accesses, collective I O, asynchronous I O, consistency and atomicity semantics, user-supplied hints, shared file pointers, portable data representation, and file preallocation. We describe how we implemented each of these features on various machines and file systems. The machines we consider are the HP Exemplar, IBM SP, Intel Paragon, NEC SX-4, SGI Origin2000, and networks of workstations; and the file systems we consider are HP HFS, IBM PIOFS, Intel PFS, NEC SFS, SGI XFS, NFS, and any general Unix file system (UFS). We also present our thoughts on how a file system can be designed to better support MPI-IO. We provide a list of features desired from a file system that would help in implementing MPI-IO correctly and with high performance. We propose a strategy for implementing parallel I O interfaces portably and efficiently. We have defined an abstract device interface for parallel I O, called ADIO. Any parallel I O API can be implemented on multiple file systems by implementing the API portably on top of ADIO, and implementing only ADIO on different file systems. This approach simplifies the task of implementing an API and yet exploits the specific high performance features of individual file systems. We have used ADIO to implement the Intel PFS interface and subsets of MPI-IO and IBM PIOFS interfaces on PFS, PIOFS, Unix, and NFS file systems. Our performance studies indicate that the overhead of using ADIO as an implementation strategy is very low. The I O access patterns of parallel programs often consist of accesses to a large number of small, noncontiguous pieces of data. If an application's I O needs are met by making many small, distinct I O requests, however, the I O performance degrades drastically. To avoid this problem, MPI-IO allows users to access a noncontiguous data set with a single I O function call. This feature provides MPI-IO implementations an opportunity to optimize data access. We describe how our MPI-IO implementation, ROMIO, delivers high performance in the presence of noncontiguous requests. We explain in detail the two key optimizations ROMIO performs: data sieving for noncontiguous requests from one process and collective I O for noncontiguous requests from multiple processes. We describe how one can implement these optimizations portably on multiple machines and file systems, control their memory requirements, and also achieve high performance. We demonstrate the performance and portability with performance results for three applications--an astrophysics-application template (DIST3D), the NAS BTIO benchmark, and an unstructured code (UNSTRUC)--on five different parallel machines: HP Exemplar, IBM SP, Intel Paragon, NEC SX-4, and SGI Origin2000. ROMIO is a high-performance, portable implementation of MPI-IO (the I O chapter in MPI-2). This document describes how to install and use ROMIO version 1.0.0 on the following machines: IBM SP; Intel Paragon; HP Convex Exemplar; SGI Origin 2000, Challenge, and Power Challenge; and networks of workstations (Sun4, Solaris, IBM, DEC, SGI, HP, FreeBSD, and Linux). We are developing a compiler and runtime support system called PASSION: Parallel and Scalable Software for Input-Output. PASSION provides software support for I O intensive out-of-core loosely synchronous problems. This paper gives an overview of the PASSION Runtime Library and describes two of the optimizations incorporated in it, namely data prefetching and data sieving. Performance improvements provided by these optimizations on the Intel Touchstone Delta are discussed together with an out-of-core median filtering application. > As scientists expand their models to describe physical phenomena of increasingly large extent, I O becomes crucial and a system with limited I O capacity can severely constrain the performance of the entire program.We provide experimental results, performed on an lntel Touchtone Delta and nCUBE 2 I O system, to show that the performance of existing parallel I O systems can vary by several orders of magnitude as a function of the data access pattern of the parallel program. We then propose a two-phase access strategy, to be implemented in a runtime system, in which the data distribution on computational nodes is decoupled from storage distribution. Our experimental results show that performance improvements of several orders of magnitude over direct access based data distribution methods can be obtained, and that performance for most data access patterns can be improved to within a factor of 2 of the best performance. Further, the cost of redistribution is a very small fraction of the overall access cost.