1. Field of the Invention
This invention is directed to the field of large data input/output (I/O) transfer into, and out of, large compute systems and large data storage systems. It is particularly directed towards reducing the data access latency times and increasing the data I/O transfer rates for compute clients processing extremely large data sets within large, shared-memory compute systems.
2. Description of Prior Art
Demand is growing to find solutions to the “big data” challenge faced by government and business, for example, in areas such as fraud detection, remote sensor data analysis, financial modeling, social network analysis, and risk analysis. Many data intensive computing applications need to process an enormous amount of data in a timely manner, and if too much time is spent loading a required data set into a compute client's memory, then the entire production workflow will be delayed. The same logic holds for processes that are generating very large data sets. The newly generated large data sets need to be quickly made available for the next step in the larger production workflow. To support such an environment, a supercomputer architecture is required that incorporates hundreds or thousands of CPU cores for handling multiple processing threads and supersized memories to hold more of the very large data sets. Simply put, the more data the supercomputer can process at any given time, the quicker the results are presented to decision makers. When it comes to mission-critical pattern discovery problems in government and business, whoever obtains the answers first is one step ahead of their adversary or competitor.
The challenges to meeting “big data” demands are many:
The compute hardware needs to be able to scale up to an extraordinary level—thousands of CPU cores and multiple terabytes of memory—which is far beyond the capacity of any commodity X86 server. Computer software needs to be able to take advantage of the scaled-up hardware platform. And, an extremely efficient data access input/output system that can quickly load the “big data” into system memory and just as quickly store the processed data onto reliable media is needed to complete the system.
The ability to build larger and larger compute systems has been facilitated by advances in massively parallel compute system techniques, algorithmic resource orchestration techniques (Map Reduce and others), low-cost/high-performance CPU's, and the advent of both open-source and proprietary Parallel File Systems. Parallel File Systems such as “Lustre” [See “Lustre scalable storage” Copyright 2006 Cluster File Systems —Rights owned by ORACLE Corp.], and IBM's GPFS (General Parallel File System) [Refer to F. B. Schmuck and R. L Haskin, GPFS: a shared-disk file system for large computing clusters, in proceedings of Conference of Files and Storage Technologies (FAST'02), 2002] have allowed these newer and larger parallel compute systems to scale up to higher and higher numbers of compute nodes and compute-cores and still have enough File System I/O Bandwidth to allow for very good compute performance on many classes of applications.
Parallel File Systems generate very high “Total I/O” Bandwidth levels by utilizing multiple “File System Data Storage Servers” like the dual socket server 110 in FIG. 1 (Lustre uses “Object Storage Servers (OSS's) and IBM's GPFS uses NSD Servers (Network Shared Disk Servers)). Each File System Data Storage Server (OSS or NSD Server) is attached to one or more “Data Storage Devices,” like 109 in FIG. 1, via a storage network fabric 106 that in many HPC environments is implemented using Infiniband (IB) switches and (IB) Host Channel Adapters (HCA's) 104. The “Data Storage Device” 109 often consists of a “Storage Controller” 107, and a High-Performance or High-Capacity Disk Storage Devices 108 or Solid-State Disk (SSD) Storage Devices (many new systems are using SSDs in combination with SATA or SAS Disks to balance total I/O operations with total storage capacity). Sometimes “File System Data Storage Servers,” like 110 in FIG. 1, are paired with Storage Subsystems, like 109 in FIG. 1, that are implemented as raw “Trays-of-Disks” without Disk-Controllers. These Trays-of-Disks are referred to as JBOD's (just a Bunch Of Disks), and require additional Data-Block management software to run within the “File System Data Storage Server” 110 to accomplish the functionality required for the “JBOD” to serve as a suitable Storage Device 109. The “Zetta File System,” or ZFS [open source licensed By Oracle Corp.], is one candidate application for handling JBOD's and is being utilized by the Lawrence Livermore National Labs (LLNL) team, in combination with the Lustre File System as a part of their “Next-Generation” Sequoia Super Computer installation which hopes to have a “Total File System I/O Bandwidth of between 500 GB/s and 1000 GB/s.
The data blocks in the Parallel File System are “Striped” or spread across multiple “File System Data Storage Servers,” like the 111 grouping of FIG. 1, and their associated collective sets of Storage Devices 113 in FIG. 1. The sequence of events for a compute client to receive the data stored in the Parallel File Systems after a client data request is:
1—The data blocks are copied from “Disk Storage Devices —108 data storage locations” and into “Storage Controllers 107 memory—Data Movement #1,
2—The data blocks are then copied from “Storage Controller 107 memory” into “File System Data Storage Servers 110 memory”—Data Movement #2,
3—The data blocks are then copied from all of the “File System Data Storage Servers 110 memory” into the client 101-1 memory space—Data Movement #3.
Moving the data blocks three times before they can be utilized is very inefficient, but for many parallel applications the inefficiency is accepted as current practice and the job schedules are allocated by compute time required as well as data I/O transfer time. All of these “Data Movement processes” are running in parallel, simultaneously across all of the File System Data Storage Servers and their respective Storage Controllers and Disk Storage Devices. The aggregate data I/O speed for the entire File System is a function of the available bandwidth for all of the File System Data Storage Servers operating in unison. The grouping of File System Data Storage Servers 111 in FIG. 1 has the potential I/O bandwidth of 8 GB/s because each of the 4 servers can maintain 2 GB/s and the set of 2 disk Storage Devices, 109 in FIG. 1, can each maintain 4 GB/s.
The total I/O performance of the Parallel File Systems can be scaled up by adding more File System Data Storage Servers, see 201 in FIG. 2, and more Disk Storage Devices, 202 in FIG. 2. The additional server and storage resources allow the example Parallel File System in FIG. 2 to provide an aggregate potential data I/O bandwidth of 20 GB/s instead of the original potential of I/O speed of 8 GB/s. The combination of adding more Storage Servers and more Storage Devices creates a very high Total potential I/O Bandwidth solution for use by the massively parallel compute systems and their applications.
The use of Parallel File Systems, like Lustre and GPFS, for massively parallel compute systems has worked well because each compute node in the massively parallel compute system typically needs to use only a small portion of the I/O Bandwidth to accomplish its compute tasks and to deliver intermediate or final results. The High Speed Parallel File Systems in use at Large Supercomputing Centers are used to take very rapid “Snap-shots” of intermediate compute results for complex compute jobs that can sometimes take days or weeks to complete. The snapshots are called “Check Points” and they require very High Total I/O Bandwidth since each of the many thousands of compute nodes are sending copies of their intermediate compute results out for analysis and algorithmic tuning.
One of the HPC systems at Oak Ridge National Laboratory, Jaguar, has over 26,000 compute nodes with approximately 180,000 compute-cores, and its Lustre Parallel File Systems operate at 240 GB/s and 44 GB/s. That meant that each “File System Client”, like 101-2 in FIG. 1, on a small compute node, could “Simultaneously Share” about 9-12 MB/s on average of the “Total File System I/O,” when all of the many thousands of compute nodes were working together in parallel on one large problem. The peak File System I/O performance for a single specific “File System Client,” like 101-2 in FIG. 1, on a Jaguar compute node was just over 1.25 GB/s. This means that the peak “File System Client” I/O Bandwidth performance available for a single node on Jaguar is 0.5% of the “Total I/O” Bandwidth available from the entire File System. This level of I/O performance works very well for the large scale scientific simulation problems that have been tailored to work on the massively scaled parallel compute systems like “jaguar” with it's thousands of separate compute nodes that typically begin processing jobs with a few small-sized data sets for each compute node. But there are many additional types of compute problems that require the manipulation and processing of very large data sets that must first be loaded into the memory space of one compute system before processing can begin.
Many US Government projects have requirements to rapidly access and process large numbers of files in the 5 GB to 15 GB range. There are many fielded systems are producing data sets at a rate of 1 Tera Byte (TB) or more per hour, and there are several ongoing projects that produce data sets that range from 4 to 6 TB in size. The users of these large, Multi-TB data sets would like to have the ability to rapidly load and process entire multi-TB files in “Large Shared Memory” compute systems, like 112 in FIGS. 1 and 2, and utilize the compute systems 100's or 1000's of compute-cores to reduce the “Raw multi-TB” data sets into useful, user friendly, result sets. The task of crafting algorithms to process these Very Large Multi-TB data sets is much easier to accomplish if the entire data set can fit within the internal memory address space of a large “Common Global Shared Memory” compute system and be acted on by all of the 100's or 1000's of compute-cores resident with the single large “Common Global Shared Memory” compute system.
An example of using a “Common Global Shared Memory” compute system in combination with a typical Parallel File System to process very large data sets can be further examined by using the performance characteristics of Oak Ridge's “Jaguar” Supercomputer. Jaguar's “Peak I/O Performance” for a single compute node was 1.25 GB/s. Using that I/O Bandwidth value for the I/O performance of a single “File System Client” within a “Large Shared Memory” compute system, such as 112 in FIG. 1 or 2, results in a 15 GB file loading into the “Large Shared Memory” Compute system in just 12 seconds. An acceptable data I/O transfer time for many situations. But the time required to load a 6 TB file from a modular storage device or a current implementation of a typical Parallel File System, 111 and 201 and 202 from FIG. 2, into the same large “Common Global Shared Memory” compute system, 101-1 in 112 of FIG. 2, would be at least 4,800 seconds! Having 100's or 1000's of processing cores waiting over 4,000 seconds for a data set to load into a system like 112 in FIG. 1 or 2, would be a terrible waste of an expensive compute resource.
The “Peak I/O bandwidth” available for a specific application compute client from the total aggregate I/O provided by the Parallel File Systems' entire collection of “File System's Data Storage Servers,” like 111 and 201 added together in FIG. 2, can be limited by many of the physical and logical elements that interconnect a typical Lustre or GPFS Parallel File System implementation.
A significant implementation element that can limit the “I/O Bandwidth Performance” for a specific “File Systems' Client” is the total number, and bandwidth capacity, of the physical or logical I/O network pathways available to link the specific “Application Compute Client” with the “Data Storage Servers” of the Parallel File System. The designs and implementations of “Application Compute Clients” will vary across the range of commercially available Parallel File Systems. The number of I/O network pathways supported by a specific “Application compute client” implementation, combined with the I/O bandwidth available for “Application compute client” use within each supported I/O network pathway, will significantly affect the total I/O Bandwidth for the specific “File System Client.”
The physical linkage between the “Application compute client” processes 101-1 and the collective set of “File System Data Storage Servers” 111 and 201, is depicted in FIGS. 1 and 2 with the HCA's 104 supporting the external I/O connectivity from the 101-1 Client within the large “Common Global Memory Compute System,” 112, and the Infiniband Switch 105 that in-turn provides the I/O network pathway connectivity to the entire collection of “File System Data Storage Servers” 111 and 201 in FIG. 2.
One specific current example of how a Parallel File System's design and implementation of its “Application compute client” can limit the total amount of File System I/O Bandwidth available to a specific “Application compute client” is Lustre's current Client design and implementation. As currently implemented, a Lustre Client can have one physical Infiniband connection. This means that a QDR (Quad Data Rate=4 GB/s) or DDR (Double Data Rate=2 GB/s) IB connection between a Lustre File System Client 101-1 and HCA 104 in FIG. 2 over to the 105 IB switch, would be the only I/O pathway between the Lustre File System Client and the collection of “File System Data Storage Servers” (OSS's for Lustre). Current Lustre Client instances utilizing a single QDR IB connection as its I/O network pathway to the “File System Data Storage Servers” have been able to achieve sustained I/O rates of up to 3.2 GB/s after considerable tuning adjustments were made, and while the Lustre Client was servicing several separate processes within a “Large Shared Memory” compute system with data blocks from several unique files that were striped across all of the “File System Data Storage Servers” (OSS servers) within the specific Lustre File System.
A redesign and re-implementation of the Lustre Client software would be required to permit the Lustre “File System Client” process to utilize two or more physical IB connections as its I/O network pathway to the “File System Data Storage Servers.” An implementation that allowed a Lustre File System Client to utilize two QDR IB connections would potentially be able to achieve I/O bandwidth rates up to a maximum of 6 or 7 GB/s, but even this level of File System Client I/O Bandwidth performance is still a very small percentage of the total I/O Bandwidth available from Parallel File Systems that can achieve 100's of GB/s of total I/O Bandwidth. If the File System Client 101-1 in 112 of FIG. 2 was able to operate at 6 GB/s of I/O Bandwidth, it would still be utilizing less than ⅓ of the total potential File System I/O of 20 GB/s derived from the Parallel File System's “File System Data Storage Servers” in 111 and 201 in FIG. 2.
Existing Parallel File Systems (like Lustre and GPFS) perform well at distributing a small share of the total file system I/O to the thousands of compute nodes in today's large supercomputing compute clusters. However, for cases where very large data files need to be quickly loaded into a single, multi-processor compute node, the existing Parallel File Systems are not capable of providing more than a very small percentage of the total I/O Bandwidth to an individual compute client. A more efficient use of the Large Shared Memory Compute System, 112 in FIG. 2, would be had if there were a way for all of the Parallel File Systems aggregate I/O to be available for the compute client or clients within the Large Shared Memory Compute system.
Therefore, there is a need for a solution to improve the “Peak I/O Bandwidth Performance” for a single “File System Client,” 101-1 in FIGS. 1 and 2, in a large “Common Global Shared Memory” Compute System, 112 in FIGS. 1 and 2.