We present results from our cosmological N-body simulation which consisted of 2048x2048x2048 particles and ran distributed across three supercomputers throughout Europe. The run, which was performed as the concluding phase of the Gravitational Billion Body Problem DEISA project, integrated a 30 Mpc box of dark matter using an optimized Tree Particle Mesh N-body integrator. We ran the simulation up to the present day (z=0), and obtained an efficiency of about 0.93 over 2048 cores compared to a single supercomputer run. In addition, we share our experiences on using multiple supercomputers for high performance computing and provide several recommendations for future projects.
There are a several other projects which have run high performance computing applications across multiple supercomputers. These include simulations of a galaxy collision @cite_20 , a materials science problem @cite_1 as well as an analysis application for arthropod evolution @cite_22 . A larger number of groups performed distributed computing across sites of PCs rather than supercomputers (e.g., @cite_4 @cite_7 @cite_15 ). Several software tools have been developed to facilitate high performance computing across sites of PCs (e.g., @cite_3 @cite_5 @cite_18 @cite_9 @cite_12 ) and within volatile computing environments @cite_17 . The recently launched MAPPER EU-FP7 project @cite_8 seeks to run multiscale applications across a distributed supercomputing environment, where individual subcodes periodically exchange information and (in some cases) run concurrently on different supercomputing architectures.
@cite_18 @cite_4 @cite_22 @cite_7 @cite_8 @cite_9 @cite_1 @cite_3 @cite_5 @cite_15 @cite_12 @cite_20 @cite_17
1825216778 2024343295 2169736303 2059715946 2004261242 204986382 2077783617 1996254287 2153626282 2088186136 1985967703 2008170189
A large number of MPI implementations are currently available, each of which emphasize different aspects of high-performance computing or are intended to solve a specific research problem. The result is a myriad of incompatible MPI implementations, all of which require separate installation, and the combination of which present significant logistical challenges for end users. Building upon prior research, and influenced by experience gained from the code bases of the LAM MPI, LA-MPI, and FT-MPI projects, Open MPI is an all-new, production-quality MPI-2 implementation that is fundamentally centered around component concepts. Open MPI provides a unique combination of novel features previously unavailable in an open-source, production-quality implementation of MPI. Its component architecture provides both a stable platform for third-party research as well as enabling the run-time composition of independent software add-ons. This paper presents a high-level overview the goals, design, and implementation of Open MPI. We discuss the performance of direct summation codes used in the simulation of astrophysical stellar systems on highly distributed architectures. These codes compute the gravitational interaction among stars in an exact way and have an O(N^2) scaling with the number of particles. They can be applied to a variety of astrophysical problems, like the evolution of star clusters, the dynamics of black holes, the formation of planetary systems, and cosmological simulations. The simulation of realistic star clusters with sufficiently high accuracy cannot be performed on a single workstation but may be possible on parallel computers or grids. We have implemented two parallel schemes for a direct N-body code and we study their performance on general purpose parallel computers and large computational grids. We present the results of timing analyzes conducted on the different architectures and compare them with the predictions from theoretical models. We conclude that the simulation of star clusters with up to a million particles will be possible on large distributed computers in the next decade. Simulating entire galaxies however will in addition require new hybrid methods to speedup the calculation. Maximum likelihood analysis is a powerful technique for inferring evolutionary histories from genetic sequence data. During the fall of 2003, an international team of computer scientists, biologists, and computer centers created a global grid to analyze the evolution of hexapods (arthropods with six legs). We created a global grid of computers using systems located in eight countries, spread across six continents (every continent but Antarctica). This work was done as part of the SC03 HPC challenge, and this project was given an HPC challenge award for the "most distributed application". More importantly, the creation of this computing grid enabled investigation of important questions regarding the evolution of arthropods - research that would not have otherwise been undertaken. Grid computing thus leads directly to new scientific insights. This paper argues that computational grids can be used for far more types of applications than just trivially parallel ones. Algorithmic optimizations like latency-hiding and exploiting locality can be used effectively to obtain high performance on grids, despite the relatively slow wide-area networks that connect the grid resources. Moreover, the bandwidth of wide-area networks increases rapidly, allowing even some applications that are extremely communication intensive to run on a grid, provided the underlying algorithms are latency-tolerant. We illustrate large-scale parallel computing on grids with three example applications that search large state spaces: transposition-driven search, retrograde analysis, and model checking. We present several performance results on a state-of-the-art computer science grid (DAS-3) with a dedicated optical network. Large scale supercomputing applications typically run on clusters using vendor message passing libraries, limiting the application to the availability of memory and CPU resources on that single machine. The ability to run inter-cluster parallel code is attractive since it allows the consolidation of multiple large scale resources for computational simulations not possible on a single machine, and it also allows the conglomeration of small subsets of CPU cores for rapid turnaround, for example, in the case of high-availability computing. MPIg is a grid-enabled implementation of the Message Passing Interface (MPI), extending the MPICH implementation of MPI to use Globus Toolkit services such as resource allocation and authentication. To achieve co-availability of resources, HARC, the Highly-Available Resource Co-allocator, is used. Here we examine two applications using MPIg: LAMMPS (Large-scale Atomic Molecular Massively Parallel Simulator), is used with a replica exchange molecular dynamics approach to enhance binding affinity calculations in HIV drug research, and HemeLB, which is a lattice-Boltzmann solver designed to address fluid flow in geometries such as the human cerebral vascular system. The cross-site scalability of both these applications is tested and compared to single-machine performance. In HemeLB, communication costs are hidden by effectively overlapping non-blocking communication with computation, essentially scaling linearly across multiple sites, and LAMMPS scales almost as well when run between two significantly geographically separated sites as it does at a single site. The advanced networking department at Sandia National Laboratories has used the annual Supercomputing conference sponsored by the IEEE and ACM for the past several years as a forum to demonstrate and focus communication and networking developments. At Supercomputing 96, for the first time, Sandia National Laboratories, Los Alamos National Laboratory, and Lawrence Livermore National Laboratory combined their Supercomputing 96 activities within a single research booth under the ASO banner. Sandia provided the network design and coordinated the networking activities within the booth. At Supercomputing 96, Sandia elected: to demonstrate wide area network connected Massively Parallel Processors, to demonstrate the functionality and capability of Sandia s new edge architecture, to demonstrate inter-continental collaboration tools, and to demonstrate ATM video capabilities. This paper documents those accomplishments, discusses the details of their implementation, and describes how these demonstrations support Sandia s overall strategies in ATM networking. Application development for distributed-computing "Grids" can benefit from tools that variously hide or enable application-level management of critical aspects of the heterogeneous environment. As part of an investigation of these issues, we have developed MPICH-G2, a Grid-enabled implementation of the Message Passing Interface (MPI) that allows a user to run MPI programs across multiple computers, at the same or different sites, using the same commands that would be used on a parallel computer. This library extends the Argonne MPICH implementation of MPI to use services provided by the Globus Toolkit for authentication, authorization, resource allocation, executable staging, and I O, as well as for process creation, monitoring, and control. Various performance-critical operations, including startup and collective operations, are configured to exploit network topology information. The library also exploits MPI constructs for performance management; for example, the MPI communicator construct is used for application-level discovery of, and adaptation to, both network topology and network quality-of-service mechanisms. We describe the MPICH-G2 design and implementation, present performance results, and review application experiences, including record-setting distributed simulations. Distributed computing is a means to overcome the limitations of single computing systems. In this paper we describe how clusters of heterogeneous supercomputers can be used to run a single application or a set of applications. We concentrate on the communication problem in such a configuration and present a software library called PACX-MPI that was developed to allow a single system image from the point of view of an MPI programmer. We describe the concepts that have been implemented for heterogeneous clusters of this type and give a description of real applications using this library. The concept of topology-aware grid applications is derived from parallelized computational models of complex systems that are executed on heterogeneous resources, either because they require specialized hardware for certain calculations, or because their parallelization is flexible enough to exploit such resources. Here we describe two such applications, a multi-body simulation of stellar evolution, and an evolutionary algorithm that is used for reverse-engineering gene regulatory networks. We then describe the topology-aware middleware we have developed to facilitate the modeling-implementing-executing'' cycle of complex systems applications. The developed middleware allows topology-aware simulations to run on geographically distributed clusters with or without firewalls between them. Additionally, we describe advanced coallocation and scheduling techniques that take into account the applications topologies. Results are given based on running the topology-aware applications on the Grid'5000 infrastructure. While computational grids with multiple batch systems (batch grids) have been used for efficient executions of loosely-coupled and workflow-based parallel applications, they can also be powerful infrastructures for executing long-running multi-component parallel applications. In this paper, we have constructed a generic middleware framework for executing long-running multi-component applications with execution times much greater than execution time limits of batch queues. Our framework coordinates the distribution, execution, migration and restart of the components of the application on the multiple queues, where the component jobs of the different queues can have different queue waiting and startup times. We have used our framework with a foremost long-running multi-component application for climate modeling, the Community Climate System Model (CCSM). We have performed real multiple-site CCSM runs for 6.5 days of wallclock time spanning three sites with four queues and emulated external workloads. Our experiments indicate that multi-site executions can lead to good throughput of application execution. As a part of the Supercomputing '95 High Performance Computing Challenge, the authors have used the re sources of three of the National Science Foundation NSF supercomputing centers as a single, distributed, heteroge neous metacomputer to carry out the largest simulation of colliding galaxies yet attempted. The metacomputer con sisted of the following machines: TMC CM-5, Cray C90 T3D, IBM SP-2, and SGI Power Challenge. This paper describes the scalable parallel numerical algorithms used to carry out the hybrid N-body gas dynamical simulation, as well as the inter-MPP communication system the authors developed to connect them. The system, called sclib, is simple yet flexible, and consists of a network of communicating objects, or agents. Each communication agent is a process that resides on the front-end processor of a parallel supercomputer and brokers the interactions between the various components of the distributed com putation. The design is independent of host architecture and execution model yet takes advantage of scheduling resources local to the compute servers. As such, it serves as a useful prototype for future research in distributed, heterogeneous, high-performance computing. Parallel computing on volatile distributed resources requires schedulers that consider job and resource characteristics. We study unconventional computing environments containing devices spread throughout a single large organization. The devices are not necessarily typical general purpose machines; instead, they could be processors dedicated to special purpose tasks (for example printing and document processing), but capable of being leveraged for distributed computations. Harvesting their idle cycles can simultaneously help resources cooperate to perform their primary task and enable additional functionality and services. A new burstiness metric characterizes the volatility of the high-priority native tasks. A burstiness-aware scheduling heuristic opportunistically introduces grid jobs (a lower priority workload class) to avoid the higher-priority native applications, and effectively harvests idle cycles. Simulations based on real workload traces indicate that this approach improves makespan by an average of 18.3 over random scheduling, and comes within 7.6 of the theoretical upper bound.