Big PanDA Workflow Management on Titan for High Energy and Nuclear Physics and for Future Extreme Scale Scientific Applications
Thrust Area
OLCF Groups:
Algorithms & Performance Analysis
Data Lifecycle & Scalable Workflows
Staff Contributors:
Jason Kincl, Don Maxwell, Valentine Anantharaj
This project will provide an important model for future exascale computing, increasing the coherence between the technology base used for high-performance computing and that used for data-analytic computing, and thereby directly advancing the first of the five objectives of the National Strategic Computing Initiative announced by the White House in June 2015. This proposal also directly advances the first of five strategic themes articulated by the White House Office of Science and Technology Policy (OSTP) to "create systems that can apply exaflops of computing power to exabytes of data."
Achieving operational goals are important steps towards enabling the convergence of HPC and HTC "paradigms" for a range of application types and communities. In meeting the operational goals of this demonstrator project, we have both developed practical experience as well conceptual insight into some of the challenges which will ultimately help ASCR devise a working plan of engagement and support with science applications from other DOE offices. These include but are not limited to supporting diverse use-cases and requirements, sharing workloads across HTC and HPC resources, and investigating the operational implications of running traditional HTC workloads on HPC resources.
This project is a partnership with Brookhaven National Laboratory (PI, Alexei Klimentov), University of Texas at Arlington (Prof. Kaushik De), and Rutgers University (Prof. Shantenu Jha). The funding for these partners comes from DOE/SC/ASCR Next Generation Networking for Science (NGNS), Rich Carlson program manager. The goal of this project is to translate the R&D artifacts and accomplishments from the BigPanDA and AIMES projects into LCF operational advances and enhancement, each of which previously received funding from the DOE/SC's NGNS program.