A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> P-GRADE portal is one of the most widely used general-purpose grid portal in Europe. The paper summarizes the most advanced features of P-GRADE, such as parameter sweep workflow execution, multi-grid workflow execution and integration with the DSpace workflow repository. It also shows the NGS P-GRADE portal that extends P-GRADE with the GEMLCA legacy code execution support in Grid systems, as well as with coarse-grain workflow interoperability services. Next, the paper introduces the second generation P-GRADE portal called WS-PGRADE that merges the advanced features of the first generation P-GRADE portals and extends them with new workflow and architecture concepts. Finally, the application-specific science gateway of the CancerGrid project is briefly described to demonstrate that application-specific portals can easily be developed on top of the general-purpose WS-PGRADE portal. Copyright © 2010 John Wiley & Sons, Ltd. <s> BIB001 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> In this installment of Trend Wars, we discuss cloud computing and OpenNebula with Ignacio M. Llorente and Ruben S. Montero, who are the principal investigator and the chief architect, respectively, of the open source OpenNebula project. <s> BIB002 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> The WS-PGRADE/gUSE generic DCI gateway framework has been developed to support a large variety of user communities. It provides a generic purpose, workflow-oriented graphical user interface to create and run workflows on various DCIs including clusters, Grids, desktop Grids and clouds. The framework can be used by NGIs to support small user communities who cannot afford to develop their own customized science gateway. The WS-PGRADE/gUSE framework also provides two API interfaces (Application Specific Module API and Remote API) to create application-specific science gateways according to the needs of different user communities. The paper describes in detail the workflow concept of WS-PGRADE, the DCI Bridge service that enables access to most of the popular European DCIs and the Application Specific Module and Remote API concepts to generate application-specific science gateways. <s> BIB003 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> Over 110 effective recipes to help you build and operate OpenStack cloud computing, storage, networking, and automation About This Book Explore many new features of OpenStack's Juno and Kilo releases Install, configure, and administer core projects with the help of OpenStack Object Storage, Block Storage, and Neutron Networking services Harness the abilities of experienced OpenStack administrators and architects, and run your own private cloud successfully Practical, real-world examples of each service and an accompanying Vagrant environment that helps you learn quickly In Detail OpenStack Open Source software is one of the most used cloud infrastructures to support software development and big data analysis. It is developed by a thriving community of individual developers from around the globe and backed by most of the leading players in the cloud space today. It is simple to implement, massively scalable, and can store a large pool of data and networking resources. OpenStack has a strong ecosystem that helps you provision your cloud storage needs. Add OpenStack's enterprise features to reduce the cost of your business. This book will show you the steps to build up a private cloud environment. At the beginning, you'll discover the uses of cloud services such as the identity service, image service, and compute service. You'll dive into Neutron, the OpenStack Networking service, and get your hands dirty with configuring ML2, networks, routers, and Distributed Virtual Routers. You'll then gather more expert knowledge on OpenStack cloud computing by managing your cloud's security and migration. After that, we delve in to OpenStack Object storage and how to manage servers and work with objects, cluster, and storage functionalities. Also, as you go deeper into the realm of OpenStack, you'll learn practical examples of Block storage, LBaaS, and FWaaS: installation and configuration covered ground up. Finally, you will learn OpenStack dashboard, Ansible and Foreman, Keystone, and other interesting topics. What You Will Learn Understand, install, configure, and manage Novathe OpenStack Cloud Compute resource Configure ML2, networks, routers, and Distributed Virtual Routers with Neutron Use and secure Keystone, the OpenStack Authentication service Install and set up Swift and Container Replication between datacenters Gain hands-on experience and familiarity with Horizon, the OpenStack Dashboard user interface Automate complete solutions with our recipes on Heat, the OpenStack Orchestration service Use Ansible and Foreman to automate OpenStack installations successfully Follow practical advice and examples to run OpenStack in production Who This Book Is For This book is aimed at cloud system engineers, system administrators, and technical architects who are moving from a virtualized environment to cloud environments. This book assumes that you are familiar with cloud computing platforms, and have knowledge of virtualization, networking, and managing Linux environments. Style and approach Clear, step-by-step instructions coupled with practical and applicable recipes that'll enable you to use and implement the latest features of OpenStack. <s> BIB004 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> Today there exist a wide variety of scientific workflow management systems, each designed to fulfill the needs of a certain scientific community. Unfortunately, once a workflow application has been designed in one particular system it becomes very hard to share it with users working with different systems. Portability of workflows and interoperability between current systems barely exists. In this work, we present the fine-grained interoperability solution proposed in the SHIWA European project that brings together four representative European workflow systems: ASKALON, MOTEUR, WS-PGRADE, and Triana. The proposed interoperability is realised at two levels of abstraction: abstract and concrete. At the abstract level, we propose a generic Interoperable Workflow Intermediate Representation (IWIR) that can be used as a common bridge for translating workflows between different languages independent of the underlying distributed computing infrastructure. At the concrete level, we propose a bundling technique that aggregates the abstract IWIR representation and concrete task representations to enable workflow instantiation, execution and scheduling. We illustrate case studies using two real-workflow applications designed in a native environment and then translated and executed by a foreign workflow system in a foreign distributed computing infrastructure. <s> BIB005 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> The Molecular Simulation Grid (MoSGrid) science gateway developed on top of WS-PGRADE/gUSE targets the needs of the molecular simulation community in a virtual research environment. The existing science gateway infrastructure has been extended on several levels and layers to especially support the computing- and data-intensive research domains of quantum chemistry, molecular dynamics, and docking. On the security level, the MoSGrid project has implemented a granular security concept, which applies Security Assertion Markup Language (SAML) assertions for trust delegations. Support of the highly scalable grid middleware UNICORE has been added to the middleware layer as has the support of the object-based file system XtreemFS for distributed data management. Furthermore, developers are supported by the PortletAPI, which eases the creation of domain-specific portlets. The standardized data exchange format of Molecular Simulation Markup Language (MSML) which sets the stage for the MoSGrid metadata management features, is a main building block for the PortletAPI. <s> BIB006 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> Seismic hazard assessment and risk analysis are critical for human life since these fields study and model expected earthquakes to produce seismic hazard maps and to determine the risk of damage from potential earthquakes to buildings, dams, etc. The Statistical Seismology Science Gateway, built on the gUSE framework, holds a comprehensive set of tools and models covering seismic data analysis, hazard assessment and risk analysis. It provides an environment in such a way that skilled users can experiment their own approaches by simply accessing the gateway services, while novice users can easily use existing applications. This chapter focuses on the development of the workflows implementing these tools and models as well as the portlets to access them. It also discusses how easily gUSE framework was customised to build this science gateway and how it is operated. <s> BIB007 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> The availability of large-scale digital surveys offers tremendous opportunities for advancing scientific knowledge in the astrophysics community. Nevertheless, the analysis of these data often requires very powerful computational resources. This chapter focuses on the development issues to design and implement a science gateway and a mobile application tailored for astrophysics needs by customizing the WS-PGRADE/gUSE technologies. The VisIVO Gateway integrates services for processing and visualizing large-scale multidimensional astrophysical datasets on distributed computing infrastructures. The core tools and services supported, employing the gUSE ASM API are presented. The gateway is exploited by a mobile application called VisIVO Mobile, which allows smartphone devices to perform analysis and visual discovery of large-scale astrophysical datasets. The mobile application configures and submits the VisIVO workflows by means of the gUSE remote API. <s> BIB008 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> Computational neuroscientists face challenges to manage ever-increasing large volume of data and to process them with applications that require great computational power. The Brain Imaging Centre of the Academic Medical Centre of the University of Amsterdam is a community of neuroscientists who are involved in various computational neuroscience research studies. They face various challenges to process and manage a growing amount of neuroimaging data. The goal of the computational neuroscience gateway is to facilitate large-scale data processing on distributed infrastructures and to enhance data management and collaboration for scientific research. The gateway is based on the WS-PGRADE/gUSE generic science gateway framework as platform for distributed computing, and it is connected to a data server based on the eXtensible Neuroimaging Archive Toolkit (XNAT). This chapter presents the design and architecture of the gateway with focus on the utilization of the WS-PGRADE/gUSE framework, and the lessons learned during its implementation and operation. <s> BIB009 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> Various WS-PGRADE/gUSE science gateways have been extensively used in educational contexts, supporting courses offered by different European universities and organizations. This chapter presents some examples of how WS-PGRADE/gUSE generic and customized gateways have been used in such courses. These examples include practical cases from a variety of scientific fields and educational styles. For each case, the educational context and the course organization are presented, with emphasis on how the respective portal has been adopted for the practical exercises. A summary of experiences are also reported, including advantages and difficulties faced for using these gateways in teaching. <s> BIB010 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> Besides core project partners, the SCI-BUS project also supported several external user communities in developing and setting up customized science gateways. The focus was on large communities typically represented by other European research projects. However, smaller local efforts with the potential of generalizing the solution to wider communities were also supported. This chapter gives an overview of support activities related to user communities external to the SCI-BUS project. A generic overview of such activities is provided, followed by the detailed description of three gateways developed in collaboration with European projects: the agINFRA Science Gateway for Workflows for agricultural research, the VERCE Science Gateway for seismology, and the DRIHM Science Gateway for weather research and forecasting. <s> BIB011 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> Although originally an academic and research product, the WS-PGRADE/gUSE framework is increasingly applied by commercial institutions too. Within the SCI-BUS project, several commercial gateways have been developed by various companies. WS-PGRADE/gUSE is also intensively used within another European research project, CloudSME (Cloud-based Simulation Platform for Manufacturing and Engineering). This chapter provides an overview and describes in detail some commercial WS-PGRADE/gUSE-based gateway implementations. Two representative case studies from the SCI-BUS project, the Build and Test portal and the eDOX Archiver Gateway, are introduced. An overview of WS-PGRADE/gUSE-based gateways for running simulation applications in the cloud within the CloudSME project is also provided. <s> BIB012 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> This chapter introduces the data-driven workflow concept supported by the WS-PGRADE/gUSE system. Workflow management systems were investigated by Workflow Management Coalition, among others in aspects of implemented data flow structures, and several workflow patterns are identified as commonly used and meaningful workflow structures. The workflow concept of gUSE is shown by introducing the supported data patterns and illustrating their creation in the system. Moreover, the possibilities of utilizing parallelization techniques are described, and then the different views of a workflow design and management are described covering the whole lifecycle of a workflow development. Finally, more complex composition of patterns and their creation techniques are shown. <s> BIB013 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> WS-PGRADE/gUSE is a gateway framework that offers a set of high-level grid and cloud services by which interoperation between grids, clouds, and scientific user communities can be achieved. gUSE is also a workflow system that enables scientific communities to compose and execute a series of computational or data manipulation steps in a scientific application on Distributed Computing Infrastructures (DCIs). The current chapter summarizes the most important features of WS-PGRADE/gUSE. <s> BIB014 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> Solving distributed computing infrastructure (DCI) incompatibility issues in a generic way is a challenging and complex task. Gateways and workflow management systems are often tightly bound to some limited number of supported DCIs. To enable gateways access to many different DCIs and to solve DCI compatibility among the very different workflow management systems, we have developed a generic solution, the DCI Bridge. In this chapter we describe its internal architecture, provide usage scenario and show how the DCI Bridge resolves interoperability issues between various middleware-based DCIs. We also provide insight about the capabilities of the realized system. The generic DCI Bridge service seamlessly enables the execution of workflows (and jobs) on major DCI platforms such as ARC, Globus, gLite, UNICORE, SGE, PBS, as well as web services or clouds. <s> BIB015 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> Cloud computing is becoming more and more popular compared to grid computing due to its virtualization capabilities, flexibility and elasticity. In order to keep up with this trend, WS-PGRADE/gUSE has been extended with the possibility to use cloud infrastructures in workflows, either directly as described in Chap. 4 or via the CloudBroker Platform service. In this chapter we present the integration of WS-PGRADE/gUSE and the CloudBroker Platform, which is particularly useful in the case of accessing commercial clouds and/or using preregistered applications in commercial and academic clouds. <s> BIB016 </s> A Survey of Data-Intensive Scientific Workflow Management <s> WS-PGRADE/gUSE <s> E-scientists want to run their scientific experiments on Distributed Computing Infrastructures (DCI) to be able to access large pools of resources and services. To run experiments on these infrastructures requires specific expertise that e-scientists may not have. Workflows can hide resources and services as a virtualization layer providing a user interface that e-scientists can use. There are many workflow systems used by research communities but they are not interoperable. To learn a workflow system and create workflows in this workflow system may require significant efforts from e-scientists. Considering these efforts it is not reasonable to expect that research communities will learn new workflow systems if they want to run workflows developed in other workflow systems. The solution is to create workflow interoperability solutions to allow workflow sharing. The FP7 Sharing Interoperable Workflow for Large-Scale Scientific Simulation on Available DCIs (SHIWA) project developed two interoperability solutions to support workflow sharing: Coarse-Grained Interoperability (CGI) and Fine-Grained Interoperability (FGI). The project created the SHIWA Simulation Platform (SSP) to implement the Coarse-Grained Interoperability approach as a production-level service for research communities. The paper describes the CGI approach and how it enables sharing and combining existing workflows into complex applications and run them on Distributed Computing Infrastructures. The paper also outlines the architecture, components and usage scenarios of the simulation platform. <s> BIB017
WS-PGRADE/gUSE is a science gateway framework widely used in various disciplines such as biology BIB006 , seismology BIB007 , astronomy BIB008 , and neuroscience BIB009 . It is an open source software BIB003 used for teaching BIB010 , research BIB011 and commercial activities BIB012 . The architecture of WS-PGRADE/gUSE framework is shown in Fig. 13 . WS-PGRADE portal is a web-portal interface to help the users designing scientific workflows. The grid and cloud User Support Environment (gUSE) is a middle layer for different user services. The DCI bridge is a web service based application that provides access to divers infrastructures such as grid and cloud BIB003 . In the presentation layer, the WS-PGRADE portal BIB001 has a web browser based interface, which supports the definition of different kinds of scientific workflows, including meta-workflows and parameter sweep workflows. The meta-workflows can contain embedded workflows, which are supported by the SHIWA repository, as sub-workflows. In the user services layer, WS-PGRADE supports workflow information sharing between the users of WS-PGRADE/gUSE through a build-in workflow repository, which enables publishing, searching and downloading programs, workflow designs and templates BIB013 . A workflow template is an available workflow pattern that can be changed to other workflows by modifying corresponding parameters. In addition, it can exploit the SHIWA repository BIB005 to support workflow sharing between the users of different SWfMSs. The monitoring functionality is supported by the gUSE services. However, the framework lacks provenance support. In the WEP generation layer, a scientific workflow is represented in XML format. The framework may use data and independent parallelism according to the structure of the workflow BIB013 . The DCI bridge dynamically schedules the tasks through a submit queue. The task execution is handled by web services enabled by a web container, e.g. Tomcat or Glassfish. Fig. 13 Architecture of WS-PGRADE/gUSE BIB014 Provisioning and data storage are provided by the DCI Bridge BIB015 and CloudBroker framework. The DCI Bridge can dynamically create VMs through existing images and cloud configurations. The CloudBroker is able to exploit resources distributed in multiple clouds BIB016 , including major cloud types, e.g. Amazon, OpenStack BIB004 and OpenNebula BIB002 . Moreover, it can take advantage of GPUs to execute the workflows that invoke GPU programs BIB013 . The SHIWA simulation platform enables reusing of workflows in ten different SWfMS environments BIB017 . First, the users can search for workflows in the SHIWA repository, where workflow developers can upload available programs or workflows. Then, the workflows can be downloaded and adjusted to an Interoperable Workflow Intermediate Representation (IWIR) language to compose a meta-workflow BIB005 . Then, when the meta-workflow is submitted in the platform, each sub-workflow of the meta-workflow can be scheduled to the appropriate SWfMS execution environment.