A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Utility grids such as the Amazon EC2 cloud and Amazon S3 offer computational and storage resources that can be used on-demand for a fee by compute and data-intensive applications. The cost of running an application on such a cloud depends on the compute, storage and communication resources it will provision and consume. Different execution plans of the same application may result in significantly different costs. Using the Amazon cloud fee structure and a real-life astronomy application, we study via simulation the cost performance tradeoffs of different execution and resource provisioning plans. We also study these trade-offs in the context of the storage and communication fees of Amazon S3 when used for long-term application data archival. Our results show that by provisioning the right amount of storage and compute resources, cost can be significantly reduced with no significant impact on application performance. <s> BIB001 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Many scientific workflows are composed of fine computational granularity tasks, yet they are composed of thousands of them and are data intensive in nature, thus requiring resources such as the TeraGrid to execute efficiently. In order to improve the performance of such applications, we often employ task clustering techniques to increase the computational granularity of workflow tasks. The goal is to minimize the completion time of the workflow by reducing the impact of queue wait times. In this paper, we examine the performance impact of the clustering techniques using the Pegasus workflow management system. Experiments performed using an astronomy workflow on the NCSA TeraGrid cluster show that clustering can achieve a significant reduction in the workflow completion time (up to 97%). <s> BIB002 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> The need for design primitives for scientific workflows has steadily increased over the years and, actually, has become more pronounced in recent years, with the employment of user-friendly scientific workflow management systems. In this paper, we conduct a comparative study between business and scientific workflows initiatives based on common workflow patterns found in business workflow. This study demonstrates some precise differences and identifies some key scientific workflow patterns that can be used in dataflow oriented scientific workflow systems without compromising the data-oriented modeling in scientific workflow. <s> BIB003 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Montage is a portable software toolkit to construct custom, science-grade mosaics that preserve the astrometry and photometry of astronomical sources. The user specifies the dataset, wavelength, sky location, mosaic size, coordinate system, projection, and spatial sampling. Montage supports massive astronomical datasets that may be stored in distributed archives. Montage can be run on both single- and multi-processor computers, including clusters and grids. Standard grid tools are used to access remote data or run Montage on remote computers. This paper describes the architecture, algorithms, performance, and usage of Montage as both a software toolkit and a grid portal. <s> BIB004 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Most of the large-scale scientific experiments modeled as scientific workflows produce a large amount of data and require workflow parallelism to reduce workflow execution time. Some of the existing Scientific Workflow Management Systems (SWfMS) explore parallelism techniques - such as parameter sweep and data fragmentation. In those systems, several computing resources are used to accomplish many computational tasks in homogeneous environments, such as multiprocessor machines or cluster systems. Cloud computing has become a popular high performance computing model in which (virtualized) resources are provided as services over the Web. Some scientists are starting to adopt the cloud model in scientific domains and are moving their scientific workflows (programs and data) from local environments to the cloud. Nevertheless, it is still difficult for the scientist to express a parallel computing paradigm for the workflow on the cloud. Capturing distributed provenance data at the cloud is also an issue. Existing approaches for executing scientific workflows using parallel processing are mainly focused on homogeneous environments whereas, in the cloud, the scientist has to manage new aspects such as initialization of virtualized instances, scheduling over different cloud environments, impact of data transferring and management of instance images. In this paper we propose SciCumulus, a cloud middleware that explores parameter sweep and data fragmentation parallelism in scientific workflow activities (with provenance support). It works between the SWfMS and the cloud. SciCumulus is designed considering cloud specificities. We have evaluated our approach by executing simulated experiments to analyze the overhead imposed by clouds on the workflow execution time. <s> BIB005 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Bioinformatics experiments are rapidly evolving with genomic projects that analyze large amounts of data. This fact demands high performance computation and opens up for exploring new approaches to provide better control and performance when running experiments, including Phylogeny/ Phylogenomics. We designed a phylogenetic scientific workflow, named SciPhy, to construct phylogenetic trees from a set of drug target enzymes found in protozoan genomes. Our contribution is the development, implementation and test of SciPhy in public cloud computing environments. SciPhy can be used in other Bioinformatics experiments to control a systematic execution with high performance while producing provenance data. <s> BIB006 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> We describe a new model for representing semantic workflows as semantically labeled graphs, together with a related model for knowledge intensive similarity measures. The application of this model to scientific and business workflows is discussed. Experimental evaluations show that similarity measures can be modeled that are well aligned with manual similarity assessments. Further, new algorithms for workflow similarity computation based on A* search are described. A new retrieval algorithm is introduced that goes beyond traditional sequential retrieval for graphs, interweaving similarity computation with case selection. Significant reductions on the overall retrieval time are demonstrated without sacrificing the correctness of the computed similarity. <s> BIB007 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Recent studies of evolution at molecular level address two important issues: reconstruction of the evolutionary relationships between species and investigation of the forces of the evolutionary process. Both issues experienced an explosive growth in the last two decades due to massive generation of genomic data, novel statistical methods and computational approaches to process and analyze this large volume of data. Most experiments in molecular evolution are based on computing intensive simulations preceded by other computation tools and post-processed by computing validators. All these tools can be modeled as scientific workflows to improve the experiment management while capturing provenance data. However, these evolutionary analyses experiments are very complex and may execute for weeks. These workflows need to be executed in parallel in High Performance Computing (HPC) environments such as clouds. Clouds are becoming adopted for bioinformatics experiments due to its characteristics, such as, elasticity and availability. Clouds are evolving into HPC environments. In this paper, we introduce SciEvol, a bioinformatics scientific workflow for molecular evolution reconstruction that aims at inferring evolutionary relationships (i.e. to detect positive Darwinian selection) on genomic data. SciEvol is designed and implemented to execute in parallel over the clouds using SciCumulus workflow engine. Our experiments show that SciEvol can help scientists by enabling the reconstruction of evolutionary relationships using the cloud environment. Results present performance improvements of up to 94.64% in the execution time when compared to the sequential execution, which drops from around 10 days to 12 hours. <s> BIB008 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Illnesses caused by parasitic protozoan are a research priority. A representative group of these illnesses is the commonly known as Neglected Tropical Diseases (NTD). NTD specially attack low socioeconomic population around the world and new anti-protozoan inhibitors are needed and several drug discovery projects focus on researching new drug targets. Pharmacophylogenomics is a novel bioinformatics field that aims at reducing the time and the financial cost of the drug discovery process. Pharmacophylogenomic analyses are applied mainly in the early stages of the research phase in drug discovery. Pharmacophylogenomic analysis executes several bioinformatics programs in a coherent flow to identify homologues sequences, construct phylogenetic trees and execute evolutionary and structural experiments. This way, it can be modeled as scientific workflows. Pharmacophylogenomic analysis workflows are complex, computing and data intensive and may execute during weeks. This way, it benefits from parallel execution. We propose SciPPGx, a scientific workflow that aims at providing thorough inferring support for pharmacophylogenomic hypotheses. SciPPGx is executed in parallel in a cloud using SciCumulus workflow engine. Experiments show that SciPPGx considerably reduces the total execution time up to 97.1% when compared to a sequential execution. We also present representative biological results taking advantage of the inference covering several related bioinformatics overviews. <s> BIB009 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Scientific experiments based on computer simulations can be defined, executed and monitored using Scientific Workflow Management Systems (SWfMS). Several SWfMS are available, each with a different goal and a different engine. Due to the exploratory analysis, scientists need to run parameter sweep (PS) workflows, which are workflows that are invoked repeatedly using different input data. These workflows generate a large amount of tasks that are submitted to High Performance Computing (HPC) environments. Different execution models for a workflow may have significant differences in performance in HPC. However, selecting the best execution model for a given workflow is difficult due to the existence of many characteristics of the workflow that may affect the parallel execution. We developed a study to show performance impacts of using different execution models in running PS workflows in HPC. Our study contributes by presenting a characterization of PS workflow patterns (the basis for many existing scientific workflows) and its behavior under different execution models in HPC. We evaluated four execution models to run workflows in parallel. Our study measures the performance behavior of small, large and complex workflows among the evaluated execution models. The results can be used as a guideline to select the best model for a given scientific workflow execution in HPC. Our evaluation may also serve as a basis for workflow designers to analyze the expected behavior of an HPC workflow engine based on the characteristics of PS workflows. <s> BIB010 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Data analysis is an exploratory process that demands high performance computing (HPC). SciPhylomics, for example, is a data-intensive workflow that aims at producing phylogenomic trees based on an input set of protein sequences of genomes to infer evolutionary relationships among living organisms. SciPhylomics can benefit from parallel processing techniques provided by existing approaches such as SciCumulus cloud workflow engine and MapReduce implementations such as Hadoop. Despite some performance fluctuations, computing clouds provide a new dimension for HPC due to its elasticity and availability features. In this paper, we present a performance evaluation for SciPhylomics executions in a real cloud environment. The workflow was executed using two parallel execution approaches (SciCumulus and Hadoop) at the Amazon EC2 cloud. Our results reinforce the benefits of parallelizing data for the phylogenomic inference workflow using MapReduce-like parallel approaches in the cloud. The performance results demonstrate that this class of bioinformatics experiment is suitable to be executed in the cloud despite its need for high performance capabilities. The evaluated workflow shares many features of several data intensive workflows, which present first insights that these cloud execution results can be extrapolated to other classes of experiments. <s> BIB011 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Large-scale scientific experiments based on computer simulations are typically modeled as scientific workflows, which eases the chaining of different programs. These scientific workflows are defined, executed and monitored by Scientific Workflow Management Systems (SWfMS). As these experiments manage large amounts of data, it becomes critical to execute them in High Performance Computing (HPC) environments, such as clusters, grids and clouds. However, few SWfMS provide parallel support. The ones that do so are usually labor-intensive for workflow developers and have limited primitives to optimize workflow execution. To address these issues, we developed a workflow algebra to specify and enable the optimization of parallel execution of scientific workflows. In this paper, we show how the workflow algebra is efficiently implemented in Chiron, an algebraic based parallel scientific workflow engine. Chiron has a unique native distributed provenance mechanism that enables run-time queries in a relational database. We developed two studies to evaluate the performance of our algebraic approach implemented in Chiron; the first study compares Chiron with different approaches while the second one evaluates the scalability of Chiron. By analyzing the results, we conclude that Chiron is efficient in executing scientific workflows, with the benefits of declarative specification and runtime provenance support. <s> BIB012 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> This chapter introduces the data-driven workflow concept supported by the WS-PGRADE/gUSE system. Workflow management systems were investigated by Workflow Management Coalition, among others in aspects of implemented data flow structures, and several workflow patterns are identified as commonly used and meaningful workflow structures. The workflow concept of gUSE is shown by introducing the supported data patterns and illustrating their creation in the system. Moreover, the possibilities of utilizing parallelization techniques are described, and then the different views of a workflow design and management are described covering the whole lifecycle of a workflow development. Finally, more complex composition of patterns and their creation techniques are shown. <s> BIB013 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> Abstract Workflow technology continues to play an important role as a means for specifying and enacting computational experiments in modern science. Reusing and re-purposing workflows allow scientists to do new experiments faster, since the workflows capture useful expertise from others. As workflow libraries grow, scientists face the challenge of finding workflows appropriate for their task, understanding what each workflow does, and reusing relevant portions of a given workflow. We believe that workflows would be easier to understand and reuse if high-level views (abstractions) of their activities were available in workflow libraries. As a first step towards obtaining these abstractions, we report in this paper on the results of a manual analysis performed over a set of real-world scientific workflows from Taverna, Wings, Galaxy and Vistrails. Our analysis has resulted in a set of scientific workflow motifs that outline (i) the kinds of data-intensive activities that are observed in workflows ( Data-Operation motifs ), and (ii) the different manners in which activities are implemented within workflows ( Workflow-Oriented motifs ). These motifs are helpful to identify the functionality of the steps in a given workflow, to develop best practices for workflow design, and to develop approaches for automated generation of workflow abstractions. <s> BIB014 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Scientific Workflow Examples <s> WS-PGRADE/gUSE is a gateway framework that offers a set of high-level grid and cloud services by which interoperation between grids, clouds, and scientific user communities can be achieved. gUSE is also a workflow system that enables scientific communities to compose and execute a series of computational or data manipulation steps in a scientific application on Distributed Computing Infrastructures (DCIs). The current chapter summarizes the most important features of WS-PGRADE/gUSE. <s> BIB015
Scientific workflows have been used in various scientific domains. In the astronomy domain, Montage 1 is a computing and data-intensive application that can be modeled as a scientific workflow initially defined for the Pegasus SWfMS. This application is the result of a national virtual observatory project that stitches tiles of images of the sky from diverse sky surveys into a photorealistic single image BIB001 . Montage is able to handle a wide range of astronomical image data including the Two Micron All Sky Survey, (2MASS), 2 the Digitized Palomar Observatory Sky Survey, (DPOSS), 3 and the Sloan Digital Sky Survey (SDSS) 4 BIB004 . Each survey possesses huge amounts of data and covers a corresponding part of sky in visible wavelengths or near-infrared wavelengths. 2MASS has roughly 10 terabytes, DPOSS has roughly 3 terabytes and SDSS contains roughly 7.4 terabytes. All the data can be downloaded from a corresponding 1 Montage project: http://montage.ipac.caltech.edu/ 2 2MASS: http://www.ipac.caltech.edu/2mass/ 3 DPOSS: http://www.astro.caltech.edu/ ∼ george/dposs/ 4 SDSS: http://www.sdss.org/ Fig. 1 The structure of a small Montage workflow BIB002 server at the aforementioned links and then staged into the execution environment, such as a shared-disk file system or a database, in order to be processed. The structure of a small montage workflow is shown in Fig. 1 . The number within a node represents the name of an activity in the workflow. The activities (1−6) have no parent activities. Each of them exploits an mProject program to project a single image to the scale defined in a pseudo-FITS header template file. Though they invoke the same program, they have different output data dependencies and thus, they cannot be viewed as one activity. Activities (7 − 14) utilize an mDiffFit program to create a table of image-to-image difference parameters. Activity 15 takes advantage of an mFitplane program to fit the images generated by former activities (7−14) to an image. Activity 16 uses an mBgModel program to interactively determine a set of corrections to apply to each image to achieve a "best" global fit according to the image-to-image difference parameter table. The activities (17 − 22) remove a background from a single image through an mBackground program. Activity 23 employs an mImgtbl program to extract the FITS header information (information about one or more scientific coordinate systems that are overlaid on the image itself) from a set of files and to create an ASCII image metadata table. Finally, Activity 24 pieces together the projected images using the uniform FITS header template and the information from the same metadata table generated by Activity 23. This activity applies an mAdd program. In the bioinformatics domain, SciEvol [100] is a workflow for molecular evolution reconstruction that aims at inferring evolutionary relationships on genomic data. To be executed in the Chiron SWfMS, SciEvol consists of 12 activities as shown in Fig. 2 . The first activity (pre-processing FASTA file) is a Python script to format the multi-fasta input file. FASTA file is a textual presenting format for nucleotide or peptide sequences. The second activity (MSA construction) constructs a Multiple Sequence Alignment (MSA) using a MAFFT program (or other MSA programs). A MAFFT program is generally for generating the alignment of three or more biological sequences (protein or nucleic acids) of similar length. The third activity (MSA conversion) executes ReadSeq to convert the MSA in FASTA format to that in PHYLIP format, which is used in the phylogenetic tree construction activity. The fourth activity (pre-processing PHYLIP file) formats the input BIB008 file (referenced as "phylip-file-one") according to the format definition and generates a second file (referenced as "phylip-file-two"). The fifth activity (tree construction) receives the "phylip-file-one" as input and produces a phylogenetic tree  as output. The sixth activities (evolutionary analysis from 6.1 to 6.6) analyze the phylogenetic tree with corresponding parameters and generate a set of files containing evolutionary information as output. Each of the activities (evolutionary phases) is related to one of six codon substitution models, which are used to verify if the groups of genes are under positive Darwinian selection. These activities exploit the same program using different parameters. The last activity (data analysis) automatically processes the output files obtained from the previous activities. There are many other data-intensive workflows in bioinformatics. For instance, SciPhylomics BIB011 is designed for producing phylogenomic trees based on an input set of protein sequences of genomes to infer evolutionary relationships among living organisms. SciPPGx BIB009 is a computing and data-intensive pharmacophylogenomic analysis workflow for providing thorough inferring support for pharmacophylogenomic hypotheses. SciPhy BIB006 is used to construct phylogenetic trees from a set of drug target enzymes found in protozoan genomes. All these bioinformatics workflows have been executed using SciCumulus SWfMS BIB005 . The components of scientific workflows can be classified by their functionality, motifs, or structure patterns. The functionality can be data processing, activity scheduling, activity execution, and resource management  . The motifs may be data-oriented and workflow-oriented. Data-oriented motifs consist of recurring activities such as data storage BIB013 , data analysis, data cleaning, data moving BIB013 and data visualization. Workflow-oriented motifs may correspond to remote invocations, repetitive activities, parameter sweep workflows and meta-workflows BIB013 BIB014 . A parameter sweep workflow is a workflow with multiple input parameter sets, which needs to be executed for each input parameter set BIB010 BIB015 . A meta-workflow is a workflow composed of subworkflows. Workflow structure patterns can be patterns for parallelization, e.g. representing scientific workflows as algebraic expressions BIB012 , or component structure patterns, e.g. single activity with one or more input/output dependencies, sequential control and sequential/concurrent data, synchronization of sequential data, data duplication BIB003 . Moreover, similar structure patterns of scientific workflows can be found based on a similarity model of nodes and edges in the workflow DAG BIB007 . Identifying scientific workflows or workflow components of the same type enables workflow information sharing and reuse (see Section 2.2.2) among workflow designers BIB003 .