A Survey of Data-Intensive Scientific Workflow Management <s> Dynamic Scheduling <s> Dynamic mapping (matching and scheduling) heuristics for a class of independent tasks using heterogeneous distributed computing systems are studied. Two types of mapping heuristics are considered, immediate mode and batch mode heuristics. Three new heuristics, one for batch mode and two for immediate mode, are introduced as part of this research. Simulation studies are performed to compare these heuristics with some existing ones. In total five immediate mode heuristics and three batch mode heuristics are examined. The immediate mode dynamic heuristics consider, to varying degrees and in different ways, task affinity for different machines and machine ready times. The batch mode dynamic heuristics consider these factors, as well as aging of tasks waiting to execute. The simulation results reveal that the choice of which dynamic mapping heuristic to use in a given heterogeneous environment depends on parameters such as (a) the structure of the heterogeneity among tasks and machines and (b) the arrival rate of the tasks. <s> BIB001 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Dynamic Scheduling <s> Scheduling is the key to the performance of grid workflow applications. Various strategies are proposed, including static scheduling strategies which map jobs to resources before execution time, or dynamic alternatives which schedule individual job only when it is ready to execute. While sizable work supports the claim that the static scheduling performs better for workflow applications than the dynamic one, it is questioned how a static schedule works effectively in a grid environment which changes constantly. This paper proposes a novel adaptive rescheduling concept, which allows the workflow planner works collaboratively with the run time executor and reschedule in a proactive way had the grid environment changes significantly. An HEFT-based adaptive rescheduling algorithm is presented, evaluated and compared with traditional static and dynamic strategies respectively. The experiment results show that the proposed strategy not only outperforms the dynamic one but also improves over the traditional static one. Furthermore we observed that it performs more efficiently with data intensive application of higher degree of parallelism. <s> BIB002 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Dynamic Scheduling <s> Desktop grids are being increasingly used as the execution platform for a variety of applications that can be structured as bag-of-tasks (BoT). Scheduling BoT applications on desktop grids has thus attracted the attention of the scientific community, and various schedulers tailored towards them have been proposed in the literature. However, previous work has focused on scheduling a single BoT application at a time, thus neglecting other scenarios in which several users submit multiple BoT applications at the same time. This paper aims at filling this gap by proposing a set of scheduling algorithms able to deal with multiple BoT applications. The performance of these algorithm has been evaluated, by means of simulation, for a large set of operational scenarios obtained by varying both the workload submitted to the desktop grid and the characteristics of the involved resources. Our results show that, although there is no a clear winner among the proposed solutions, knowledge-free strategies (that is, strategies that do not require any information concerning the applications or the resources) can provide good performance. <s> BIB003 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Dynamic Scheduling <s> Scientists, engineers, and statisticians must execute domain-specific application programs many times on large collections of file-based data. This activity requires complex orchestration and data management as data is passed to, from, and among application invocations. Distributed and parallel computing resources can accelerate such processing, but their use further increases programming complexity. The Swift parallel scripting language reduces these complexities by making file system structures accessible via language constructs and by allowing ordinary application programs to be composed into powerful parallel scripts that can efficiently utilize parallel and distributed resources. We present Swift's implicitly parallel and deterministic programming model, which applies external applications to file collections using a functional style that abstracts and simplifies distributed parallel execution. <s> BIB004 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Dynamic Scheduling <s> Galaxy is an open source web based framework for data intensive biomedical research. It is an environment for interactive analysis that transparently tracks the details of analyses, a workflow system for convenient reuse, data management, sharing and publishing of the results. In this paper we describe extending the Galaxy framework to the GARUDA Grid computing infrastructure for running bioinformatics applications on various grid. This involves developing a grid job adaptor for the Galaxy workflow. GARUDA grid is an aggregation of heterogeneous resources and advanced capabilities for scientific applications. Grid way is the met scheduler for running the jobs on the Garuda grid. Here we present the integration of Galaxy workflow with the Grid way middleware by developing a grid job runner to interface. This grid enabled galaxy facilitates computational biologists to perform complex problems on the grid environment through a web browser. <s> BIB005 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Dynamic Scheduling <s> Over the last two decades, scientific workflow management systems (SWfMS) have emerged as a means to facilitate the design, execution, and monitoring of reusable scientific data processing pipelines. At the same time, the amounts of data generated in various areas of science outpaced enhancements in computational power and storage capabilities. This is especially true for the life sciences, where new technologies increased the sequencing throughput from kilobytes to terabytes per day. This trend requires current SWfMS to adapt: Native support for parallel workflow execution must be provided to increase performance; dynamically scalable "pay-per-use" compute infrastructures have to be integrated to diminish hardware costs; adaptive scheduling of workflows in distributed compute environments is required to optimize resource utilization. In this survey we give an overview of parallelization techniques for SWfMS, both in theory and in their realization in concrete systems. We find that current systems leave considerable room for improvement and we propose key advancements to the landscape of SWfMS. <s> BIB006 </s> A Survey of Data-Intensive Scientific Workflow Management <s> Dynamic Scheduling <s> Large-scale scientific experiments based on computer simulations are typically modeled as scientific workflows, which eases the chaining of different programs. These scientific workflows are defined, executed and monitored by Scientific Workflow Management Systems (SWfMS). As these experiments manage large amounts of data, it becomes critical to execute them in High Performance Computing (HPC) environments, such as clusters, grids and clouds. However, few SWfMS provide parallel support. The ones that do so are usually labor-intensive for workflow developers and have limited primitives to optimize workflow execution. To address these issues, we developed a workflow algebra to specify and enable the optimization of parallel execution of scientific workflows. In this paper, we show how the workflow algebra is efficiently implemented in Chiron, an algebraic based parallel scientific workflow engine. Chiron has a unique native distributed provenance mechanism that enables run-time queries in a relational database. We developed two studies to evaluate the performance of our algebraic approach implemented in Chiron; the first study compares Chiron with different approaches while the second one evaluates the scalability of Chiron. By analyzing the results, we conclude that Chiron is efficient in executing scientific workflows, with the benefits of declarative specification and runtime provenance support. <s> BIB007
Dynamic scheduling produces SPs that distribute and allocate executable tasks to computing nodes during workflow execution BIB006 . This kind of scheduling is appropriate for scientific workflows, in which the workload of tasks is difficult to estimate, or for environments where the capabilities of the computers varies a lot during execution. Dynamic scheduling can achieve load balancing while it takes time to dynamically generate SPs during execution. The scheduling algorithms can be based on the queue techniques in a publish/subscribe model with different strategies such as First In First Out (FIFO), adaptive and so on. SWfMSs such as Swift BIB004 , Chiron BIB007 , and Galaxy BIB005 exploit dynamic scheduling algorithms. Dynamic SPs may be generated by adapting a static scheduling method to dynamic environment. Yu and Shi BIB002 introduce an HEFT-based dynamic scheduling algorithm. This algorithm is suited to the situation where a scientific workflow has been executed partially before scheduling. It schedules the tasks by applying an HEFT-based algorithm according to a dynamically generated rank of tasks. There are some original approaches to generate dynamic SPs. Maheswaran et al. BIB001 present a minmin algorithm that is designed as a batch mode scheduling of two steps. First, a list of tasks ready to be executed is created. This phase is called "task prioritizing" phase. Then, the tasks in the list are scheduled to computing nodes based on a heuristic. The heuristic maps the task T to the computing node M such that T is the task that has minimum expected execution time in the non-mapped tasks and that M is the computing node that is executing a task having minimum expected execution time in the mapped tasks. This phase is called the "resource selection" phase. Anglano and Canonico BIB003 present several knowledge-free scheduling algorithms that are able to schedule multiple bags of tasks. A knowledge-free algorithm does not require any information on the resources for scheduling. These algorithms implement different policies: First Come First Served -Exclusive (FCFS-Excl), First Come First Served -Shared (FCFS-Share), Round Robin (RR), Round Robin -No Replica First (RR-NRF) and Longest Idle. With the FCFS-Excl policy, bags of tasks are scheduled in order of arrival. Different from FCFS-Excl, FCFS-Share can allocate more than one bag of tasks to a computing node. The RR policy schedules bags of tasks in a fixed circular order while all the bags have the same probability to be scheduled. With the RR-NRF policy, a bag of tasks that does not have any task executed will be given priority. In this policy, all the bags of tasks have at least a task running. The longest idle policy tries to reduce waiting time by giving preference to the bag of tasks hosting the task that exhibits the largest waiting time. The paper shows that the FCFS-based policy performs better for small task granularity scheduling.