Increasing adoption of scientific workflows in the community has urged for the development of multi-tenant platforms that provides these workflows execution as a service. Workflow as a Service (WaaS) concept has been brought up by researchers to address the future design of Workflow Management Systems (WMS) that can serve a large number of users from a single point of service. This platform differs from a traditional WMS in handling the workload of workflows at runtime. A traditional WMS is usually designed to execute a single workflow in a dedicated process while the WaaS platforms enhance the process by exploiting multiple workflows execution in a resource-sharing environment model. In this paper, we explore a novel resource-sharing policy to improve system utilization and to fulfill various Quality of Service (QoS) requirements from multiple users. We propose an Elastic Budget-constrained resource Provisioning and Scheduling algorithm for Multiple workflows designed for WaaS platforms that is able to reduce the computational overhead by encouraging resource-sharing policy to minimize workflows' makespan while meeting user-defined budget. Our experiments show that the EBPSM algorithm is able to utilize the resource-sharing policy to achieve higher performance in terms of minimizing the makespan compared to the state-of-the-art budget-constraint scheduling algorithm.
One of the challenges of executing scientific workflows in the clouds is related to the data-locality. Scientific workflows are known as data-intensive applications that involve a vast amount of data to be processed. Therefore, the communication overhead for transferring the data between tasks' execution may take a considerable amount of time that might impact the overall makespan. A work by Stavrinides and Karatza @cite_28 shows that the use of distributed in-memory storage to store the datasets locally for tasks' execution can reduce the communication overhead. Our work is similar regarding the data-locality policy to minimize the data transfer between tasks' execution. However, we use the cached datasets in VMs local storage from tasks' execution to endorse the locality of data. We enhance this policy so that the algorithm can intelligently decide which task to be scheduled in particular VMs that can provide the minimum execution time given the available cached datasets.
@cite_28
2011893902 2774502204 2161692763 1818445599
Cloud computing is gaining tremendous momentum in both academia and industry. In this context, we define the term "Cloud Workflow" as the specification, execution and provenance tracking of large-scale scientific workflows, as well as the management of data and computing resources to support the execution of large-scale scientific workflows in the Cloud. In this paper, we first analyze the gap between these two complementary technologies, and what it means to bring Clouds and workflows together. Then, we present the key challenges in supporting Cloud workflows, and present our reference framework for scientific workflow management in the Cloud. Last we present our experience in integrating a scientific workflow management system-Swift into the Cloud. We discuss the performance of cluster provisioning within the OpenNebula Cloud platform, the Eucalyptus Cloud platform and Amazon EC2, and we demonstrate the capability and efficiency of the integration using a NASA MODIS image processing workflow and the Montage image mosaic workflow.Note to practitioners. Scientific workflow management plays a very important role for scientific computing and application coordination, while Cloud computing offers scalability and resource on-demand. We devise autonomous methods to integrate scientific workflow management systems with Cloud platforms and also provision resources for large scale workflows, which can facilitate scientists to easily manage their workflows in the Cloud, and take advantage of large scale Cloud resources. There are a few integration options and many challenges in the process, and the experience we gain will help researchers in migrating their workflow management systems and workflow applications into the Cloud. We analyze the major challenges of running scientific workflows on the Cloud.We propose a reference framework to standardize the integration.The implementation experience proves that the framework is feasible and extendible.Cluster-recycling mechanism can improve the resource provisioning efficiency. Scientific workflows are increasingly containerised, which requires rethinking central processing unit (CPU) sharing policies to accommodate different workload types. However, container engines running scientific workflows struggle to share the CPU fairly, as workload characteristics are not taken into account. This paper proposes a sharing policy called the Adaptive Completely Fair Scheduling policy (adCFS), which considers the future state of CPU usage and proactively shares CPU cycles between various containers based on their corresponding workload metrics (e.g., CPU usage, task runtime, #tasks). adCFS estimates the weight of workload characteristics and redistributes the CPU based on the corresponding weights. The Markov chain model is used to predict CPU state use, and the adCFS policy is triggered to dynamically allocate containers to the proper CPU portions. Experimental results show enhanced container CPU response time for those containers that run heavy and large jobs: these display 12 faster response time compared with the default CFS (Completely Fair Scheduler). adCFS therefore enhances CFS by considering workload metrics, which leads to the CPU being shared fairly when it is fully used. Scientific experiments and large-scale simulations produce massive amounts of data. Many of these scientific datasets are arrays, and are stored in file formats such as HDF5 and NetCDF. Although scientific data management systems, such as SciDB, are designed to manipulate arrays, there are challenges in integrating these systems into existing analysis workflows. Major barriers include the expensive task of preparing and loading data before querying, and converting the final results to a format that is understood by the existing post-processing and visualization tools. As a consequence, integrating a data management system into an existing scientific data analysis workflow is time-consuming and requires extensive user involvement. In this paper, we present the design of a new scientific data analysis system that efficiently processes queries directly over data stored in the HDF5 file format. This design choice eliminates the tedious and error-prone data loading process, and makes the query results readily available to the next processing steps of the analysis workflow. Our design leverages the increasing main memory capacities found in supercomputers through bitmap indexing and in-memory query execution. In addition, query processing over the HDF5 data format can be effortlessly parallelized to utilize the ample concurrency available in large-scale supercomputers and modern parallel file systems. We evaluate the performance of our system on a large supercomputing system and experiment with both a synthetic dataset and a real cosmology observation dataset. Our system frequently outperforms the relational database system that the cosmology team currently uses, and is more than 10X faster than Hive when processing data in parallel. Overall, by eliminating the data loading step, our query processing system is more effective in supporting in situ scientific analysis workflows. High intensive computation applications can usually take days to months to nish an execution. During this time, it is common to have variations of the available resources when considering that such hardware is usually shared among a plurality of researchers departments within an organization. On the other hand, High Performance Clusters can take advantage of Cloud Computing bursting techniques for the execution of applications together with on-premise resources. In order to meet deadlines, high intensive computational applications can use the Cloud to boost their performance when they are data and task parallel. This article presents an ongoing work towards the use of extended resources of an HPC execution platform together with Cloud. We propose an unied view of such heterogeneous environments and a method that monitors, predicts the application execution time, and dynamically shifts part of the domain previously running in local HPC hardware to be computed in the Cloud, meeting then a specic deadline. The method is exemplied along with a seismic application that, at runtime, adapts itself to move part of the processing to the Cloud (in a movement called bursting) and also auto-scales (the moved part) over cloud nodes. Our preliminary results show that there is an expected overhead for performing this movement and for synchronizing results, but the outcomes demonstrate it is an important feature for meeting deadlines in the case an on-premise cluster is overloaded or cannot provide the capacity needed for a particular project.