Private cloud infrastructures are now widely deployed and adopted across technology industries and research institutions. Although cloud computing has emerged as a reality, it is now known that a single cloud provider cannot fully satisfy complex user requirements. This has resulted in a growing interest in developing hybrid cloud solutions that bind together distinct and heterogeneous cloud infrastructures. In this paper we describe the orchestration approach for heterogeneous clouds that has been implemented and used within the INDIGO-DataCloud project. This orchestration model uses existing open-source software like OpenStack and leverages the OASIS Topology and Specification for Cloud Applications (TOSCA) open standard as the modeling language. Our approach uses virtual machines and Docker containers in an homogeneous and transparent way providing consistent application deployment for the users. This approach is illustrated by means of two different use cases in different scientific communities, implemented using the INDIGO-DataCloud solutions.
Cloud orchestration within science applications has been tackled before by several authors, in works related to specific scientific areas such as bioinformatics and biomedical applications @cite_9 , neuroscience @cite_76 , phenomenology physics @cite_41 , astrophysics @cite_83 , environmental sciences @cite_77 , engineering @cite_66 , high energy physics @cite_28 , etc. This has been also addressed in more generic approaches, not bounded to a scientific discipline @cite_75 @cite_74 @cite_42 . However, these works tend to be, in general, too tied to a given type of application and workload, and they need to be generalized in order to be reused outside their original communities.
@cite_41 @cite_28 @cite_9 @cite_42 @cite_77 @cite_83 @cite_74 @cite_76 @cite_75 @cite_66
2071345192 2114161868 2281439845 2006772912 2039623259 2556138242 2520792696 2466302097 2011893902 2040858596
We present a new environment for computations in particle physics phenomenology employing recent developments in cloud computing. On this environment users can create and manage “virtual” machines on which the phenomenology codes tools can be deployed easily in an automated way. We analyze the performance of this environment based on “virtual” machines versus the utilization of physical hardware. In this way we provide a qualitative result for the influence of the host operating system on the performance of a representative set of applications for phenomenology calculations. The challenge of providing a resilient and scalable computational and data management solution for massive scale research environments requires continuous exploration of new technologies and techniques. In this project the aim has been to design a scalable and resilient infrastructure for CERN HEP data analysis. The infrastructure is based on OpenStack components for structuring a private Cloud with the Gluster File System. We integrate the state-of-the-art Cloud technologies with the traditional Grid middleware infrastructure. Our test results show that the adopted approach provides a scalable and resilient solution for managing resources without compromising on performance and high availability. Abstract Fast and simple access to computing resources for in-silico experiments and processing of large amounts of data have become crucial factors for researchers in the bioinformatics and biomedical domains. Cloud computing offers the possibility of accessing computing resources for a flexible amount of time and with varying requirements. We discuss how a full cloud stack ranging from Infrastructure-as-a-Service (IaaS) via Platform-as-a-Service (PaaS) to Software-as-a-Service (SaaS) can be built on open source technologies including the integration of high-throughput data transfers in all service types. Based on the scaling capabilities provided by the tools used on the IaaS level we devise a strategy to build a dynamically scaling PaaS offering for building and running workflows using Galaxy. Finally, applications — available as SaaS offerings — for two distinct use cases from the bioinformatics and biomedical domains are presented to demonstrate the feasibility and performance of our solution. Composite Web services (WS) can be seen as software systems designed according to workflow-based orchestration of building blocks or simpler WS. Each block has its own specifications concerning both functional and non-functional properties. While the characteristics of each block have a scope limited to its domain, the WS must guarantee service levels that are usually described by global end-to-end metrics. The problem of relating local to global objectives in WS orchestration is hard to approach. In this context, some WS components have to be deployed in distributed service oriented infrastructure mixing heterogeneous systems belonging to private and or public providers. In this paper we propose a performance-driven technique for designing and deploying composite WS on heterogeneous service oriented infrastructure. Users having different requirements in terms of resource demands and performance objectives are considered. Several WS deployment alternatives, involving both physical and virtual resources provided by the infrastructure, are evaluated to identify the logical (workflow) and physical (deployment) configuration allowing to meet the requirements. In order to demonstrate the suitability of the proposed approach to the service oriented context, an example of a travel management WS is described and the optimal deployment of the components in a hybrid infrastructure is investigated. This work introduces Ophidia, a big data analytics research effort aiming at supporting the access, analysis and mining of scientific (n-dimensional array based) data. The Ophidia platform extends, in terms of both primitives and data types, current relational database system implementations (in particular MySQL) to enable efficient data analysis tasks on scientific array-based data. To enable big data analytics it exploits well-known scientific numerical libraries, a distributed and hierarchical storage model and a parallel software framework based on the Message Passing Interface to run from single tasks to more complex dataflows. The current version of the Ophidia platform is being tested on NetCDF data produced by CMCC climate scientists in the context of the international Coupled Model Intercomparison Project Phase 5 (CMIP5). An efficient exploitation of Distributed Computing Infrastructures (DCIs) is needed to deal with the data deluge that the scientific community is facing, in particular the Astrophysics one due to the emerging Square Kilometre Array (SKA) telescope that will reach data rates in the exascale domain. Hence, Science Gateways are being enriched with advanced tools that not only enable the scientists to build their experiments but also to optimize their adaptation to different infrastructures. In this work we present a method, called "two-level workflow system", to build this kind of tools and we apply it to a set of analysis tasks of interest for some use applications to the SKA. This method uses the Software-as-a-Service model to keep the scientists insulated from technical complexity of DCIs, and the COMPSs programming model to achieve an efficient use of the computing resources. The workflow interoperability problem was successfully solved by the SHIWA project if the workflows to be integrated were running in the same grid infrastructure. However, in the more generic case when the workflows were running in different infrastructures the problem has not been solved yet. In the current paper we show a solution for this problem by introducing a new type of workflow called infrastructure-aware workflow. These are scientific workflows extended with new node types that enable the on-the-fly creation and destruction of the required infrastructures in the clouds. The paper shows the semantics of these new types of nodes and workflows and also how they can solve the workflow interoperability problem. The paper also describes how these new type of workflows can be implemented by a new service called Occopus, and how this service can be integrated with the existing SHIWA Simulation Platform services like the WS-PGRADE gUSE portal to provide the required functionalities of solving the workflow interoperability problem. Simulations in neuroscience are performed on local servers or High Performance Computing (HPC) facilities. Recently, cloud computing has emerged as a potential computational platform for neuroscience simulation. In this paper we compare and contrast HPC and cloud resources for scientific computation, then report how we deployed NEURON, a widely used simulator of neuronal activity, in three clouds: Chameleon Cloud, a hybrid private academic cloud for cloud technology research based on the OpenStack software; Rackspace, a public commercial cloud, also based on OpenStack; and Amazon Elastic Cloud Computing, based on Amazon’s proprietary software. We describe the manual procedures and how to automate cloud operations. We describe extending our simulation automation software called NeuroManager (Stockton and Santamaria, Frontiers in Neuroinformatics, 2015), so that the user is capable of recruiting private cloud, public cloud, HPC, and local servers simultaneously with a simple common interface. We conclude by performing several studies in which we examine speedup, efficiency, total session time, and cost for sets of simulations of a published NEURON model. Cloud computing is gaining tremendous momentum in both academia and industry. In this context, we define the term "Cloud Workflow" as the specification, execution and provenance tracking of large-scale scientific workflows, as well as the management of data and computing resources to support the execution of large-scale scientific workflows in the Cloud. In this paper, we first analyze the gap between these two complementary technologies, and what it means to bring Clouds and workflows together. Then, we present the key challenges in supporting Cloud workflows, and present our reference framework for scientific workflow management in the Cloud. Last we present our experience in integrating a scientific workflow management system-Swift into the Cloud. We discuss the performance of cluster provisioning within the OpenNebula Cloud platform, the Eucalyptus Cloud platform and Amazon EC2, and we demonstrate the capability and efficiency of the integration using a NASA MODIS image processing workflow and the Montage image mosaic workflow.Note to practitioners. Scientific workflow management plays a very important role for scientific computing and application coordination, while Cloud computing offers scalability and resource on-demand. We devise autonomous methods to integrate scientific workflow management systems with Cloud platforms and also provision resources for large scale workflows, which can facilitate scientists to easily manage their workflows in the Cloud, and take advantage of large scale Cloud resources. There are a few integration options and many challenges in the process, and the experience we gain will help researchers in migrating their workflow management systems and workflow applications into the Cloud. We analyze the major challenges of running scientific workflows on the Cloud.We propose a reference framework to standardize the integration.The implementation experience proves that the framework is feasible and extendible.Cluster-recycling mechanism can improve the resource provisioning efficiency. st Century Smart Manufacturing (SM) is manufacturing in which all information is available when it is needed, where it is needed, and in the form it is most useful (1,2) to drive optimal actions and responses. The 21st Century SM enterprise is data driven, knowledge enabled, and model rich with visibility across the enterprise (internal and external) such that all operating actions are determined and executed proactively by applying the best information and a wide range of performance metrics. SM also encompasses the sophisticated practice of generating and applying data-driven Manufacturing Intelligence throughout the lifecycle of design, engineering, planning and production. Workflow is foundational in orchestrating dynamic, adaptive, actionable decision-making through the contextualization and understanding of data. Pervasive deployment of architecturally consistent workflow applications creates the enterprise environment for manufacturing intelligence. Workflow as a Service (WfaaS) software allows task orchestration and facilitates workflow services and manage environment to integrate interrelated task components. Apps, and toolkits are required to assemble customized SM applications on a common, standards based workflow architecture and deploy on infrastructure that is accessible by small, medium, and large companies. Incorporating dynamic decision-making steps through contextualization of real-time data requires scientific workflow software such as Kepler. By combining workflow, private cloud computing and web services technologies, we built a prototype test bed to test a furnace temperature control model.