High intensive computation applications can usually take days to months to nish an execution. During this time, it is common to have variations of the available resources when considering that such hardware is usually shared among a plurality of researchers departments within an organization. On the other hand, High Performance Clusters can take advantage of Cloud Computing bursting techniques for the execution of applications together with on-premise resources. In order to meet deadlines, high intensive computational applications can use the Cloud to boost their performance when they are data and task parallel. This article presents an ongoing work towards the use of extended resources of an HPC execution platform together with Cloud. We propose an unied view of such heterogeneous environments and a method that monitors, predicts the application execution time, and dynamically shifts part of the domain previously running in local HPC hardware to be computed in the Cloud, meeting then a specic deadline. The method is exemplied along with a seismic application that, at runtime, adapts itself to move part of the processing to the Cloud (in a movement called bursting) and also auto-scales (the moved part) over cloud nodes. Our preliminary results show that there is an expected overhead for performing this movement and for synchronizing results, but the outcomes demonstrate it is an important feature for meeting deadlines in the case an on-premise cluster is overloaded or cannot provide the capacity needed for a particular project.
Analyzing Cloud as stand alone execution platform for HPC applications, like seismic, the authors of @cite_5 evaluated the Linpack workload on the Amazon EC2 cloud. Their conclusions indicate that the tested cloud environment has a potential, but it is not mature to provide a price-performance for HPC applications. @cite_4 also evaluated the EC2 for a number of kernels used by HPC applications, coming also to a conclusion that such cloud services need an order of magnitude in performance improvement to better serve the scientific community. It is hard to evaluate one provider or another, but they are evolving to offers that are private and with specialized infrastructures, like with GPUs and Infiniband. In our study, we evaluated the Cloud (virtualized nodes) and preliminary results indicate the environment as cost-effective in budget and performance when at least combined with on-premise clusters in a dynamic changing scenario.
@cite_5 @cite_4
1987512728 2088445131 1818445599 1997826804
In this paper, we report on the experimental results of running a large, tightly coupled, distributed multiscale computation over a hybrid High Performance Computing (HPC) infrastructures. We connected EC2 based cloud clusters located in USA to university clusters located in Switzerland. We ran a concurrent multiscale MPI based application on this infrastructure and measured the overhead induced by extending our HPC clusters with EC2 resources. Our results indicate that accommodating some parts of the multiscale computation on cloud resources can lead to low performance without a proper adjustment of CPUs power and workload. However, by enforcing a load-balancing strategy one can benefit from the extra Cloud resources. We connect an EC2-cloud cluster to a university cluster located in Switzerland.We run a distributed multiscale CFD computation over this extended infrastructure.We evaluate and compare the distributed execution to a local execution.We describe our experience of running parallel CFD application on hybrid platforms.Multiscale computation on cloud requires an adjustment of CPUs power and workload. HPC applications are increasingly being used in academia and laboratories for scientific research and in industries for business and analytics. Cloud computing offers the benefits of virtualization, elasticity of resources and elimination of cluster setup cost and time to HPC applications users. However, poor network performance, performance variation and OS noise are some of the challenges for execution of HPC applications on Cloud. In this paper, we propose that Cloud can be viable platform for some HPC applications depending upon application characteristics such as communication volume and pattern and sensitivity to OS noise and scale. We present an evaluation of the performance and cost tradeoffs of HPC applications on a range of platforms varying from Cloud (with and without virtualization) to HPC-optimized cluster. Our results show that Cloud is viable platform for some applications, specifically, non communicationintensive applications such as embarrassingly parallel and tree-structured computations up to high processor count and for communication-intensive applications up to low processor count. High intensive computation applications can usually take days to months to nish an execution. During this time, it is common to have variations of the available resources when considering that such hardware is usually shared among a plurality of researchers departments within an organization. On the other hand, High Performance Clusters can take advantage of Cloud Computing bursting techniques for the execution of applications together with on-premise resources. In order to meet deadlines, high intensive computational applications can use the Cloud to boost their performance when they are data and task parallel. This article presents an ongoing work towards the use of extended resources of an HPC execution platform together with Cloud. We propose an unied view of such heterogeneous environments and a method that monitors, predicts the application execution time, and dynamically shifts part of the domain previously running in local HPC hardware to be computed in the Cloud, meeting then a specic deadline. The method is exemplied along with a seismic application that, at runtime, adapts itself to move part of the processing to the Cloud (in a movement called bursting) and also auto-scales (the moved part) over cloud nodes. Our preliminary results show that there is an expected overhead for performing this movement and for synchronizing results, but the outcomes demonstrate it is an important feature for meeting deadlines in the case an on-premise cluster is overloaded or cannot provide the capacity needed for a particular project. Testing and executing large-scale computational applications in public clouds is becoming prevalent due to cost saving, elasticity, and scalability. However, how to increase the reliability and reduce the cost to run large-scale applications in public clouds is still a big challenge. In this paper, we analyzed the pricing schemes of Amazon Elastic Compute Cloud (EC2) and found the disturbance effect that the price of the spot instances can be heavily affected due to the large number of spot instances required. We proposed a dynamic approach which schedules and runs large-scale computational applications on a dynamic pool of cloud computational instances. We use hybrid instances, including both on-demand instances for high priority tasks and backup, and spot instances for normal computational tasks so as to further reduce the cost without significantly increasing the completion time. Our proposed method takes the dynamic pricing of cloud instances into consideration, and it reduces the cost and tolerates the failures for running large-scale applications in public clouds. We conducted experimental tests and an agent based Scalable complex System modeling for Sustainable city (S3) application is used to evaluate the scalability, reliability and cost saving. The results show that our proposed method is robust and highly flexible for researchers and users to further reduce cost in real practice.