Integrating Apache Airavata with Docker, Marathon, and Mesos
Science Gateways provide scientists with tools for creating, executing, and monitoring scientific experiments on multiple resource infrastructures. Apache Airavata abstracts interactions between gateways and distributed computing infrastructures like Extreme Science and Engineering Discovery Environment, international grids, and campus clusters. Airavata consists of several component services such as the API server, Orchestrator, Workflow Interpreter, Credential Store, and Application Factory. In addition, Airavata uses third party software, including RabbbitMQ for messaging, MySQL for production database management, and Apache Zookeeper for internal communications. In this paper, we discuss our initial experiences with leveraging open source technologies to manage Airavata and its dependent components to deploy, detect, and restart failed components in an auto scaling platform. Such capabilities will allow Airavata services to be deployed in a wide area, large Virtual Machine (VM) based cluster, and a developer's laptop. The emerging technologies in cloud computing and Big Data that address these needs are the following: Docker, Marathon, and Apache Mesos. Docker is a Linux based lightweight container that allows different applications to run isolated from each other but safely share the machine's resources. Docker images of applications can be published in registries and retrieved for execution in the target infrastructures. Marathon provides a cluster wide init and control system for services, including Docker containers. Mesos provides a cluster wide framework to schedule tasks based on fine grained resource needs. Mesosphere provides the packages, scripts, and web interface to ease the use of these technologies. We present the design, experience, and lessons learned from integrating Mesos, Docker, and Marathon with Apache Airavata. Copyright (c) 2015 John Wiley Sons, Ltd.
Concurr. Comput. Pract. Exp.