<commit_before><!--main.md-->

Cloudbreak
================

*Cloudbreak is a powerful left surf that breaks over a coral reef, a mile off southwest the island of Tavarua, Fiji.*

*Cloudbreak is a cloud agnostic Hadoop as a Service API. Abstracts the provisioning and ease management and monitoring of on-demand clusters.*

Cloudbreak [API documentation](http://docs.cloudbreak.apiary.io/).

<!--main.md-->

<!--overview.md-->

##Overview

Cloudbreak is a RESTful Hadoop as a Service API. Once it is deployed in your favourite servlet container exposes a REST API allowing to span up Hadoop clusters of arbitrary sizes on your selected cloud provider. Provisioning Hadoop has never been easier.
Cloudbreak is built on the foundation of cloud providers API (Amazon AWS, Microsoft Azure, Google Cloud Compute...), Apache Ambari, Docker containers, Serf and dnsmasq.

##Benefits

###Secure
Supports basic, token based and OAuth2 authentication model. The cluster is provisioned in a logically isolated network (Virtual Private Cloud) of your favourite cloud provider.
Cloudbreak does not store or manages your cloud credentials - it is the end user's responsibility to link the Cloudbreak user with her/his cloud account. We provide utilities to ease this process (IAM on Amazon, certificates on Azure).

###Elastic
Using Cloudbreak API you can provision an arbitrary number of Hadoop nodes - the API does the hard work for you, and span up the infrastructure, configure the network and the selected Hadoop components and services without any user interaction.
POST once and use it anytime after.

###Scalable
As your workload changes, the API allows you to add or remove nodes on the fly. Cloudbreak does the hard work of reconfiguring the infrastructure, provision or decommission Hadoop nodes and let the cluster be continuously operational.
Once provisioned, new nodes will take up the load and increase the cluster throughput.

###Declarative Hadoop clusters
We support declarative Hadoop cluster creation - using blueprints. Blueprints are a declarative definition of a Hadoop cluster. With a blueprint, you specify a stack, the component layout and the configurations to materialise a Hadoop cluster instance. Hostgroups defined in blueprints can be associated to different VPC subnets and availability zones, thus you can span up a highly available cluster running on different datacenters or availability zones.

###Flexible
You have the option to choose your favourite cloud provider and their different pricing models. The API translated the calls towards different vendors - you develop and use one common API, no need to rewrite your code when changing between cloud providers.

<!--overview.md-->

<!--quickstart.md-->

##QuickStart and installation

We provide you three different ways to start using Cloudbreak. The simplest and easiest solution is hosted by SequenceIQ, however we have two DIY _(do it yourself)_ options as well.

###Using Cloudbreak - hosted by SequenceIQ
The easiest way to start your own Hadoop cluster in your favourite cloud provider is to use our hosted solution. We host, maintain and support [Cloudbreak](https://cloudbreak.sequenceiq.com/) for you.

Please note that Cloudbreak is launching Hadoop clusters on the user's behalf - on different cloud providers. We do not store your cloud provider account details (such as username, password, keys, private SSL certificates, etc), but work around the concept that Identity and Access Management is fully controlled by you - the end user.

Though Cloudbreak controls your Hadoop cluster lifecycle (start, stop, pause), we **do not** have access to the launched instances. The Hadoop clusters created by Cloudbreak are private to you.


###DIY - Running Cloudbreak API using Docker

####Database
The only dependency that Cloudbreak needs is a postgresql database. The easiest way to spin up a postgresql is of course Docker. Run it first with this line:
```
docker run -d --name="postgresql" -p 5432:5432 -v /tmp/data:/data -e USER="seqadmin" -e DB="cloudbreak" -e PASS="<PASSWORD>_" paintedfox/postgresql
```
####Cloudbreak REST API
After postgresql is running, Cloudbreak can be started locally in a Docker container with the following command. By linking the database container, the necessary environment variables for the connection are set. The postgresql address can be set explicitly through the environment variable: CB_DB_PORT_5432_TCP_ADDR.
```
VERSION=0.1-20140623140412

docker run -d --name cloudbreak \
-e "VERSION=$VERSION" \
-e "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" \
-e "AWS_SECRET_KEY=$AWS_SECRET_KEY" \
-e "CB_HBM2DDL_STRATEGY=create" \
-e "CB_SMTP_SENDER_USERNAME=$CB_SMTP_SENDER_USERNAME" \
-e "CB_SMTP_SENDER_PASSWORD=$CB_SMTP_SENDER_PASSWORD" \
-e "CB_SMTP_SENDER_HOST=$CB_SMTP_SENDER_HOST" \
-e "CB_SMTP_SENDER_PORT=$CB_SMTP_SENDER_PORT" \
-e "CB_SMTP_SENDER_FROM=$CB_SMTP_SENDER_FROM" \
-e "CB_HOST_ADDR=$CB_HOST_ADDR" \
-e "CB_AZURE_IMAGE_URI=$CB_AZURE_IMAGE_URI" \
-e "CB_BLUEPRINT_DEFAULTS=$CB_BLUEPRINT_DEFAULTS" \
-e "CB_SNS_SSL=false"
--link postgresql:cb_db -p 8080:8080 \
dockerfile/java bash \
-c 'curl -o /tmp/cloudbreak-$VERSION.jar https://s3-eu-west-1.amazonaws.com/seq-repo/releases/com/sequenceiq/cloudbreak/$VERSION/cloudbreak-$VERSION.jar && java -jar /tmp/cloudbreak-$VERSION.jar'

```

Note: The system properties prefixed with MAIL_SENDER_ are the SNMP settings required to send emails.

###DIY - Running Cloudbreak API on the host

The Cloudbreak application can be run outside a docker container - on an arbitrary machine. The only thing required for this is to have the following list of environment variables set:

```

# The address of the host running the application. This should be reachable from the internet (it's also used to deal with SNS notifications)
CB_HOST_ADDR=

# database settings
CB_DB_ENV_USER=
CB_DB_ENV_PASS=
CB_DB_PORT_5432_TCP_ADDR=
CB_DB_PORT_5432_TCP_PORT=

# SMTP settings for sending confirmation and password recovery emails to customers
CB_SMTP_SENDER_USERNAME=
CB_SMTP_SENDER_PASSWORD=
CB_SMTP_SENDER_HOST=
CB_SMTP_SENDER_PORT=
CB_SMTP_SENDER_FROM=

# AWS related (optional) settings - not setting them causes AWS related operations to fail
AWS_ACCESS_KEY_ID=
AWS_SECRET_KEY=
<commit_msg>CLOUD-24: deploy changes<commit_after><!--main.md-->

Cloudbreak
================

*Cloudbreak is a powerful left surf that breaks over a coral reef, a mile off southwest the island of Tavarua, Fiji.*

*Cloudbreak is a cloud agnostic Hadoop as a Service API. Abstracts the provisioning and ease management and monitoring of on-demand clusters.*

Cloudbreak [API documentation](http://docs.cloudbreak.apiary.io/).

<!--main.md-->

<!--overview.md-->

##Overview

Cloudbreak is a RESTful Hadoop as a Service API. Once it is deployed in your favourite servlet container exposes a REST API allowing to span up Hadoop clusters of arbitrary sizes on your selected cloud provider. Provisioning Hadoop has never been easier.
Cloudbreak is built on the foundation of cloud providers API (Amazon AWS, Microsoft Azure, Google Cloud Compute...), Apache Ambari, Docker containers, Serf and dnsmasq.

##Benefits

###Secure
Supports basic, token based and OAuth2 authentication model. The cluster is provisioned in a logically isolated network (Virtual Private Cloud) of your favourite cloud provider.
Cloudbreak does not store or manages your cloud credentials - it is the end user's responsibility to link the Cloudbreak user with her/his cloud account. We provide utilities to ease this process (IAM on Amazon, certificates on Azure).

###Elastic
Using Cloudbreak API you can provision an arbitrary number of Hadoop nodes - the API does the hard work for you, and span up the infrastructure, configure the network and the selected Hadoop components and services without any user interaction.
POST once and use it anytime after.

###Scalable
As your workload changes, the API allows you to add or remove nodes on the fly. Cloudbreak does the hard work of reconfiguring the infrastructure, provision or decommission Hadoop nodes and let the cluster be continuously operational.
Once provisioned, new nodes will take up the load and increase the cluster throughput.

###Declarative Hadoop clusters
We support declarative Hadoop cluster creation - using blueprints. Blueprints are a declarative definition of a Hadoop cluster. With a blueprint, you specify a stack, the component layout and the configurations to materialise a Hadoop cluster instance. Hostgroups defined in blueprints can be associated to different VPC subnets and availability zones, thus you can span up a highly available cluster running on different datacenters or availability zones.

###Flexible
You have the option to choose your favourite cloud provider and their different pricing models. The API translated the calls towards different vendors - you develop and use one common API, no need to rewrite your code when changing between cloud providers.

<!--overview.md-->

<!--quickstart.md-->

##QuickStart and installation

We provide you three different ways to start using Cloudbreak. The simplest and easiest solution is hosted by SequenceIQ, however we have two DIY _(do it/deploy it yourself)_ options as well.

###Hosted by SequenceIQ - Cloudbreak UI and API  
The easiest way to start your own Hadoop cluster in your favourite cloud provider is to use our hosted solution. We host, maintain and support [Cloudbreak](https://cloudbreak.sequenceiq.com/) for you.

Please note that Cloudbreak is launching Hadoop clusters on the user's behalf - on different cloud providers. We do not store your cloud provider account details (such as username, password, keys, private SSL certificates, etc), but work around the concept that Identity and Access Management is fully controlled by you - the end user.

Though Cloudbreak controls your Hadoop cluster lifecycle (start, stop, pause), we **do not** have access to the launched instances. The Hadoop clusters created by Cloudbreak are private to you.


###DIY - Deploying Cloudbreak API using Docker

####Database
The only dependency that Cloudbreak needs is a postgresql database. The easiest way to spin up a postgresql is of course Docker. Run it first with this line:
```
docker run -d --name="postgresql" -p 5432:5432 -v /tmp/data:/data -e USER="seqadmin" -e DB="cloudbreak" -e PASS="<PASSWORD>_" paintedfox/postgresql
```
####Cloudbreak REST API
After postgresql is running, Cloudbreak can be started locally in a Docker container with the following command. By linking the database container, the necessary environment variables for the connection are set. The postgresql address can be set explicitly through the environment variable: CB_DB_PORT_5432_TCP_ADDR.
```
VERSION=0.1-20140623140412

docker run -d --name cloudbreak \
-e "VERSION=$VERSION" \
-e "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" \
-e "AWS_SECRET_KEY=$AWS_SECRET_KEY" \
-e "CB_HBM2DDL_STRATEGY=create" \
-e "CB_SMTP_SENDER_USERNAME=$CB_SMTP_SENDER_USERNAME" \
-e "CB_SMTP_SENDER_PASSWORD=$<PASSWORD>" \
-e "CB_SMTP_SENDER_HOST=$CB_SMTP_SENDER_HOST" \
-e "CB_SMTP_SENDER_PORT=$CB_SMTP_SENDER_PORT" \
-e "CB_SMTP_SENDER_FROM=$CB_SMTP_SENDER_FROM" \
-e "CB_HOST_ADDR=$CB_HOST_ADDR" \
-e "CB_AZURE_IMAGE_URI=$CB_AZURE_IMAGE_URI" \
-e "CB_BLUEPRINT_DEFAULTS=$CB_BLUEPRINT_DEFAULTS" \
-e "CB_SNS_SSL=false"
--link postgresql:cb_db -p 8080:8080 \
dockerfile/java bash \
-c 'curl -o /tmp/cloudbreak-$VERSION.jar https://s3-eu-west-1.amazonaws.com/seq-repo/releases/com/sequenceiq/cloudbreak/$VERSION/cloudbreak-$VERSION.jar && java -jar /tmp/cloudbreak-$VERSION.jar'

```

Note: The system properties prefixed with MAIL_SENDER_ are the SNMP settings required to send emails.

###DIY - Deploying Cloudbreak API on any host

The Cloudbreak application can be run outside a docker container - on an arbitrary machine. The only thing required for this is to have the following list of environment variables set:

```

# The address of the host running the application. This should be reachable from the internet (it's also used to deal with SNS notifications)
CB_HOST_ADDR=

# database settings
CB_DB_ENV_USER=
CB_DB_ENV_PASS=
CB_DB_PORT_5432_TCP_ADDR=
CB_DB_PORT_5432_TCP_PORT=

# SMTP settings for sending confirmation and password recovery emails to customers
CB_SMTP_SENDER_USERNAME=
CB_SMTP_SENDER_PASSWORD=
CB_SMTP_SENDER_HOST=
CB_SMTP_SENDER_PORT=
CB_SMTP_SENDER_FROM=

# AWS related (optional) settings - not setting them causes AWS related operations to fail
AWS_ACCESS_KEY_ID=
AWS_SECRET_KEY=

{'fraction_non_alphanumeric': 0.06747584444621896, 'fraction_numerical': 0.007825600894354388, 'mean_word_length': 5.032755298651252, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2677302', 'n_tokens_mistral': 3624, 'n_tokens_neox': 3424, 'n_words': 1593}