<commit_before><!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
##Table of Contents##

- [Overview: AWS CoreOS cluster provisioning with [Terraform](http://www.terraform.io/downloads.html)](#overview-aws-coreos-cluster-provisioning-with-terraformhttpwwwterraformiodownloadshtml)
- [Install tools and setup AWS credentials](#install-tools-and-setup-aws-credentials)
- [Quick Start](#quick-start)
- [Build multi-node cluster](#build-multi-node-cluster)
- [Destroy all resources](#destroy-all-resources)
- [Manage individual platform Resources](#manage-individual-platform-resources)
- [How Does it work](#how-does-it-work)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Overview: AWS CoreOS cluster provisioning with [Terraform](http://www.terraform.io/downloads.html)

This is a practical implementation of [CoreOS cluster architectures ] (https://coreos.com/os/docs/latest/cluster-architectures.html) built on AWS. The cluster follows CoreOS production cluster model that contains 3-node etcd cluster in an autoscalting group, a central service node that you can run shared services such as CI, logging and monigoring, a private docker registry, and a fleet of workers to run other service containers. 

The entire infrastructure is managed by Terraform. 

AWS compoments includes: VPC, security groups, IAM, S3, ELB, Route53, Autoscaling, RDS etc. 

AWS resources are defined in Terraform resource folders. The build process will copy all resources defined in the repository to a *build* directory. The view, plan, apply, and destroy operations are performed under *build*, keepting the original Terraform files in the repo intact. The *build* directory is ignored in .gitignore so that you don't accidentally checkin sensitive data. 


**WIP**

## Install Tools and Setup AWS credentials

1. Install [Terraform](http://www.terraform.io/downloads.html)

    For MacOS,
    ```
    $ brew update
    $ brew install terraform
    ```
    or
    ```
    $ mkdir -p ~/bin/terraform
    $ cd ~/bin/terraform
    $ curl -L -O https://dl.bintray.com/mitchellh/terraform/terraform_0.6.0_darwin_amd64.zip
    $ unzip terraform_0.6.0_darwin_amd64.zip
    ```

1. Install AWS CLI
    ```
    $ brew install awscli
    ```
    or

    ```
    $ sudo easy_install pip
    $ sudo pip install --upgrade awscli
    ```

1. Install [Jq](http://stedolan.github.io/jq/)
    ```
    $ brew install jq
    ```

1. Setup AWS Credentials at [AWS Console](https://console.aws.amazon.com/)
    1. Create a group `coreos-cluster` with `AaminstratorAccess` policy.
    2. Create a user `coreos-cluster` and download user credentials.
    3. Add user `coreos-cluster` to group `coreos-cluster`.

1. Configure AWS profile with `coreos-cluster` credentials
    ```
    $ aws configure --profile coreos-cluster
    ```


## Quick Start

This default build will create one etcd node and one worker node cluster in a VPC, with application buckets for data, necessary iam roles, polices, keypairs and keys. The nodes are t2.micro instance and run the latest CoreOS beta release.
Reources are defined under aws-terraform/resources/terraform directory. You should review and make changes there if needed. 

To build:
```
$ git clone git@github.com:xuwang/aws-terraform.git
$ cd aws-terraform
$ make
... build steps info ...
... at last, shows the worker's ip:
worker public ips: 172.16.31.10
...
```
To see the list of resources created:
```<commit_msg>formatting<commit_after><!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
##Table of Contents##

- [Overview: AWS CoreOS cluster provisioning with [Terraform](http://www.terraform.io/downloads.html)](#overview-aws-coreos-cluster-provisioning-with-terraformhttpwwwterraformiodownloadshtml)
- [Install tools and setup AWS credentials](#install-tools-and-setup-aws-credentials)
- [Quick start](#quick-start)
- [Build multi-node cluster](#build-multi-node-cluster)
- [Destroy all resources](#destroy-all-resources)
- [Manage individual platform Resources](#manage-individual-platform-resources)
- [How does it work](#how-does-it-work)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Overview: AWS CoreOS cluster provisioning with [Terraform](http://www.terraform.io/downloads.html)

This is a practical implementation of [CoreOS cluster architectures ] (https://coreos.com/os/docs/latest/cluster-architectures.html) built on AWS. The cluster follows CoreOS production cluster model that contains 3-node etcd cluster in an autoscalting group, a central service node that you can run shared services such as CI, logging and monigoring, a private docker registry, and a fleet of workers to run other service containers. 

The entire infrastructure is managed by Terraform. 

AWS compoments includes: VPC, security groups, IAM, S3, ELB, Route53, Autoscaling, RDS etc. 

AWS resources are defined in Terraform resource folders. The build process will copy all resources defined in the repository to a *build* directory. The view, plan, apply, and destroy operations are performed under *build*, keepting the original Terraform files in the repo intact. The *build* directory is ignored in .gitignore so that you don't accidentally checkin sensitive data. 


**WIP**

## Install Tools and Setup AWS credentials

1. Install [Terraform](http://www.terraform.io/downloads.html)

    For MacOS,
    ```
    $ brew update
    $ brew install terraform
    ```
    or
    ```
    $ mkdir -p ~/bin/terraform
    $ cd ~/bin/terraform
    $ curl -L -O https://dl.bintray.com/mitchellh/terraform/terraform_0.6.0_darwin_amd64.zip
    $ unzip terraform_0.6.0_darwin_amd64.zip
    ```

1. Install AWS CLI
    ```
    $ brew install awscli
    ```
    or

    ```
    $ sudo easy_install pip
    $ sudo pip install --upgrade awscli
    ```

1. Install [Jq](http://stedolan.github.io/jq/)
    ```
    $ brew install jq
    ```

1. Setup AWS Credentials at [AWS Console](https://console.aws.amazon.com/)
    1. Create a group `coreos-cluster` with `AaminstratorAccess` policy.
    2. Create a user `coreos-cluster` and download user credentials.
    3. Add user `coreos-cluster` to group `coreos-cluster`.

1. Configure AWS profile with `coreos-cluster` credentials
    ```
    $ aws configure --profile coreos-cluster
    ```


## Quick start

This default build will create one etcd node and one worker node cluster in a VPC, with application buckets for data, necessary iam roles, polices, keypairs and keys. The nodes are t2.micro instance and run the latest CoreOS beta release.
Reources are defined under aws-terraform/resources/terraform directory. You should review and make changes there if needed. 

To build:
```
$ git clone <EMAIL>:xuwang/aws-terraform.git
$ cd aws-terraform
$ make
... build steps info ...
... at last, shows the worker's ip:
worker public ips: 172.16.31.10
...
```
To see the list of resources created:
```
{'fraction_non_alphanumeric': 0.10338028169014085, 'fraction_numerical': 0.009014084507042254, 'mean_word_length': 3.797972972972973, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 6, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2154969', 'n_tokens_mistral': 2146, 'n_tokens_neox': 2059, 'n_words': 776}