<commit_before># What is Pentagon?

**Pentagon is a cli tool to generate repeatable, cloud-based [Kubernetes](https://kubernetes.io/) infrastructure**.
Pentagon is “batteries included”- not only does one get a network with a cluster, but the defaults include these commonly desired features:
- At it's core, powered by Kubernetes. Configured to be highly-available: masters and nodes are clustered
- Segregated multiple development / non-production environments
- VPN-based access control
- A highly-available network, built across multiple Availability Zones

## How does it work?
 **Pentagon produces a directory.** The directory defines a basic set of configurations for [Ansible](https://www.ansible.com/), [Terraform](https://www.terraform.io/), and [kops](https://github.com/kubernetes/kops). When these tools are run in a specific order the result is a VPC with a VPN and a Kubernetes cluster in AWS. (GKE Support is in the works). Pentagon is designed to be customizable but has defaults that fit most software infrastructure needs. 

# Getting Started with Pentagon

## Requirements
* python2 >= 2.7 [Install Python](https://www.python.org/downloads/)
* pip [Install Pip](https://pip.pypa.io/en/stable/installing/)
* git [Install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
* Terraform >=0.9 [Install Terraform ](https://www.terraform.io/downloads.html)
* Ansible [Install Ansible](http://docs.ansible.com/ansible/latest/intro_installation.html)
* Kubectl [Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* kops [Install kops](https://github.com/kubernetes/kops#installing)
* jq [Install JQ](https://stedolan.github.io/jq/download/)

## Installation
* `pip install -e git+https://github.com/reactiveops/pentagon.git#egg=pentagon`
  * The `-e` is important to include. The next steps will fail without it.
  * May require the `python-dev` and `libffi-dev` packages on some Linux distributions
  * Not necessary, but we suggest installing Pentagon into a [VirtualEnv](https://virtualenv.pypa.io/en/stable/)

## Quick Start
### Create a Pentagon Project
* `pentagon start-project <project-name> --aws-access-key <aws-access-key> --aws-secret-key <aws-secret-key> --aws-default-region <aws-default-region>`
  * With the above basic options set, defaults will be set for you. See [Advanced Project Initialization](#advanced-project-initialization) for more options.
  * Arguments may also be set using environment variable in the format `PENTAGON_<argument_name_with_underscores>`.
* `cd <project-name>-infrastructure`

#### Automatic 
* `make all`
  * conducts all following steps up short of Kubernets Cluster Creation
* skip to [Create Kubernetes Cluster](#create-kubernetes-cluster)

#### Manual steps
* `pip install -r requirements.txt`
* `source config/local/env-vars.sh`
  * Sources environment variables required for the following steps. This will be required each time you work with the infrastructure repository or if you move the repository to another location.
* `bash config/local/local-config-init`

### Create a VPC
This creates the VPC and private, public, and admin subnets in that VPC for non Kubernetes resources. Read more about networking [here](network.md).
* `cd default/vpc`
* Edit `terraform.tfvars`and verify the generated `aws_azs` actually exist in `aws_region`
* `make all`
* In `default/clusters/*/vars.sh`, set `VPC_ID` using the newly created VPC ID. You can find that ID in Terraform output or using the AWS web console.

### Configure DNS and Route53
If you don't already have a Route53 Hosted Zone configured, do that now.
* Create a Route53 Hosted Zone (e.g. `pentagon.mycompany.com`)
* In `default/account/vars.yml` set `canonical_zone` to match your Hosted Zone
* In `default/clusters/*/vars.sh`
  * Set `CLUSTER_NAME` to a hostname that ends with your hosted zone (e.g. `working-1.pentagon.mycompany.com`)
  * Set `DNS_ZONE` to your Hosted Zone (e.g. `pentagon.mycompany.com`)

### Setup a VPN
This creates a AWS instance running [OpenVPN](https://openvpn.net/). Read more about the VPN [here](vpn.md).
* From the root of your project run `ansible-galaxy install -r ansible-requirements.yml`
* `cd default/resources/admin-environment`
* In `env.yml`, set the list of user names that should have access to the VPN under `openvpn_clients`. You can add more later.
* Run Ansible a few times
  * Run `ansible-playbook vpn.yml` until it fails on `VPN security groups`
  * Run `ansible-playbook vpn.yml` until it fails `Gathering Facts` after you agree to trust the SSH key for the host.
  * Run `ansible-playbook vpn.yml` one last time and it will succeed.
  * Edit `config/private/ssh_config` and add the IP address from the SSH key prompt to the `#VPN instance` section.

### Configure a Kubernetes Cluster
Pentagon used Kops to create clusters in AWS. The default layout creates configurations for two Kubernetes clusters: `working` and `production`. See [Overview](overview.md) for a more comprehensive description of the directory layout.

* Make sure your KOPS variables are set correctly with `source default/account/vars.sh`
* Move into to the path for the cluster you want to work on with `cd default/clusters/<production|working>`
* Run `bash cluster-config/kops.sh` to create a cluster.spec file for this cluster. This does not create any resources in AWS.

### Create Kubernetes Cluster
* Use the Kops component to create your cluster. [components#kops.cluster]
* By default a `vars.yml` will be created at `default/clusters/working` and `default/clusters/production`. Those files are sufficient to create a cluster using the kops.cluster though, if you are not using `make all` from above you will need to enter `nat_gateways` and `vpc-id` as described in [kops component documentation](components.md#kops-cluster)

* Use [kops](https://github.com/kubernetes/kops/blob/master/docs/cli/kops.md) to manage the cluster if necessary.
  * Run `kops edit cluster <clustername>` to view or edit the `cluster.spec`
  * You may also wish to edit the instance groups prior to cluster creation:
    * `kops get instancegroups --name <clustername>` to list them (one master group per AZ and one node group)
    * `kops edit instancegroups --name <clustername> <instancegroupname>` to edit any of them
* Run `kops update cluster <clustername>` and review the out put to ensure it matches the cluster you wish to create
* Run `kops update cluster <clustername> --yes` to create the cluster
* While waiting for the cluster to create, consult the [kops documentation](https://github.com/kubernetes/kops/blob/master/docs/README.md) for more information about using Kops and interacting with your new cluster

### Creating Resources Outside of Kubernetes

Typically infrastructure will be required outside of your Kubernetes cluster. Other EC2, RDS, or Elasticache instances, etc are often require for an application.

Pentagon convention suggests you use Ansible to create these resources and that the Ansible playbooks can be saved in the `default/resources/` or the `default/clusters/<cluster>/resoures/` directory. This depends on the scope with which the play book will be utilized. If the resources are not specific to either cluster, then we suggest you save it at the `default/resources/` level. Likewise, if it is a resource that will only be used by one cluster, such as a staging database or a production database, then we suggest writing the Ansible playbook at the `default/cluster/<cluster>/resources/` level. Writing Ansible roles can be very helpful to DRY up your resource configurations.


======================================<commit_msg>Fixing references to env-vars.sh<commit_after># What is Pentagon?

**Pentagon is a cli tool to generate repeatable, cloud-based [Kubernetes](https://kubernetes.io/) infrastructure**.
Pentagon is “batteries included”- not only does one get a network with a cluster, but the defaults include these commonly desired features:
- At it's core, powered by Kubernetes. Configured to be highly-available: masters and nodes are clustered
- Segregated multiple development / non-production environments
- VPN-based access control
- A highly-available network, built across multiple Availability Zones

## How does it work?
 **Pentagon produces a directory.** The directory defines a basic set of configurations for [Ansible](https://www.ansible.com/), [Terraform](https://www.terraform.io/), and [kops](https://github.com/kubernetes/kops). When these tools are run in a specific order the result is a VPC with a VPN and a Kubernetes cluster in AWS. (GKE Support is in the works). Pentagon is designed to be customizable but has defaults that fit most software infrastructure needs. 

# Getting Started with Pentagon

## Requirements
* python2 >= 2.7 [Install Python](https://www.python.org/downloads/)
* pip [Install Pip](https://pip.pypa.io/en/stable/installing/)
* git [Install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
* Terraform >=0.9 [Install Terraform ](https://www.terraform.io/downloads.html)
* Ansible [Install Ansible](http://docs.ansible.com/ansible/latest/intro_installation.html)
* Kubectl [Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* kops [Install kops](https://github.com/kubernetes/kops#installing)
* jq [Install JQ](https://stedolan.github.io/jq/download/)

## Installation
* `pip install -e git+https://github.com/reactiveops/pentagon.git#egg=pentagon`
  * The `-e` is important to include. The next steps will fail without it.
  * May require the `python-dev` and `libffi-dev` packages on some Linux distributions
  * Not necessary, but we suggest installing Pentagon into a [VirtualEnv](https://virtualenv.pypa.io/en/stable/)

## Quick Start
### Create a Pentagon Project
* `pentagon start-project <project-name> --aws-access-key <aws-access-key> --aws-secret-key <aws-secret-key> --aws-default-region <aws-default-region>`
  * With the above basic options set, defaults will be set for you. See [Advanced Project Initialization](#advanced-project-initialization) for more options.
  * Arguments may also be set using environment variable in the format `PENTAGON_<argument_name_with_underscores>`.
* `cd <project-name>-infrastructure`

#### Automatic 
* `make all`
  * conducts all following steps up short of Kubernets Cluster Creation
* skip to [Create Kubernetes Cluster](#create-kubernetes-cluster)

#### Manual steps
* `pip install -r requirements.txt`
* `. yam_source config/local/vars.yml`
* `. yam_source config/private/secrets.yml`
  * Sources environment variables required for the following steps. This will be required each time you work with the infrastructure repository or if you move the repository to another location.
* `bash config/local/local-config-init`

### Create a VPC
This creates the VPC and private, public, and admin subnets in that VPC for non Kubernetes resources. Read more about networking [here](network.md).
* `cd default/vpc`
* Edit `terraform.tfvars`and verify the generated `aws_azs` actually exist in `aws_region`
* `make all`
* In `default/clusters/*/vars.sh`, set `VPC_ID` using the newly created VPC ID. You can find that ID in Terraform output or using the AWS web console.

### Configure DNS and Route53
If you don't already have a Route53 Hosted Zone configured, do that now.
* Create a Route53 Hosted Zone (e.g. `pentagon.mycompany.com`)
* In `default/account/vars.yml` set `canonical_zone` to match your Hosted Zone
* In `default/clusters/*/vars.sh`
  * Set `CLUSTER_NAME` to a hostname that ends with your hosted zone (e.g. `working-1.pentagon.mycompany.com`)
  * Set `DNS_ZONE` to your Hosted Zone (e.g. `pentagon.mycompany.com`)

### Setup a VPN
This creates a AWS instance running [OpenVPN](https://openvpn.net/). Read more about the VPN [here](vpn.md).
* From the root of your project run `ansible-galaxy install -r ansible-requirements.yml`
* `cd default/resources/admin-environment`
* In `env.yml`, set the list of user names that should have access to the VPN under `openvpn_clients`. You can add more later.
* Run Ansible a few times
  * Run `ansible-playbook vpn.yml` until it fails on `VPN security groups`
  * Run `ansible-playbook vpn.yml` until it fails `Gathering Facts` after you agree to trust the SSH key for the host.
  * Run `ansible-playbook vpn.yml` one last time and it will succeed.
  * Edit `config/private/ssh_config` and add the IP address from the SSH key prompt to the `#VPN instance` section.

### Configure a Kubernetes Cluster
Pentagon used Kops to create clusters in AWS. The default layout creates configurations for two Kubernetes clusters: `working` and `production`. See [Overview](overview.md) for a more comprehensive description of the directory layout.

* Make sure your KOPS variables are set correctly with `. yam_source config/local/vars.yml && . yam_source config/private/secrets.yml`
* Move into to the path for the cluster you want to work on with `cd default/clusters/<production|working>`
* Run `bash cluster-config/kops.sh` to create a cluster.spec file for this cluster. This does not create any resources in AWS.

### Create Kubernetes Cluster
* Use the Kops component to create your cluster. [components#kops.cluster]
* By default a `vars.yml` will be created at `default/clusters/working` and `default/clusters/production`. Those files are sufficient to create a cluster using the kops.cluster though, if you are not using `make all` from above you will need to enter `nat_gateways` and `vpc-id` as described in [kops component documentation](components.md#kops-cluster)

* Use [kops](https://github.com/kubernetes/kops/blob/master/docs/cli/kops.md) to manage the cluster if necessary.
  * Run `kops edit cluster <clustername>` to view or edit the `cluster.spec`
  * You may also wish to edit the instance groups prior to cluster creation:
    * `kops get instancegroups --name <clustername>` to list them (one master group per AZ and one node group)
    * `kops edit instancegroups --name <clustername> <instancegroupname>` to edit any of them
* Run `kops update cluster <clustername>` and review the out put to ensure it matches the cluster you wish to create
* Run `kops update cluster <clustername> --yes` to create the cluster
* While waiting for the cluster to create, consult the [kops documentation](https://github.com/kubernetes/kops/blob/master/docs/README.md) for more information about using Kops and interacting with your new cluster

### Creating Resources Outside of Kubernetes

Typically infrastructure will be required outside of your Kubernetes cluster. Other EC2, RDS, or Elasticache instances, etc are often require for an application.

Pentagon convention suggests you use Ansible to create these resources and that the Ansible playbooks can be saved in the `default/resources/` or the `default/clusters/<cluster>/resoures/` directory. This depends on the scope with which the play book will be utilized. If the resources are not specific to either cluster, then we suggest you save it at the `default/resources/` level. Likewise, if it is a resource that will only be used by one cluster, such as a staging database or a production database, then we suggest writing the Ansible playbook at the `default/cluster/<cluster>/resources/` level. Writing Ansible roles can be very helpful to DRY up your resource configurations.


======================================
{'fraction_non_alphanumeric': 0.09808379343942839, 'fraction_numerical': 0.0018187723286781422, 'mean_word_length': 4.637495422922006, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 37, 'https://': 32, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '625485', 'n_tokens_mistral': 4422, 'n_tokens_neox': 4211, 'n_words': 1855}