<commit_before># Builder Services Development Environment

## Overview
This document outlines the steps to start and run a Builder environment for development. The builder environment includes the builder services, as well as the depot web site.

## Pre-Reqs
1. Use a Linux OS - either native or VM.
1. Clone the habitat repo to your local filesystem. (If using a VM, you should use the local filesystem for the hab repo instead of using a mounted filesystem, otherwise things may fail later when running services).
1. The sample commands below use the 'httpie' tool. Install it if not present on your system (https://github.com/jkbrzt/httpie).
1. A recent version of the habitat cli should also be installed in your dev environment (https://www.habitat.sh/docs/get-habitat/)
1. Ensure that `sudo -E` functions correctly. Some OSes don't respect the -E flag. If this is the case, remove the `secure_path` section of your sudoers file.
1. Copy `habitat-builder-dev.2017-10-02.private-key.pem` from 1Password into `/src/.secrets/builder-github-app.pem`

## Bootstrap the OS with required packages
You need to make sure you have the required packages installed.
Run the appropriate shell script under the `support/linux` folder to install the packages.
Refer to [BUILDING.md](./BUILDING.md) doc for the detailed steps.

## Create Builder encryption key pair
If you haven't already done this earlier, you'll need to create a key pair for builder to encrypt credentials.
Do this via the `hab user key generate bldr` command. This will generate a key pair with the prefix `bldr`
in your local `hab/cache/keys` directory. Copy the key pair over to some location that is accessible by the running services.
You will need to specify that location in the config files below.

## Create configuration files
Some capabilities (such as allowing builder permissions, and turning on auto-building when new packages are uploaded to the depot) require passing in a custom config to the Builder services.

Create the following files somewhere on your local filesystem (Note: the client_id and client_secret below are for development purposes only):

`config_api.toml`
```toml
[depot]
builds_enabled = true
key_dir = "/path/to/bldr-key-pair"

[github]
url = "https://api.github.com"
client_id = "0c2f738a7d0bd300de10"
client_secret = "438223113eeb6e7edf2d2f91a232b72de72b9bdf"
```

`config_jobsrv.toml`
```toml
key_dir = "/path/to/bldr-key-pair"

[archive]
backend = "local"
local_dir = "/tmp"
```

`config_scheduler.toml`
```toml
auth_token = "<your github token>"
depot_url = "http://localhost:9636"
```

`config_sessionsrv.toml`
```toml
[permissions]
admin_team = 1995301
build_worker_teams = [1995301]
early_access_teams = [1995301]

[github]
url = "https://api.github.com"
client_id = "0c2f738a7d0bd300de10"
client_secret = "438223113eeb6e7edf2d2f91a232b72de72b9bdf"
```

`config_worker.toml`
```toml
auth_token = "<your github token>"
depot_url = "http://localhost:9636"
auto_publish = true
```


(Note: If you want your log files to persist across restarts of your development machine, replace `/tmp` with some other directory. It *must* exist and be writable before you start the job server).

Now, modify the `Procfile` (located in your hab repo in the `support` folder) to point the api, sessionsrv, jobsrv, scheduler and worker services to the previously created config files, e.g.

```
api: target/debug/bldr-api start --path /tmp/depot --config /home/your_alias/habitat/config_api.toml
sessionsrv: target/debug/bldr-session-srv start --config /home/your_alias/habitat/config_sessionsrv.toml
worker: target/debug/bldr-worker start --config /home/your_alias/habitat/config_worker.toml
jobsrv: target/debug/bldr-job-srv start --config /home/your_alias/habitat/config_jobsrv.toml
scheduler: target/debug/bldr-scheduler start --config /home/your_alias/habitat/config_scheduler.toml
```

## Build the Builder services for the first time
1. Open a new terminal window.
1. Run `make build-srv`

Note: this only needs to be done the first time to get the pieces in place. If you clean the project, make sure to unset the variables show below when you do a clean build, otherwise the build will look for packages on your local build service!<commit_msg>Update builder dev doc<commit_after># Builder Services Development Environment

## Overview
This document outlines the steps to start and run a Builder environment for development. The builder environment includes the builder services, as well as the depot web site.

## Pre-Reqs
1. Use a Linux OS - either native or VM.
1. Clone the habitat repo to your local filesystem. (If using a VM, you should use the local filesystem for the hab repo instead of using a mounted filesystem, otherwise things may fail later when running services).
1. The sample commands below use the 'httpie' tool. Install it if not present on your system (https://github.com/jkbrzt/httpie).
1. A recent version of the habitat cli should also be installed in your dev environment (https://www.habitat.sh/docs/get-habitat/)
1. Ensure that `sudo -E` functions correctly. Some OSes don't respect the -E flag. If this is the case, remove the `secure_path` section of your sudoers file.
1. Copy `habitat-builder-dev.2017-10-02.private-key.pem` from 1Password into `/src/.secrets/builder-github-app.pem`

## Bootstrap the OS with required packages
You need to make sure you have the required packages installed.
Run the appropriate shell script under the `support/linux` folder to install the packages.
Refer to [BUILDING.md](./BUILDING.md) doc for the detailed steps.

## Create Builder encryption key pair
If you haven't already done this earlier, you'll need to create a key pair for builder to encrypt credentials.
Do this via the `hab user key generate bldr` command. This will generate a key pair with the prefix `bldr`
in your local `hab/cache/keys` directory. Copy the key pair over to some location that is accessible by the running services.
You will need to specify that location in the config files below.

## Create configuration files
Some capabilities (such as allowing builder permissions, and turning on auto-building when new packages are uploaded to the depot) require passing in a custom config to the Builder services.

Create the following files somewhere on your local filesystem (Note: the client_id and client_secret below are for development purposes only):

`config_api.toml`
```toml
[depot]
builds_enabled = true
non_core_builds_enabled = true
key_dir = "/path/to/bldr-key-pair"
```

`config_jobsrv.toml`
```toml
key_dir = "/path/to/bldr-key-pair"

[archive]
backend = "local"
local_dir = "/tmp"
```

`config_scheduler.toml`
```toml
```

`config_sessionsrv.toml`
```toml
[permissions]
admin_team = 1995301
build_worker_teams = [1995301]
early_access_teams = [1995301]
```

`config_worker.toml`
```toml
auth_token = "<<PASSWORD>>"
bldr_url = "http://localhost:9636"
auto_publish = true
```


(Note: If you want your log files to persist across restarts of your development machine, replace `/tmp` with some other directory. It *must* exist and be writable before you start the job server).

Now, modify the `Procfile` (located in your hab repo in the `support` folder) to point the api, sessionsrv, jobsrv, scheduler and worker services to the previously created config files, e.g.

```
api: target/debug/bldr-api start --path /tmp/depot --config /home/your_alias/habitat/config_api.toml
sessionsrv: target/debug/bldr-session-srv start --config /home/your_alias/habitat/config_sessionsrv.toml
worker: target/debug/bldr-worker start --config /home/your_alias/habitat/config_worker.toml
jobsrv: target/debug/bldr-job-srv start --config /home/your_alias/habitat/config_jobsrv.toml
scheduler: target/debug/bldr-scheduler start --config /home/your_alias/habitat/config_scheduler.toml
```

## Build the Builder services for the first time
1. Open a new terminal window.
1. Run `make build-srv`

Note: this only needs to be done the first time to get the pieces in place. If you clean the project, make sure to unset the variables show below when you do a clean build, otherwise the build will look for packages on your local build service!
{'fraction_non_alphanumeric': 0.08487446417636252, 'fraction_numerical': 0.019350887936313534, 'mean_word_length': 4.659043659043659, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4904025', 'n_tokens_mistral': 2527, 'n_tokens_neox': 2382, 'n_words': 1038}