<commit_before># DATA Act Broker Installation and Setup

If you want to install and run a local DATA Act broker at your agency, the instructions below are for you. Because we designed the broker's components to be used together, the installation and setup directions for the DATA Act core, broker API, validator, and broker website are consolidated here.

**Note:** If you're a developer who wants to contribute to this code or modify it for use at your agency, your setup instructions are a little different. See the [contributing guide](CONTRIBUTING.md "DATA Act broker developer install") for more details.

## Install Backend Prerequisites

The core, the broker API, and the validator compose the backend of the DATA Act broker. These components are necessary for running the broker website, so we'll install them first.

To run the backend DATA Act broker applications, you'll need to have the following up and running on your machine:

* PostgreSQL
* DynamoDB
* Python

### PostgreSQL

[PostgreSQL](https://en.wikipedia.org/wiki/PostgreSQL) is an object-relational database management system (ORDBMS) with an emphasis on extensibility and standards-compliance.

You can find PostgreSQL installers on [EnterpriseDB](http://www.enterprisedb.com/products-services-training/pgdownload). Download and run the correct installer for your operating system (we recommend PostgreSQL 9.4.x.). As you proceed through the installation process, note your choices for port number, username, and password. You will need those when setting up the broker. For other PostgreSQL installation options, more complete documentation is available on the PostgreSQL [wiki](https://wiki.postgresql.org/wiki/Detailed_installation_guides).

**Note:** If you're setting up the broker on Mac OSX, we recommend using [homebrew](http://brew.sh) to install PostgreSQL.

## Create a Local DynamoDB

**Optional**

Note: If you'd prefer to use an Amazon-hosted version of DynamoDB, you can skip this section for installing DynamoDB on your machine.

Otherwise, you'll need to set up a local version of DynamoDB. DynamoDB local is a Java executable that runs on Windows, Mac, and Linux systems and is compatible with version 7+ of the Java Development Kit, which you will have to install as a first step: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html "download Java Development Kit").


After installing Java, [download the local DynamoDB files](http://dynamodb-local.s3-website-us-west-2.amazonaws.com/dynamodb_local_latest.zip "download local DynamoDB") and follow [Amazon's instructions](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLocal.html "running DynamoDB on your computer") to run it.

Note that a local version of DynamoDB is **not** recommend for production.

Don't worry about creating tables in DynamoDB: the broker's [initialization process](#initialize-broker-backend-applications "initialize broker") handles that.

### Python and Virtualenv

The broker's backend components currently run on Python 2.7 (with the intention of migrating to Python 3x for the final release). Install Python 2.7 and pip (Python's package manager) using [these directions](https://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-python "Install Python").

We highly recommend creating a [Python virtual environment](https://virtualenv.readthedocs.org/en/latest/installation.html "Python virtualenv") to house the DATA Act broker components. A virtual environment (aka *virtualenv*) will isolate the broker software and its libraries from the libraries running on your local system and prevent potential conflicts. In addition, we recommend using [virtualenvwrapper](https://virtualenvwrapper.readthedocs.org/en/latest/install.html "Python virtualenvwrapper documentation") to manage your Python environments.

1. Use pip to install virtualenv: `pip install virtualenv`
2. Use pip to install virtualenvwrapper: `pip install virtualenvwrapper`
3. If you're using Windows, also do `pip install virtualenvwrapper-powershell`.
3. Tell virtualenvwrapper where on your machine to create virtual environments and add it to your profile. This is a one-time virtualenvwrapper setup step, and the process varies by operating system. [This tutorial](http://newcoder.io/begin/setup-your-machine/ "Python: setting up your computer") covers setting up virtualenvwrapper on OSX, Linux, and Windows.
4. Create a virtual environment for the DATA Act software. In this example we've named the environment *data-act*, but you can call it anything: `mkvirtualenv data-act`.

    **Note:** If you're running multiple versions of Python on your machine, you can make sure your data act environment is running the correct Python version by pointing to a specific binary: `mkvirtualenv --python=[path to installed Python 2.7 executable] data-act`

5. You should see some output that looks similar to the example below. Essentially, this command creates and activates a new virtualenv named `data-act` with its own set of Python libraries.  Anything you pip install from this point forward will be installed into the *data-act* environment rather than your machine's global Python environment. Your command line prompt indicates which (if any) virtualenv is active.

        rebeccasweger@GSA-xs-MacBook-Pro-4 ~
        $ mkvirtualenv --python=/usr/local/bin/python2.7 data-act-test
        Running virtualenv with interpreter /usr/local/bin/python2.7
        New python executable in data-act-test/bin/python2.7
        Also creating executable in data-act-test/bin/python
        Installing setuptools, pip...done.

        (data-act-test)
        rebeccasweger@GSA-xs-MacBook-Pro-4 ~

6. This new environment will be active until you run the `deactivate` command. You can re-activate the environment again at any time by typing `workon data-act`.

## Create an S3 Bucket

**Optional**

The DATA Act broker can upload file submissions to an AWS S3 bucket if you so choose. If you're planning to store file submissions locally instead of using S3, skip to the [next section](#create-a-local-dynamodb "Create DynamoDB").

Assuming that you already have an AWS account, [create an AWS S3 bucket](http://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html "Create a bucket") that will receive file submissions from the broker.

## Create a Local DynamoDB

**Optional**

The broker tracks sessions in an Amazon DynamoDB table. If you'd like to use DynamoDB in the cloud with your AWS account, you can skip this section.

Otherwise, you'll need to set up a local version of DynamoDB. This requires Java JDK 6 or higher to be installed, which can be done using the following command on Red Hat based systems:


```bash
$ su -c "yum install java-1.7.0-openjdk"
```

For Ubuntu based systems the `apt-get` can be used instead

```bash
sudo apt-get install default-jre
```

Once Java is installed, you can download the local DynamoDB [here](http://dynamodb-local.s3-website-us-west-2.amazonaws.com/dynamodb_local_latest.zip). Instructions to launch the local version once downloaded can be found in [AWS's User Guide](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLocal.html) along with the various options. Note that a local version of DynamoDB is **not** recommend for production.

Don't worry about creating tables in DynamoDB: the broker's [initialization process](#initialize-broker-backend-applications "initialize broker") handles that.

## Set Up AWS Permissions and Credentials

### Create S3 Only Role

When storing files on S3, it is a best practice to use AWS roles with the DATA Act broker. AWS roles provide a safe, automated key management mechanism to access and use AWS resources. At a minimum, this role should be granted Full S3 access permissions.

The DATA Act broker supports the creation of Security Token Service (STS) tokens that limit a user's permissions to only file uploads. To set this up, create an IAM Role on the targeted AWS account. This role should have the following permission JSON, where the `s3-bucket-name` is the name of the S3 bucket created above.

```json
{
    "Version": "2016-01-29",
    "Statement": [
        {
            "Sid": "Stmt123456",
            "Effect": "Allow",
            "Action": [
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::s3-bucket-name",
                "arn:aws:s3:::s3-bucket-name/*",
            ]
        }
    ]
}
```

In addition to the permission JSON, create a Trust Relationship for the IAM role, allowing the broker to assume the S3 uploading role during token creation.

If not using a local Dynamo, the broker should also be granted read/write permissions to DynamoDB. The following JSON can be added to the role to grant this access:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "dynamodb:*"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/BrokerSession"
    }
  ]
}
```
The `REGION` should be replaced with region of the AWS account and the `ACCOUNT_ID` should be replaced with the AWS account ID.

### AWS Command Line Interface (CLI) Tools

AWS credentials should be managed by the AWS CLI.

Install the Python version of AWS Command Line Interface (CLI) tools following [these directions](https://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-the-aws-cli-using-pip "install AWS CLI").

After you've installed the AWS CLI tools, configure them:

        $ aws configure

As prompted, enter the following information about your AWS account. Specify `json` as the default output format.

* `AWS Access Key ID`
* `AWS Secret Access Key`
* `Default region name [us-east-1]`
* `Default output format [json]`

## Install Broker Backend Applications

Make sure that your Python virtual environment is activated (replace *data-act* with the name of the environment you created above):

        $ workon data-act

Use Python's pip installer to install each broker back-end application:

        $ pip install git+git://github.com/fedspendingtransparency/data-act-core.git
        $ pip install git+git://github.com/fedspendingtransparency/data-act-broker.git
        $ pip install git+git://github.com/fedspendingtransparency/data-act-validator.git

## Initialize Broker Backend Applications

If everything installed correctly in the steps above, you're ready to initialize the broker backend applications. To do this, you'll run a few scripts that set up the databases and create the configuration files.

**Note:** The scripts below delete existing broker databases and data. DO NOT run them in a production environment.

Run the broker API initialization script:

        $ sudo webbroker -i
        [when prompted, ented your sudo password]

You will now answer a series of questions to guide you through broker configuration. Below is some guidance, but the answers may vary depending on your specific environment:

* `Would you like to configure your S3 connection`: `y` if you're setting up an Amazon Web Service S3 bucket to use for broker file submission, `n` if you're running everything locally with no cloud dependencies

    * `Enter you bucket name`: the name of the S3 bucket you're using for broker file submission
    * `Enter your S3 role`: the [AWS role you created](https://github.com/fedspendingtransparency/data-act-core#creating-s3-only-role "Creating S3-only role") for uploading files to the broker<commit_msg>add directions for consolidated config file<commit_after># DATA Act Broker Installation and Setup

If you want to install and run a local DATA Act broker at your agency, the instructions below are for you. Because we designed the broker's components to be used together, the installation and setup directions for the DATA Act core, broker API, validator, and broker website are consolidated here.

**Note:** If you're a developer who wants to contribute to this code or modify it for use at your agency, your setup instructions are a little different. See the [contributing guide](CONTRIBUTING.md "DATA Act broker developer install") for more details.

## Install Backend Prerequisites

The core, the broker API, and the validator compose the backend of the DATA Act broker. These components are necessary for running the broker website, so we'll install them first.

To run the backend DATA Act broker applications, you'll need to have the following up and running on your machine:

* PostgreSQL
* DynamoDB
* Python

### PostgreSQL

[PostgreSQL](https://en.wikipedia.org/wiki/PostgreSQL) is an object-relational database management system (ORDBMS) with an emphasis on extensibility and standards-compliance.

You can find PostgreSQL installers on [EnterpriseDB](http://www.enterprisedb.com/products-services-training/pgdownload). Download and run the correct installer for your operating system (we recommend PostgreSQL 9.4.x.). As you proceed through the installation process, note your choices for port number, username, and password. You will need those when setting up the broker. For other PostgreSQL installation options, more complete documentation is available on the PostgreSQL [wiki](https://wiki.postgresql.org/wiki/Detailed_installation_guides).

**Note:** If you're setting up the broker on Mac OSX, we recommend using [homebrew](http://brew.sh) to install PostgreSQL.

## Create a Local DynamoDB

**Optional**

Note: If you'd prefer to use an Amazon-hosted version of DynamoDB, you can skip this section for installing DynamoDB on your machine.

Otherwise, you'll need to set up a local version of DynamoDB. DynamoDB local is a Java executable that runs on Windows, Mac, and Linux systems and is compatible with version 7+ of the Java Development Kit, which you will have to install as a first step: [http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html "download Java Development Kit").


After installing Java, [download the local DynamoDB files](http://dynamodb-local.s3-website-us-west-2.amazonaws.com/dynamodb_local_latest.zip "download local DynamoDB") and follow [Amazon's instructions](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLocal.html "running DynamoDB on your computer") to run it.

Note that a local version of DynamoDB is **not** recommend for production.

Don't worry about creating tables in DynamoDB: the broker's [initialization process](#initialize-broker-backend-applications "initialize broker") handles that.

### Python and Virtualenv

The broker's backend components currently run on Python 2.7 (with the intention of migrating to Python 3x for the final release). Install Python 2.7 and pip (Python's package manager) using [these directions](https://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-python "Install Python").

We highly recommend creating a [Python virtual environment](https://virtualenv.readthedocs.org/en/latest/installation.html "Python virtualenv") to house the DATA Act broker components. A virtual environment (aka *virtualenv*) will isolate the broker software and its libraries from the libraries running on your local system and prevent potential conflicts. In addition, we recommend using [virtualenvwrapper](https://virtualenvwrapper.readthedocs.org/en/latest/install.html "Python virtualenvwrapper documentation") to manage your Python environments.

1. Use pip to install virtualenv: `pip install virtualenv`
2. Use pip to install virtualenvwrapper: `pip install virtualenvwrapper`
3. If you're using Windows, also do `pip install virtualenvwrapper-powershell`.
3. Tell virtualenvwrapper where on your machine to create virtual environments and add it to your profile. This is a one-time virtualenvwrapper setup step, and the process varies by operating system. [This tutorial](http://newcoder.io/begin/setup-your-machine/ "Python: setting up your computer") covers setting up virtualenvwrapper on OSX, Linux, and Windows.
4. Create a virtual environment for the DATA Act software. In this example we've named the environment *data-act*, but you can call it anything: `mkvirtualenv data-act`.

    **Note:** If you're running multiple versions of Python on your machine, you can make sure your data act environment is running the correct Python version by pointing to a specific binary: `mkvirtualenv --python=[path to installed Python 2.7 executable] data-act`

5. You should see some output that looks similar to the example below. Essentially, this command creates and activates a new virtualenv named `data-act` with its own set of Python libraries.  Anything you pip install from this point forward will be installed into the *data-act* environment rather than your machine's global Python environment. Your command line prompt indicates which (if any) virtualenv is active.

        rebeccasweger@GSA-xs-MacBook-Pro-4 ~
        $ mkvirtualenv --python=/usr/local/bin/python2.7 data-act-test
        Running virtualenv with interpreter /usr/local/bin/python2.7
        New python executable in data-act-test/bin/python2.7
        Also creating executable in data-act-test/bin/python
        Installing setuptools, pip...done.

        (data-act-test)
        rebeccasweger@GSA-xs-MacBook-Pro-4 ~

6. This new environment will be active until you run the `deactivate` command. You can re-activate the environment again at any time by typing `workon data-act`.

## Create an S3 Bucket

**Optional**

The DATA Act broker can upload file submissions to an AWS S3 bucket if you so choose. If you're planning to store file submissions locally instead of using S3, skip to the [next section](#create-a-local-dynamodb "Create DynamoDB").

Assuming that you already have an AWS account, [create an AWS S3 bucket](http://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html "Create a bucket") that will receive file submissions from the broker.

## Create a Local DynamoDB

**Optional**

The broker tracks sessions in an Amazon DynamoDB table. If you'd like to use DynamoDB in the cloud with your AWS account, you can skip this section.

Otherwise, you'll need to set up a local version of DynamoDB. This requires Java JDK 6 or higher to be installed, which can be done using the following command on Red Hat based systems:


```bash
$ su -c "yum install java-1.7.0-openjdk"
```

For Ubuntu based systems the `apt-get` can be used instead

```bash
sudo apt-get install default-jre
```

Once Java is installed, you can download the local DynamoDB [here](http://dynamodb-local.s3-website-us-west-2.amazonaws.com/dynamodb_local_latest.zip). Instructions to launch the local version once downloaded can be found in [AWS's User Guide](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLocal.html) along with the various options. Note that a local version of DynamoDB is **not** recommend for production.

Don't worry about creating tables in DynamoDB: the broker's [initialization process](#initialize-broker-backend-applications "initialize broker") handles that.

## Set Up AWS Permissions and Credentials

### Create S3 Only Role

When storing files on S3, it is a best practice to use AWS roles with the DATA Act broker. AWS roles provide a safe, automated key management mechanism to access and use AWS resources. At a minimum, this role should be granted Full S3 access permissions.

The DATA Act broker supports the creation of Security Token Service (STS) tokens that limit a user's permissions to only file uploads. To set this up, create an IAM Role on the targeted AWS account. This role should have the following permission JSON, where the `s3-bucket-name` is the name of the S3 bucket created above.

```json
{
    "Version": "2016-01-29",
    "Statement": [
        {
            "Sid": "Stmt123456",
            "Effect": "Allow",
            "Action": [
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::s3-bucket-name",
                "arn:aws:s3:::s3-bucket-name/*",
            ]
        }
    ]
}
```

In addition to the permission JSON, create a Trust Relationship for the IAM role, allowing the broker to assume the S3 uploading role during token creation.

If not using a local Dynamo, the broker should also be granted read/write permissions to DynamoDB. The following JSON can be added to the role to grant this access:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "dynamodb:*"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/BrokerSession"
    }
  ]
}
```
The `REGION` should be replaced with region of the AWS account and the `ACCOUNT_ID` should be replaced with the AWS account ID.

### AWS Command Line Interface (CLI) Tools

AWS credentials should be managed by the AWS CLI.

Install the Python version of AWS Command Line Interface (CLI) tools following [these directions](https://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-the-aws-cli-using-pip "install AWS CLI").

After you've installed the AWS CLI tools, configure them:

        $ aws configure

As prompted, enter the following information about your AWS account. Specify `json` as the default output format.

* `AWS Access Key ID`
* `AWS Secret Access Key`
* `Default region name [us-east-1]`
* `Default output format [json]`

## Install Broker Backend Applications

Make sure that your Python virtual environment is activated (replace *data-act* with the name of the environment you created above):

        $ workon data-act

Use Python's pip installer to install each broker back-end application:

        $ pip install git+git://github.com/fedspendingtransparency/data-act-core.git
        $ pip install git+git://github.com/fedspendingtransparency/data-act-broker.git
        $ pip install git+git://github.com/fedspendingtransparency/data-act-validator.git

## Create Broker Config File

Before running the broker, you'll need to provide a few configuration options. Use the provided sample config file as a starting point:

1. Navigate to the location of the data-act-core code.
2. From the `data-act-core` directory, go to `dataactcore` and open the file called `config_example.yml` in a text editor.
3. Save `config_example.yml` as `config.yml`.
4. Update the values in `config.yml` as appropriate for your environment. In many cases the default values should work just fine. The most important config values to change when getting started are:
    * under _broker_:
            * `full_url`
            * `reply_to_email`
            * `admin_email`
            * `admin_password`
            * `broker_files`
    * under _services_:
            * `error_report_path`
    * under _db_:
            * `username`
            * `password`
    * under _logging_:
            * `log_files`

## Initialize Broker Backend Applications

If everything installed correctly in the steps above, you're ready to initialize the broker backend applications. To do this, you'll run a few scripts that set up the databases and create the configuration files.

**Note:** The scripts below delete existing broker databases and data. DO NOT run them in a production environment.

Run the broker API initialization script:

        $ sudo webbroker -i
        [when prompted, ented your sudo password]

You will now answer a series of questions to guide you through broker configuration. Below is some guidance, but the answers may vary depending on your specific environment:

* `Would you like to configure your S3 connection`: `y` if you're setting up an Amazon Web Service S3 bucket to use for broker file submission, `n` if you're running everything locally with no cloud dependencies

    * `Enter you bucket name`: the name of the S3 bucket you're using for broker file submission
    * `Enter your S3 role`: the [AWS role you created](https://github.com/fedspendingtransparency/data-act-core#creating-s3-only-role "Creating S3-only role") for uploading files to the broker
{'fraction_non_alphanumeric': 0.07672613148331682, 'fraction_numerical': 0.007928642220019821, 'mean_word_length': 4.115547106041403, 'pattern_counts': {'":': 22, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 14, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4461127', 'n_tokens_mistral': 6610, 'n_tokens_neox': 6241, 'n_words': 2971}