<commit_before># NVF with Docker

This directory contains docker resources to bring up a local national voter file.

We use [docker-compose](https://docs.docker.com/compose/) to manage interactions between the containers. This is already installed with Docker if you are using Windows or Mac.

## Getting started:

1. Move to the docker directory: `% cd docker`
2. Build the docker containers: `% docker-compose build`
3. Launch the warehouse with the command `% docker-compose up`

4. Use your favorite (postgres-supporting) SQLing tool to connect (just fill in the ones your tool requires):

* URL: `jdbc:postgresql://postgis:5432/VOTER`
* Driver: `org.postgresql.Driver`

* Host: `localhost`
* Port: `5432`
* Database: `VOTER`
* User: `postgres`
* Password: <PASSWORD>

## ETL

This container is intended to run [Pentaho Data Integration](http://community.pentaho.com/projects/data-integration/) transforms and python scripts with some handy modules. These scripts and transforms are how we enrich, clean, and load data into the postgres database.

Typically, you will invoke commands in this container as 
`% docker-compose run etl` 

## Running ETL Scripts

We've provided a shell script for loading in a 1,000 row sample of Washington state data. Take a look at buildWashington.sh for setting up a simple warehouse, or as guidance for running your own ETL jobs. It assumes that the voter file can be found in the data directory of this repo
(which is in .gitignore so you have to construct your own local version)

Copy those files into data/Washington directory of the national-voter-file repo

You can get samples of this data in our [private dropbox](https://www.dropbox.com/work/getmovement%20Team%20Folder) message us on the slack channel to get access.

### populateDateDimension.ktr
This transform creates records in the date dimension that has a row for every single date going out twenty years. This is required before loading any records into the warehouse.

### saveWashingtonPrecincts.ktr
This transform joins the Washington State precinct file with IDs from [Open Civic Data API](https://opencivicdata.readthedocs.io/en/latest/ocdids.html). This data is required before the voter file itself is loaded.

<commit_msg>Fixed a misunderstadning in the README<commit_after># NVF with Docker

This directory contains docker resources to bring up a local national voter file.

We use [docker-compose](https://docs.docker.com/compose/) to manage interactions between the containers. This is already installed with Docker if you are using Windows or Mac.

## Getting started:

1. Move to the docker directory: `% cd docker`
2. Build the docker containers: `% docker-compose build`
3. Launch the warehouse with the command `% docker-compose up`

4. Use your favorite (postgres-supporting) SQLing tool to connect (just fill in the ones your tool requires):

* URL: `jdbc:postgresql://postgis:5432/VOTER`
* Driver: `org.postgresql.Driver`

* Host: `localhost` from your workstation, `postgis` when accessed from inside the ETL docker container
* Port: `5432`
* Database: `VOTER`
* User: `postgres`
* Password: <PASSWORD>

## ETL

This container is intended to run [Pentaho Data Integration](http://community.pentaho.com/projects/data-integration/) transforms and python scripts with some handy modules. These scripts and transforms are how we enrich, clean, and load data into the postgres database.

Typically, you will invoke commands in this container as 
`% docker-compose run etl` 

## Running ETL Scripts

We've provided a shell script for loading in a 1,000 row sample of Washington state data. Take a look at buildWashington.sh for setting up a simple warehouse, or as guidance for running your own ETL jobs. It assumes that the voter file can be found in the data directory of this repo
(which is in .gitignore so you have to construct your own local version)

Copy those files into data/Washington directory of the national-voter-file repo

You can get samples of this data in our [private dropbox](https://www.dropbox.com/work/getmovement%20Team%20Folder) message us on the slack channel to get access.

### populateDateDimension.ktr
This transform creates records in the date dimension that has a row for every single date going out twenty years. This is required before loading any records into the warehouse.

### saveWashingtonPrecincts.ktr
This transform joins the Washington State precinct file with IDs from [Open Civic Data API](https://opencivicdata.readthedocs.io/en/latest/ocdids.html). This data is required before the voter file itself is loaded.


{'fraction_non_alphanumeric': 0.06966586590958725, 'fraction_numerical': 0.008735531775496833, 'mean_word_length': 4.682382133995037, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2079525', 'n_tokens_mistral': 1289, 'n_tokens_neox': 1204, 'n_words': 613}