<commit_before>#National Voter File
This project will build a database containing voter files for all fifty state. It uses a data model that represents a national voter files and associated campaign measures in a shared data warehouse. 

This model is based on standard data warehouse modelling techniques with an aim of making useful queries easy to generate while preserving history of the data. If you are unfamiliar with dimensional data modelling, you may enjoy [this article](https://dwbi.org/data-modelling/dimensional-model/1-dimensional-modeling-guide).

We assume that the movement app will maintain an Operational Data Store for transactional data related to administering and operating the application. This Online Analytic Processing database will perform the heavy lifting of historic analysis and voter identification.

##Design Objectives
This model was created with the following objectives in mind:

1. We will receive voter data from a variety of sources, not all of which are reliable
2. People change addresses
3. Electoral boundaries change over time
4. People live together in households
5. People change their names and genders

The model will be used for maintaining a national voter file that can be enriched with private data sources as well as crowd-sourced edits. Over time this voter file database will accumulate history of voter location and affiliation. 

Some of the data will be exposed as an open-source resource for any political campaign. An authorization model will be developed to allow campaigns to create private resources with data sharing between allied campaigns.

The voter file will be leveraged to power campaign donation, canvassing and other campaign interactions.
<commit_msg>Added some info about the data model<commit_after>#National Voter File
This project will build a database containing voter files for all fifty state. It uses a data model that represents a national voter files and associated campaign measures in a shared data warehouse. 

This model is based on standard data warehouse modelling techniques with an aim of making useful queries easy to generate while preserving history of the data. If you are unfamiliar with dimensional data modelling, you may enjoy [this article](https://dwbi.org/data-modelling/dimensional-model/1-dimensional-modeling-guide).

We assume that the movement app will maintain an Operational Data Store for transactional data related to administering and operating the application. This Online Analytic Processing database will perform the heavy lifting of historic analysis and voter identification.

##Design Objectives
This model was created with the following objectives in mind:

1. We will receive voter data from a variety of sources, not all of which are reliable
2. People change addresses
3. Electoral boundaries change over time
4. People live together in households
5. People change their names and genders

The model will be used for maintaining a national voter file that can be enriched with private data sources as well as crowd-sourced edits. Over time this voter file database will accumulate history of voter location and affiliation. 

Some of the data will be exposed as an open-source resource for any political campaign. An authorization model will be developed to allow campaigns to create private resources with data sharing between allied campaigns.

The voter file will be leveraged to power campaign donation, canvassing and other campaign interactions.

## Database Architecture
We have a document describing the data architecture available as a [google doc](https://docs.google.com/document/d/169mIkiIdl4OetbGvnbVCzq9Srw9PKCsB6U1CErTD9aI/edit?usp=sharing)

Take a look inside [SQL Folder](https://github.com/getmovement/national-voter-file/tree/master/src/main/sql) in the github to see how to view and modify the relational model.

## How we work
Join our slack [#national-voter-file] (http://slack.progcode.co/)

We have a [google doc](https://docs.google.com/spreadsheets/d/1CtNePb4LQSz-pk8UF58wwuVoBIc_YDAsBJZnIk7hKso/edit?usp=sharing) with information on obtaining each of the voter files

{'fraction_non_alphanumeric': 0.03738771546491867, 'fraction_numerical': 0.0065549890750182084, 'mean_word_length': 5.005830903790088, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6103832', 'n_tokens_mistral': 999, 'n_tokens_neox': 917, 'n_words': 566}