<commit_before># DC Campaign Finance Report Analysis

## Setup for processing data

You need to install [Node.js](https://nodejs.org). On Linux or OS X, you can use 
[Node Version Manager](https://github.com/creationix/nvm). Once you have it installed
go to this directory and run

    npm install

For simplicity this currently uses SQLite for the database, so no database server is
needed, but it should be relatively easy to modify it to use MySQL or PostgreSQL.

## Download contributions

I want to automate this, but the .dc.gov websites are not friendly
to scraping or much of anything else.

Go to https://efiling.ocf.dc.gov/ContributionExpenditure and select
"Principle Campaign Committee" and "Contributions", click "Date" and
enter "01/01/2016" in "From Date", then click "Search".

On the next page, click the dropdown next to "Export" and select
"CSV". A file named "ContributionsExpendituresSearchResult.csv" will
be downloaded. Move it to this directory. Then to fix the character
encoding and delete the first line (a title that comes before the
header line), run

    iconv -f UTF-16 -t UTF-8 ContributionsExpendituresSearchResult.csv | tail -n +2 > contributions.csv

(If you don't have iconv on your system, you can convert the encoding
by some other means, and you can delete the first line in a text editor
instead of using tail.)

## Download committees

Go to https://efiling.ocf.dc.gov/Disclosure and select "Principle
Campaign Committee" and the election year, then click "Search".
Click the dropdown next to "Export" and select "CSV". A file named
"RegistrantDisclosureSearchResult.csv" will
be downloaded. Move it to this directory. Then to fix the character
encoding and delete the first line (a title that comes before the
header line), run

    iconv -f UTF-16 -t UTF-8 RegistrantDisclosureSearchResult.csv | tail -n +2 > committees.csv

Download expenditures
---------------------

Do the same as in "Download contributions", but select "Expenditures" instead of
"Contributions".  Move the downloaded file to expenditures-raw.csv in this
directory, and run

    iconv -f UTF-16 -t UTF-8 expenditures-raw.csv | tail -n +2 > expenditures.csv<commit_msg>add more to readme<commit_after># DC Campaign Finance Report Analysis

## Setup for processing data

You need to install [Node.js](https://nodejs.org). On Linux or OS X, you can use 
[Node Version Manager](https://github.com/creationix/nvm). Once you have it installed
go to this directory and run

    npm install

For simplicity this currently uses SQLite for the database, so no database server is
needed, but it should be relatively easy to modify it to use MySQL or PostgreSQL.

## Download contributions

I want to automate this, but the .dc.gov websites are not friendly
to scraping or much of anything else, and I haven't had time to 
work it out.

Go to https://efiling.ocf.dc.gov/ContributionExpenditure and select
"Principle Campaign Committee" and "Contributions", click "Date" and
enter "01/01/2016" in "From Date", then click "Search".

On the next page, click the dropdown next to "Export" and select
"CSV". A file named "ContributionsExpendituresSearchResult.csv" will
be downloaded. Move it to this directory. Then to fix the character
encoding and delete the first line (a title that comes before the
header line), run

    iconv -f UTF-16 -t UTF-8 ContributionsExpendituresSearchResult.csv | tail -n +2 > contributions.csv

(If you don't have iconv on your system, you can convert the encoding
by some other means, and you can delete the first line in a text editor
instead of using tail.)

## Download committees

Go to https://efiling.ocf.dc.gov/Disclosure and select "Principle
Campaign Committee" and the election year, then click "Search".
Click the dropdown next to "Export" and select "CSV". A file named
"RegistrantDisclosureSearchResult.csv" will
be downloaded. Move it to this directory. Then to fix the character
encoding and delete the first line (a title that comes before the
header line), run

    iconv -f UTF-16 -t UTF-8 RegistrantDisclosureSearchResult.csv | tail -n +2 > committees.csv

## Download expenditures

Do the same as in "Download contributions", but select "Expenditures" instead of
"Contributions".  Move the downloaded file to expenditures-raw.csv in this
directory, and run

    iconv -f UTF-16 -t UTF-8 expenditures-raw.csv | tail -n +2 > expenditures.csv

## Load data into database

Once you have the files committees.csv, contributions.csv, and expenditures.csv,
create and load the database by running

    ./load-data.js

You may want to install the SQLite command-line client (`sudo apt install sqlite3`
on Ubuntu) for looking at the database and trying out queries.
{'fraction_non_alphanumeric': 0.07329509241555131, 'fraction_numerical': 0.008710431272572764, 'mean_word_length': 4.139737991266376, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4750528', 'n_tokens_mistral': 1365, 'n_tokens_neox': 1248, 'n_words': 658}