<commit_before># pummeler

This is a set of utilities for analyzing the American Community Survey's Public Use Microdata Sample files (ACS PUMS), mostly following

> Flaxman, Wang, and Smola. Who Supported Obama in 2012? Ecological Inference through Distribution Regression. KDD 2015.
> ([official](http://dx.doi.org/10.1145/2783258.2783300), [from author](http://sethrf.com/files/ecological.pdf))

The package currently only supports the 2006-10 5-year estimates, but more will be added.


## Usage

### Installation

`pip install pummeler`; it will then be available via `import pummler` and install a `pummel` script.

If you prefer, you can also check out the source directory, which should work as long as you put the `pummel` directory on your `sys.path` (or start Python from the root of the checkout). In that case you should use the `pummel` script at the top level of the checkout.


### Getting the census data

First, download the data from the Census site. You probably want the "csv_pus.zip" file, [here](http://www2.census.gov/programs-surveys/acs/data/pums/2010/5-Year/csv_pus.zip) for the 2006-10 data (2.1 GB).


### Picking regions of analysis

Election results are generally reported by counties; PUMS data are in census blockgroups, which are related but not the same. This module ships with regions that merge all overlapping blockgroups / counties, found with [the MABLE/Geocorr tool](http://mcdc2.missouri.edu/websas/geocorr12.html), in Pandas dataframes stored in `pummeler/data/regions.h5`.

Regions are named like `AL_00_01`, which means Alabama's region number 01 in the 2000 geography. If you use 2010 geographies (i.e. you're using ACS data of vintage after 2011 – not yet supported, but it will be), those regions are named `AL_10_01`.

**Note:** Alaska electoral districts are weird. For now, I just lumped all of Alaska into one region.

This was done in the Jupyter notebook [`notebooks/get regions.ipynb`](notebooks/get regions.ipynb). Centroids are calculated in [`notebooks/region centroids.ipynb`](notebooks/region centroids.ipynb), using a shapefile for counties [from here](https://geonet.esri.com/thread/24614).


### Preprocessing
<commit_msg>note about geograhy<commit_after># pummeler

This is a set of utilities for analyzing the American Community Survey's Public Use Microdata Sample files (ACS PUMS), mostly following

> Flaxman, Wang, and Smola. Who Supported Obama in 2012? Ecological Inference through Distribution Regression. KDD 2015.
> ([official](http://dx.doi.org/10.1145/2783258.2783300), [from author](http://sethrf.com/files/ecological.pdf))

The package currently only supports the 2006-10 5-year estimates, but more will be added.


## Usage

### Installation

`pip install pummeler`; it will then be available via `import pummler` and install a `pummel` script.

If you prefer, you can also check out the source directory, which should work as long as you put the `pummel` directory on your `sys.path` (or start Python from the root of the checkout). In that case you should use the `pummel` script at the top level of the checkout.


### Getting the census data

First, download the data from the Census site. You probably want the "csv_pus.zip" file, [here](http://www2.census.gov/programs-surveys/acs/data/pums/2010/5-Year/csv_pus.zip) for the 2006-10 data (2.1 GB).


### Picking regions of analysis

Election results are generally reported by counties; PUMS data are in their own special [Public Use Microdata Areas](https://www.census.gov/geo/reference/puma.html), which are related but not the same. This module ships with regions that merge all overlapping blockgroups / counties, found with [the MABLE/Geocorr tool](http://mcdc2.missouri.edu/websas/geocorr12.html), in Pandas dataframes stored in `pummeler/data/regions.h5`.

Regions are named like `AL_00_01`, which means Alabama's region number 01 in the 2000 geography. If you use 2010 geographies (i.e. you're using ACS data of vintage after 2011 – not yet supported, but it will be), those regions are named `AL_10_01`.

**Note:** Alaska electoral districts are weird. For now, I just lumped all of Alaska into one region.

This was done in the Jupyter notebook [`notebooks/get regions.ipynb`](notebooks/get regions.ipynb). Centroids are calculated in [`notebooks/region centroids.ipynb`](notebooks/region centroids.ipynb), using a shapefile for counties [from here](https://geonet.esri.com/thread/24614).

**TODO:** Could switch to precinct-level results, which should end up with more regions in the end. Results are available [here](http://projects.iq.harvard.edu/eda/data), including shapefiles if you go into the state-by-state section, so it shouldn't be *too* much work.


### Preprocessing

{'fraction_non_alphanumeric': 0.09123624047417443, 'fraction_numerical': 0.03386960203217612, 'mean_word_length': 4.658682634730539, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1232267', 'n_tokens_mistral': 1533, 'n_tokens_neox': 1365, 'n_words': 601}