<commit_before>Mapusaurus
=======

## Description 

This small piece of software loads institution data from HMDA, and maps HMDA
reporter ids to National Information Center ids for easy identification. 


## Data

The data you can load is:

* HMDA Transmittal Sheet
* HMDA Reporter Panel 

Both are available from the FFIEC. 

Here are the 2012 files:

Transmittal sheet:
http://www.ffiec.gov/hmdarawdata/OTHER/2012HMDAInstitutionRecords.zip

Reporter panel:
http://www.ffiec.gov/hmdarawdata/OTHER/2012HMDAReporterPanel.zip

## Requirements 

This currently uses: 
Django 1.6
Python 3.2.3

Postgres 9.1.13
(You could likely use other databases, I just haven't tested them)

There's also a requirements.txt file in the repository root directory.  


## Loading the data

This uses South. 

To create the tables, you need to run:

```
    python manage.py migrate respondants
```

There's also a fixture that you need to load some information from:

```
    python manage.py loaddata agency
```

This loads static regulator agency data. 

Download the two transmittal sheet and reporter panel flat files. 

There are two management commands that will load data, and need to be run 
in the following order:

``` 
1. python manage.py load_transmittal <path/to/transmittal_sheet>
2. python manage.py load_reporter_panel <path/to/reporter_panel>
```

## GEO

The 'geo' application requires GeoDjango and PostGIS. Follow the instructions
for installing GeoDjango. 

Here are some separate instructions for running the geo application. 

```
    python manage.py migrate geo
```

Currently, we load census tract, county, and CBSA files. You can download them 
from the census' FTP site:

```
ftp://ftp2.census.gov/geo/tiger/TIGER2013/TRACT/
ftp://ftp2.census.gov/geo/tiger/TIGER2013/COUNTY/
ftp://ftp2.census.gov/geo/tiger/TIGER2013/CBSA/
```

This is how you load the data:

```
    # This example only loads census tracts from IL (FIPS code: 17); repeat 
    # for other states as needed
    python manage.py load_geos_from /path/to/tl_2013_17_tract.shp
    python manage.py load_geos_from /path/to/tl_2013_us_county.shp
    python manage.py load_geos_from /path/to/tl_2013_us_cbsa.shp
```


## Census Data

The 'censusdata' app loads census data to the census tracts found in the 'geo'
application. As such, 'censusdata' relies on 'geo'.

First, run migrate to create the appropriate tables<commit_msg>Update README with new commands<commit_after>Mapusaurus
=======

## Description 

This small piece of software loads institution data from HMDA, and maps HMDA
reporter ids to National Information Center ids for easy identification. 


## Data

The data you can load is:

* HMDA Transmittal Sheet
* HMDA Reporter Panel 

Both are available from the FFIEC. 

Here are the 2012 files:

Transmittal sheet:
http://www.ffiec.gov/hmdarawdata/OTHER/2012HMDAInstitutionRecords.zip

Reporter panel:
http://www.ffiec.gov/hmdarawdata/OTHER/2012HMDAReporterPanel.zip

## Requirements 

This currently uses: 
Django 1.6
Python 3.2.3

Postgres 9.1.13
(You could likely use other databases, I just haven't tested them)

There's also a requirements.txt file in the repository root directory.  


## Loading the data

This uses South. 

To create the tables, you need to run:

```
    python manage.py migrate respondants
```

There's also a fixture that you need to load some information from:

```
    python manage.py loaddata agency
```

This loads static regulator agency data. 

Download the two transmittal sheet and reporter panel flat files. 

There are two management commands that will load data, and need to be run 
in the following order:

``` 
1. python manage.py load_transmittal <path/to/transmittal_sheet>
2. python manage.py load_reporter_panel <path/to/reporter_panel>
```

## GEO

The 'geo' application requires GeoDjango and PostGIS. Follow the instructions
for installing GeoDjango. 

Here are some separate instructions for running the geo application. 

```
    python manage.py migrate geo
```

Currently, we load census tract, county, and CBSA files. You can download them 
from the census' FTP site:

```
ftp://ftp2.census.gov/geo/tiger/TIGER2013/TRACT/
ftp://ftp2.census.gov/geo/tiger/TIGER2013/COUNTY/
ftp://ftp2.census.gov/geo/tiger/TIGER2013/CBSA/
```

This is how you load the data:

```
    # This example only loads census tracts from IL (FIPS code: 17); repeat 
    # for other states as needed
    python manage.py load_geos_from /path/to/tl_2013_17_tract.shp
    python manage.py load_geos_from /path/to/tl_2013_us_county.shp
    python manage.py load_geos_from /path/to/tl_2013_us_cbsa.shp
```

These import scripts are set up to update geos in place -- no need to delete
records manually.

Once census tracts and counties are loaded, run the following command to
associate census tracts with their CBSAs.

```
    python manage.py set_tract_csa_cbsa
```


## Census Data

The 'censusdata' app loads census data to the census tracts found in the 'geo'
application. As such, 'censusdata' relies on 'geo'.

First, run migrate to create the appropriate tables
{'fraction_non_alphanumeric': 0.08003153952296471, 'fraction_numerical': 0.021289178001182733, 'mean_word_length': 4.033730158730159, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '997771', 'n_tokens_mistral': 1764, 'n_tokens_neox': 1556, 'n_words': 601}