<commit_before># take-shelter

## importing data from a postgresql custom dump

```ruby
pg*restore --verbose  --no-acl --no-owner --clean -d shelter-development DUMPNAME.dump
```

## importing data from files/first principles

This process creates the `hurricaneshelter` table, which is why it doesn't have
a migration.

1. Have postgresql, postgis, and gdal with postgres support installed.
`brew install gdal --with-postgres; brew install postgis` does this.
2. `rake db:create db:migrate` if you haven't created the database already.
3. `rake import:all`
<commit_msg>readme has more information about what's in the database<commit_after># take-shelter

## database configuration and entities

This project depends on PostgreSQL and PostGIS. The import scripts produce two
tables and two views.

* Table `zipcode`: a relatively simple import from geojson available from
http://gis.mdc.opendata.arcgis.com/datasets/3815768dd1c84f429b1d8a76c44e9bc3_0
* Table `hurricaneshelter`: an import from
http://gisweb.miamidade.gov/GISSelfServices/GeographicData/MDGeographicData.html
but with the wkb_geometry column re-set to a more sensible SRID.
* View `zipcodes`: a rails-friendly version of the `zipcode` table, with a
centroid instead of a huge polygon to save bandwidth.
* View `hurricaneshelters`: a rails-friendly version of the `hurricaneshelter`
table, with renamed fields.

## importing data from a postgresql custom dump

```ruby
pg*restore --verbose  --no-acl --no-owner --clean -d shelter-development DUMPNAME.dump
```

## importing data from files/first principles

This process creates the `hurricaneshelter` table, which is why it doesn't have
a migration.

1. Have postgresql, postgis, and gdal with postgres support installed.
`brew install gdal --with-postgres; brew install postgis` does this on Mac with
Homebrew.
2. `rake db:create db:migrate` if you haven't created the database already.
3. `rake import:all`

{'fraction_non_alphanumeric': 0.08571428571428572, 'fraction_numerical': 0.014545454545454545, 'mean_word_length': 4.783783783783784, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3328319', 'n_tokens_mistral': 618, 'n_tokens_neox': 571, 'n_words': 232}