<commit_before>OA Machine
==========

Scripts for running OpenAddresses on a complete data set and publishing
the results. Uses [OpenAddresses](https://github.com/openaddresses/openaddresses)
data sources to work.

Status
------

This code is being used to process the complete OA dataset on an expected-weekly
basis, with output visible at [data.openaddresses.io](http://data.openaddresses.io).

[![Build Status](https://travis-ci.org/openaddresses/machine.svg?branch=master)](https://travis-ci.org/openaddresses/machine)

Usage
-----

Machine supports two modes. Continuous integration (CI) mode listens for tasks
from the OpenAddresses Github repository, includes a persistent web server and
database, and supports a flexible number of worker processes to deal with large
changes quickly. Batch mode processes the entire OpenAddresses source
collection at once, and is intended to be run periodically to reflect new
submissions.

### CI Mode

Installation scripts for preparing a fresh install of Ubuntu 14.04 can be found
in `chef`. You will need a pre-installed PostgreSQL database initialized with
the schema `openaddr/ci/schema.pgsql` and Amazon S3 bucket with credentials.
After editing `chef/role-webhooks.json` and `chef/role-worker.json`, run them
from a Git checkout like this:

    sudo apt-get update
    sudo chef/run.sh webhooks
    sudo chef/run.sh worker

`webhooks` will install Apache, a web application to listen for new tasks, and
a small queue observer to watch for completed tasks. `worker` will install the
`openaddr-ci-worker` utility to watch for newly-scheduled tasks.

Run a single source locally with `openaddr-process-one`:

    openaddr-process-one -l <log> <path to source JSON> <output directory>

### Batch Mode & CI Workers

To run batch mode with existing CI workers and queue, prepare a complete set of
sources from master branch with `openaddr-enqueue-sources`:

    openaddr-enqueue-sources -d <database URL> -t <Github token> -o <repo owner> -r <repo name>

Development
-----------

Test the OpenAddresses machine with `test.py`:

    python test.py

Run the webhook server, queue listener, and worker processes:

    python run-debug-webhooks.py
    python -m openaddr.ci.run_dequeue
    
    python -m openaddr.ci.worker

Modify the contents of [`openaddr/paths.py`](openaddr/paths.py) with locations
of your local [openaddresses](https://github.com/openaddresses/openaddresses).

Extras
------

Convert remote ESRI feature services to GeoJSON with `openaddr-esri2geojson`:

    openaddr-esri2geojson <ESRI URL> <GeoJSON path><commit_msg>Removed dangling mention of openaddr.paths<commit_after>OA Machine
==========

Scripts for running OpenAddresses on a complete data set and publishing
the results. Uses [OpenAddresses](https://github.com/openaddresses/openaddresses)
data sources to work.

Status
------

This code is being used to process the complete OA dataset on an expected-weekly
basis, with output visible at [data.openaddresses.io](http://data.openaddresses.io).

[![Build Status](https://travis-ci.org/openaddresses/machine.svg?branch=master)](https://travis-ci.org/openaddresses/machine)

Usage
-----

Machine supports two modes. Continuous integration (CI) mode listens for tasks
from the OpenAddresses Github repository, includes a persistent web server and
database, and supports a flexible number of worker processes to deal with large
changes quickly. Batch mode processes the entire OpenAddresses source
collection at once, and is intended to be run periodically to reflect new
submissions.

### CI Mode

Installation scripts for preparing a fresh install of Ubuntu 14.04 can be found
in `chef`. You will need a pre-installed PostgreSQL database initialized with
the schema `openaddr/ci/schema.pgsql` and Amazon S3 bucket with credentials.
After editing `chef/role-webhooks.json` and `chef/role-worker.json`, run them
from a Git checkout like this:

    sudo apt-get update
    sudo chef/run.sh webhooks
    sudo chef/run.sh worker

`webhooks` will install Apache, a web application to listen for new tasks, and
a small queue observer to watch for completed tasks. `worker` will install the
`openaddr-ci-worker` utility to watch for newly-scheduled tasks.

Run a single source locally with `openaddr-process-one`:

    openaddr-process-one -l <log> <path to source JSON> <output directory>

### Batch Mode & CI Workers

To run batch mode with existing CI workers and queue, prepare a complete set of
sources from master branch with `openaddr-enqueue-sources`:

    openaddr-enqueue-sources -d <database URL> -t <Github token> -o <repo owner> -r <repo name>

Development
-----------

Test the OpenAddresses machine with `test.py`:

    python test.py

Run the webhook server, queue listener, and worker processes:

    python run-debug-webhooks.py
    python -m openaddr.ci.run_dequeue
    
    python -m openaddr.ci.worker

Extras
------

Convert remote ESRI feature services to GeoJSON with `openaddr-esri2geojson`:

    openaddr-esri2geojson <ESRI URL> <GeoJSON path>
{'fraction_non_alphanumeric': 0.08744038155802862, 'fraction_numerical': 0.0027821939586645467, 'mean_word_length': 4.188659793814433, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6357232', 'n_tokens_mistral': 1456, 'n_tokens_neox': 1392, 'n_words': 590}