<commit_before>distiller
=========
This is the repository for Open Addresses' "Distiller" software component, part of the solution we deployed for the Alpha stage of our services. 

This component extracts addresses from the [Ernest](https://github.com/OpenAddressesUK/ernest) master database, checks for duplicates, and matches address parts (towns, streets, localities and postcodes) to exact entities.

Read about our production setup [here](http://openaddressesuk.org/docs) or learn about Open Addresses in general [here](http://openaddressesuk.org).

## Dependencies

As well as Ruby version 2.1.3, you'll also need MongoDB installed. [See installation instructions](http://docs.mongodb.org/manual/installation/).

## Running

### Clone the repo:

    git clone <EMAIL>@github.<EMAIL>:OpenAddressesUK/theodolite.git

### Bundle:

    bundle install

### Add a file called `.env` with the following contents:

    ERNEST_ADDRESS_ENDPOINT={URL for the list of addresses from the Ernest master database}
  
If you're hacking away on this, you can just use the master OpenAddresses master database at http://ernest.openaddressesuk.org/addresses
    
If you're running on production, you'll need to add the following too:

    MONGOLAB_URI: {The URI to access your production MongoDB database}
    HEROKU_TOKEN: {your Heroku API token - assuming you're running on Heroku}
    HEROKU_APP: {yout Herkou app name - assuming you're running on Heroku}
    
### Import the reference tables

    rake distiller:import:all
    
This imports all the towns, streets, localities and postcodes needed for the distil to run
<commit_msg>fixed repo reference<commit_after>distiller
=========
This is the repository for Open Addresses' "Distiller" software component, part of the solution we deployed for the Alpha stage of our services. 

This component extracts addresses from the [Ernest](https://github.com/OpenAddressesUK/ernest) master database, checks for duplicates, and matches address parts (towns, streets, localities and postcodes) to exact entities.

Read about our production setup [here](http://openaddressesuk.org/docs) or learn about Open Addresses in general [here](http://openaddressesuk.org).

## Dependencies

As well as Ruby version 2.1.3, you'll also need MongoDB installed. [See installation instructions](http://docs.mongodb.org/manual/installation/).

## Running

### Clone this repo:

    git clone git@github.com:OpenAddressesUK/distiller.git

### Bundle:

    bundle install

### Add a file called `.env` with the following contents:

    ERNEST_ADDRESS_ENDPOINT={URL for the list of addresses from the Ernest master database}
  
If you're hacking away on this, you can just use the master OpenAddresses master database at http://ernest.openaddressesuk.org/addresses
    
If you're running on production, you'll need to add the following too:

    MONGOLAB_URI: {The URI to access your production MongoDB database}
    HEROKU_TOKEN: {your Heroku API token - assuming you're running on Heroku}
    HEROKU_APP: {yout Herkou app name - assuming you're running on Heroku}
    
### Import the reference tables

    rake distiller:import:all
    
This imports all the towns, streets, localities and postcodes needed for the distil to run

{'fraction_non_alphanumeric': 0.08073959938366718, 'fraction_numerical': 0.0018489984591679508, 'mean_word_length': 4.529812606473595, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2383419', 'n_tokens_mistral': 916, 'n_tokens_neox': 859, 'n_words': 374}