<commit_before># data_miner

Download, pull out of a ZIP/TAR/GZ/BZ2 archive, parse, correct, and import XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models.

Tested in MRI 1.8.7+, MRI 1.9.2+, and JRuby 1.6.7+. Thread safe.

## Real-world usage

<p><a href="http://brighterplanet.com"><img src="https://s3.amazonaws.com/static.brighterplanet.com/assets/logos/flush-left/inline/green/rasterized/brighter_planet-160-transparent.png" alt="Brighter Planet logo"/></a></p>

We use `data_miner` for [data science at Brighter Planet](http://brighterplanet.com/research) and in production at

* [Brighter Planet's reference data web service](http://data.brighterplanet.com)
* [Brighter Planet's impact estimate web service](http://impact.brighterplanet.com)

The killer combination for us is:

1. [`active_record_inline_schema`](https://github.com/seamusabshere/active_record_inline_schema) - define table structure
2. [`remote_table`](https://github.com/seamusabshere/remote_table) - download data and parse it
3. [`errata`](https://github.com/seamusabshere/errata) - apply corrections in a transparent way
4. [`data_miner`](https://github.com/seamusabshere/remote_table) (this library!) - import data idempotently

## Documentation

Check out the [extensive documentation](http://rdoc.info/github/seamusabshere/data_miner).

## Quick start

You define <code>data_miner</code> blocks in your ActiveRecord models. For example, in <code>app/models/country.rb</code>:

    class Country < ActiveRecord::Base
      self.primary_key = 'iso_3166_code'

      # the "col" class method is provided by a different library - active_record_inline_schema
      col :iso_3166_code                            # alpha-2 2-letter like GB
      col :iso_3166_numeric_code, :type => :integer # numeric like 826; aka UN M49 code
      col :iso_3166_alpha_3_code                    # 3-letter like GBR
      col :name
  
      data_miner do
        # auto_upgrade! is provided by active_record_inline_schema
        process :auto_upgrade!

        import("OpenGeoCode.org's Country Codes to Country Names list",
               :url => 'http://opengeocode.org/download/countrynames.txt',
               :format => :delimited,<commit_msg>Fix data_miner link in README<commit_after># data_miner

Download, pull out of a ZIP/TAR/GZ/BZ2 archive, parse, correct, and import XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models.

Tested in MRI 1.8.7+, MRI 1.9.2+, and JRuby 1.6.7+. Thread safe.

## Real-world usage

<p><a href="http://brighterplanet.com"><img src="https://s3.amazonaws.com/static.brighterplanet.com/assets/logos/flush-left/inline/green/rasterized/brighter_planet-160-transparent.png" alt="Brighter Planet logo"/></a></p>

We use `data_miner` for [data science at Brighter Planet](http://brighterplanet.com/research) and in production at

* [Brighter Planet's reference data web service](http://data.brighterplanet.com)
* [Brighter Planet's impact estimate web service](http://impact.brighterplanet.com)

The killer combination for us is:

1. [`active_record_inline_schema`](https://github.com/seamusabshere/active_record_inline_schema) - define table structure
2. [`remote_table`](https://github.com/seamusabshere/remote_table) - download data and parse it
3. [`errata`](https://github.com/seamusabshere/errata) - apply corrections in a transparent way
4. [`data_miner`](https://github.com/seamusabshere/data_miner) (this library!) - import data idempotently

## Documentation

Check out the [extensive documentation](http://rdoc.info/github/seamusabshere/data_miner).

## Quick start

You define <code>data_miner</code> blocks in your ActiveRecord models. For example, in <code>app/models/country.rb</code>:

    class Country < ActiveRecord::Base
      self.primary_key = 'iso_3166_code'

      # the "col" class method is provided by a different library - active_record_inline_schema
      col :iso_3166_code                            # alpha-2 2-letter like GB
      col :iso_3166_numeric_code, :type => :integer # numeric like 826; aka UN M49 code
      col :iso_3166_alpha_3_code                    # 3-letter like GBR
      col :name
  
      data_miner do
        # auto_upgrade! is provided by active_record_inline_schema
        process :auto_upgrade!

        import("OpenGeoCode.org's Country Codes to Country Names list",
               :url => 'http://opengeocode.org/download/countrynames.txt',
               :format => :delimited,
{'fraction_non_alphanumeric': 0.1164568345323741, 'fraction_numerical': 0.01933453237410072, 'mean_word_length': 3.7481323372465316, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 27, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6293761', 'n_tokens_mistral': 1498, 'n_tokens_neox': 1385, 'n_words': 363}