<commit_before>h1. ActiveWarehouse-ETL

ActiveWarehouse-ETL is a Ruby Extract-Transform-Load (ETL) tool.

This tool is both usable and used in production under its current form - but be aware the project is under reorganization: a new team is shaping up and we're working mostly on making it easier for people to contribute first. Up-to-date documentation will only come later.

"!https://secure.travis-ci.org/activewarehouse/activewarehouse-etl.png!":http://travis-ci.org/activewarehouse/activewarehouse-etl

h2. Usage

The documentation is sparse and not everything is up to date, too, but here are useful bits to get you started:

* read the "Introduction":https://github.com/activewarehouse/activewarehouse-etl/wiki/Documentation
* later on, refer to the "RDoc":http://rdoc.info/github/activewarehouse/activewarehouse-etl/master/frames (be sure to check out Processor and Transform)
* read the "source":https://github.com/activewarehouse/activewarehouse-etl/tree/master/lib/etl

If you're lost, please ask questions on the "Google Group":http://groups.google.com/group/activewarehouse-discuss and we'll take care of it.

One thing to keep in mind is that ActiveWarehouse-ETL is highly hackable: you can pretty much create all you need with extra ruby code, even if it's not currently supported.

h2. Compatibility

See "Travis":http://travis-ci.org/activewarehouse/activewarehouse-etl and our set of "gemfiles":https://github.com/activewarehouse/activewarehouse-etl/tree/master/test/config/gemfiles for the currently tested combinations.

If you meet any error, please drop a line on the "Google Group":http://groups.google.com/group/activewarehouse-discuss so that we can fix it.

h2. Contributing

Fork on GitHub and after you've committed tested patches, send a pull request.

TODO: explain how to run the tests like on Travis.

* install RVM and Bundler
* install MySQL and/or Postgresql (you can use brew for that)
* edit test/config/database.yml if needed (don't commit this file though)
* run (for instance):

    @rake "ci:run_one[mysql2,test/config/gemfiles/Gemfile.rails-3.0.x]"@

or for all the combinations:
    
    @rake ci:run_matrix@

h2. Contributors

ActiveWarehouse-ETL is the work of many people since late 2006 - here is a list, in no particular order:

* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* sasikumargn
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* Sinisa

If your name should be on the list but isn't, please leave a comment!

h2. Features

Currently supported features:

* ETL Domain Specific Language (DSL) - Control files are specified in a Ruby-based DSL
* Multiple source types. Current supported types:
** Fixed-width and delimited text files
** XML files through SAX<commit_msg>Update contributors list before merging<commit_after>h1. ActiveWarehouse-ETL

ActiveWarehouse-ETL is a Ruby Extract-Transform-Load (ETL) tool.

This tool is both usable and used in production under its current form - but be aware the project is under reorganization: a new team is shaping up and we're working mostly on making it easier for people to contribute first. Up-to-date documentation will only come later.

"!https://secure.travis-ci.org/activewarehouse/activewarehouse-etl.png!":http://travis-ci.org/activewarehouse/activewarehouse-etl

h2. Usage

The documentation is sparse and not everything is up to date, too, but here are useful bits to get you started:

* read the "Introduction":https://github.com/activewarehouse/activewarehouse-etl/wiki/Documentation
* later on, refer to the "RDoc":http://rdoc.info/github/activewarehouse/activewarehouse-etl/master/frames (be sure to check out Processor and Transform)
* read the "source":https://github.com/activewarehouse/activewarehouse-etl/tree/master/lib/etl

If you're lost, please ask questions on the "Google Group":http://groups.google.com/group/activewarehouse-discuss and we'll take care of it.

One thing to keep in mind is that ActiveWarehouse-ETL is highly hackable: you can pretty much create all you need with extra ruby code, even if it's not currently supported.

h2. Compatibility

See "Travis":http://travis-ci.org/activewarehouse/activewarehouse-etl and our set of "gemfiles":https://github.com/activewarehouse/activewarehouse-etl/tree/master/test/config/gemfiles for the currently tested combinations.

If you meet any error, please drop a line on the "Google Group":http://groups.google.com/group/activewarehouse-discuss so that we can fix it.

h2. Contributing

Fork on GitHub and after you've committed tested patches, send a pull request.

TODO: explain how to run the tests like on Travis.

* install RVM and Bundler
* install MySQL and/or Postgresql (you can use brew for that)
* edit test/config/database.yml if needed (don't commit this file though)
* run (for instance):

    @rake "ci:run_one[mysql2,test/config/gemfiles/Gemfile.rails-3.0.x]"@

or for all the combinations:
    
    @rake ci:run_matrix@

h2. Contributors

ActiveWarehouse-ETL is the work of many people since late 2006 - here is a list, in no particular order:

* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* "<NAME>":https://github.com/thbar
* <NAME>
* sasikumargn
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* <NAME>
* "<NAME>":https://github.com/pdodds
* "<NAME>":https://github.com/sgrgic

If your name should be on the list but isn't, please leave a comment!

h2. Features

Currently supported features:

* ETL Domain Specific Language (DSL) - Control files are specified in a Ruby-based DSL
* Multiple source types. Current supported types:
** Fixed-width and delimited text files
** XML files through SAX
{'fraction_non_alphanumeric': 0.10043744531933509, 'fraction_numerical': 0.004549431321084864, 'mean_word_length': 4.490874159462056, 'pattern_counts': {'":': 19, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3895442', 'n_tokens_mistral': 1819, 'n_tokens_neox': 1756, 'n_words': 655}