<commit_before>Resque Data Warehouse
=====================

A [Resque][rq] plugin. Requires Resque 1.9.10.

resque-data-warehouse allows you to use Redis to queue up and then Resque to process transactions 
on transaction-heavy tables that need to be replicated on other tables optimized for 
reporting. 

Transactions for a given object (classname + ID) are queued up behind a Redis key, 
and then processed using Resque jobs. If load is low, each transaction will be processed
almost immediately after it occurs; at higher loads, multiple transactions will queue up
before the Resque job gets to them, and then only the cumulative result of the previous transactions 
will be applied to the data warehousing table, thus minimizing database load and dynamically adjusting 
the delay in the copy to match the current load.

This only works with Rails; it has only been tested with Rails 2.3.4 in which case the after_commit
gem is also required.

Usage / Examples
----------------
<commit_msg>Update README for Rails 3.0<commit_after>Resque Data Warehouse
=====================

A [Resque][rq] plugin. Requires Resque 1.9.10 or higher.

resque-data-warehouse allows you to use Redis to queue up and then Resque to process transactions 
on transaction-heavy tables that need to be replicated on other tables optimized for 
reporting. 

Transactions for a given object (classname + ID) are queued up behind a Redis key, 
and then processed using Resque jobs. If load is low, each transaction will be processed
almost immediately after it occurs; at higher loads, multiple transactions will queue up
before the Resque job gets to them, and then only the cumulative result of the previous transactions 
will be applied to the data warehousing table, thus minimizing database load and dynamically adjusting 
the delay in the copy to match the current load.

This only works with Rails >=3.0; it has been tested with Rails 3.0.5.

Usage / Examples
----------------

{'fraction_non_alphanumeric': 0.07146503318019398, 'fraction_numerical': 0.009188361408882083, 'mean_word_length': 4.130890052356021, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2667627', 'n_tokens_mistral': 511, 'n_tokens_neox': 472, 'n_words': 289}