<commit_before>hRaven [![Build Status](https://travis-ci.org/twitter/hraven.png)](https://travis-ci.org/twitter/hraven) [![Coverage Status](https://coveralls.io/repos/twitter/hraven/badge.png?branch=master)](https://coveralls.io/r/twitter/hraven?branch=master)
==========

hRaven collects run time data and statistics from map reduce jobs running on
Hadoop clusters and stores the collected job history in an easily queryable
format. For the jobs that are run through frameworks (Pig or
Scalding/Cascading) that decompose a script or application into a DAG of map
reduce jobs for actual execution, hRaven groups job history data together by
an application construct.  This allows for easier visualization of all of the
component jobs' execution for an application and more comprehensive trending
and analysis over time.

Requirements
--------------------

* Apache HBase (1.1.3) - a running HBase cluster is required for the hRaven
  data storage
* Apache Hadoop - hRaven current supports collection of job data on specific
  versions of Hadoop:
  * Hadoop 2.6+ post hRaven 1.0.0 Hadoop 1 will no longer be supported

Quick start
--------------------

Clone the github repo or download the latest release:

    git clone git://github.com/twitter/hraven.git

If you cloned the repository, build the full tarball:

    mvn clean package assembly:single

Extract the assembly tarball on a machine with HBase client access.

Create the initial schema

    hbase [--config /path/to/hbase/conf] shell bin/create_schema.rb


Schema
--------------------

hRaven requires the following HBase tables in order to store data for map
reduce jobs:

* `job_history` - job-level statistics, one row per job
* `job_history_task` - task-level statistics, one row per task attempt<commit_msg>Bumping hRaven version to 1.0.0-SNAPSHOT<commit_after>hRaven [![Build Status](https://travis-ci.org/twitter/hraven.png)](https://travis-ci.org/twitter/hraven) [![Coverage Status](https://coveralls.io/repos/twitter/hraven/badge.png?branch=master)](https://coveralls.io/r/twitter/hraven?branch=master)
==========

hRaven collects run time data and statistics from map reduce jobs running on
Hadoop clusters and stores the collected job history in an easily queryable
format. For the jobs that are run through frameworks (Pig or
Scalding/Cascading) that decompose a script or application into a DAG of map
reduce jobs for actual execution, hRaven groups job history data together by
an application construct.  This allows for easier visualization of all of the
component jobs' execution for an application and more comprehensive trending
and analysis over time.

Requirements
--------------------

* Apache HBase (1.1.3) - a running HBase cluster is required for the hRaven
  data storage
* Apache Hadoop - hRaven current supports collection of job data on specific
  versions of Hadoop:
  * Hadoop 2.6+ post hRaven 1.0.0 Hadoop 1 will no longer be supported
* JRE 8 - As of hRaven 1.0.0 Java 8 jars are generated. Use hRaven 0.9.x for Java 7 runtimes.

Quick start
--------------------

Clone the github repo or download the latest release:

    git clone git://github.com/twitter/hraven.git

If you cloned the repository, build the full tarball:

    mvn clean package assembly:single

Extract the assembly tarball on a machine with HBase client access.

Create the initial schema

    hbase [--config /path/to/hbase/conf] shell bin/create_schema.rb


Schema
--------------------

hRaven requires the following HBase tables in order to store data for map
reduce jobs:

* `job_history` - job-level statistics, one row per job
* `job_history_task` - task-level statistics, one row per task attempt
{'fraction_non_alphanumeric': 0.1028088355604036, 'fraction_numerical': 0.007908371966184892, 'mean_word_length': 3.6965428937259923, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6129580', 'n_tokens_mistral': 1044, 'n_tokens_neox': 1008, 'n_words': 441}