<commit_before># Elephant Bird #

#### Twitter's library of [LZO](http://www.github.com/kevinweil/hadoop-lzo) and/or [Protocol Buffer](http://code.google.com/p/protobuf)-related [Hadoop](http://hadoop.apache.org) InputFormats, OutputFormats, Writables, [Pig](http://hadoop.apache.org/pig) LoadFuncs, [HBase](http://hadoop.apache.org/hbase) miscellanea, etc. The majority of these are in production at Twitter running over data every day. ####

### To Use ###

1. git clone
2. ant
3. check out javadoc, etc.
4. build examples: cd examples && ant;

Note: for any of the LZO-based code, make sure that the native LZO libraries are on your `java.library.path`.  Generally this is done by setting `JAVA_LIBRARY_PATH` in
`pig-env.sh` or `hadoop-env.sh`.  You can also add lines like
<code><pre>
PIG_OPTS=-Djava.library.path=/path/to/my/libgplcompression/dir
</pre></code>
to `pig-env.sh`. See the instructions for [Hadoop-LZO](http://www.github.com/kevinweil/hadoop-lzo) for more details.

There are a few simple examples that use the input formats.  Note how the protocol buffer-based
formats work, and also note that the examples build file uses the custom codegen stuff.  See below for
more about that.

#### Building Without Protocol Buffers ####

If you don't want to build elephant-bird with protobuf support or you don't have protobuf >= 2.3 available 
(fedora for instance only provides 2.2), you can have ant exclude all classes that rely on protobuf by using ant noproto target<commit_msg>Added note about Hive SerDe in readme<commit_after># Elephant Bird #

#### Twitter's library of [LZO](http://www.github.com/kevinweil/hadoop-lzo) and/or [Protocol Buffer](http://code.google.com/p/protobuf)-related [Hadoop](http://hadoop.apache.org) InputFormats, OutputFormats, Writables, [Pig](http://hadoop.apache.org/pig) LoadFuncs, [Hive](http://hadoop.apache.org/hive) SerDe, [HBase](http://hadoop.apache.org/hbase) miscellanea, etc. The majority of these are in production at Twitter running over data every day. ####

### To Use ###

1. git clone
2. ant
3. check out javadoc, etc.
4. build examples: cd examples && ant;

Note: for any of the LZO-based code, make sure that the native LZO libraries are on your `java.library.path`.  Generally this is done by setting `JAVA_LIBRARY_PATH` in
`pig-env.sh` or `hadoop-env.sh`.  You can also add lines like
<code><pre>
PIG_OPTS=-Djava.library.path=/path/to/my/libgplcompression/dir
</pre></code>
to `pig-env.sh`. See the instructions for [Hadoop-LZO](http://www.github.com/kevinweil/hadoop-lzo) for more details.

There are a few simple examples that use the input formats.  Note how the protocol buffer-based
formats work, and also note that the examples build file uses the custom codegen stuff.  See below for
more about that.

#### Building Without Protocol Buffers ####

If you don't want to build elephant-bird with protobuf support or you don't have protobuf >= 2.3 available 
(fedora for instance only provides 2.2), you can have ant exclude all classes that rely on protobuf by using ant noproto target
{'fraction_non_alphanumeric': 0.11620294599018004, 'fraction_numerical': 0.005237315875613748, 'mean_word_length': 4.457142857142857, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7499404', 'n_tokens_mistral': 974, 'n_tokens_neox': 941, 'n_words': 365}