<commit_before>
Shep - Integration of Splunk and Hadoop
=======================================

Shep is a collection of integration modules to enable
seamless data flow and processing between Splunk, the
NoSQL platform for machine data, and Hadoop.

If you are not in the beta program, please register here:

* http://www.splunk.com/goto/hadoop-beta

For documentation, see:

* http://docs.splunk.com/Documentation/Hadoop

Key features as of v0.4.3 are:

* Streaming of data via Splunk Forwarding to HDFS
* SplunkInputFormat and SplunkOutputFormat classes for use in Hadoop
* Automatic batch rolling of indexed data to HDFS
* Developer friendly building and testing shep

Note: earlier versions of the forwarding mechanism relied on
Flume. That dependency is now removed, and Shep will write
direct to HDFS without a Flume intermediary.

Prerequisites
-------------

### Hadoop

Currently the Hadoop version used is 0.20.203.0

You can download it from one of the mirror sites listed [here][hadoop-download].
And see the [Hadoop documentation][] for instructions on installing and more.

[hadoop-download]:http://www.apache.org/dyn/closer.cgi?path=hadoop/core/hadoop-0.20.203.0/hadoop-0.20.203.0rc1.tar.gz
[Hadoop documentation]:http://hadoop.apache.org/common/docs/r0.20.203.0/

### Splunk

Currently the Splunk version used is 4.2.4

You can download it from [here][splunk-download]
And see the [Splunk documentation][] for instructions on installing and more.

[Splunk documentation]:http://docs.splunk.com/Documentation/Splunk/latest/User
[splunk-download]:http://www.splunk.com/download


Building from Source
--------------------

### Requirements

All you need is:

* Java JDK 6

The build should work on both MacOSX and Linux.

Environment setup:

	% source setjavaenv

Building Shep:

	% ./buildit.sh

or

	% ant

If everything goes well, you should see the message

	BUILD SUCCESSFUL

### Eclipse Users

You'll need to build once, before you can use Eclipse

### Coding Conventions

The Shep code base is to follow the standard java conventions:
http://java.sun.com/docs/codeconv/html/CodeConvTOC.doc.html

Getting Started
---------------

Set the `JAVA_HOME` environment variable

Make sure that you can run `ssh localhost` without having to enter a password*

Cd into to the splunk-shep directory and run the following:

Put a .tgz packaged Splunk in the put-splunk-tgz-here directory**<commit_msg>add description for .eclipse.templates directory<commit_after>
Shep - Integration of Splunk and Hadoop
=======================================

Shep is a collection of integration modules to enable
seamless data flow and processing between Splunk, the
NoSQL platform for machine data, and Hadoop.

If you are not in the beta program, please register here:

* http://www.splunk.com/goto/hadoop-beta

For documentation, see:

* http://docs.splunk.com/Documentation/Hadoop

Key features as of v0.4.3 are:

* Streaming of data via Splunk Forwarding to HDFS
* SplunkInputFormat and SplunkOutputFormat classes for use in Hadoop
* Automatic batch rolling of indexed data to HDFS
* Developer friendly building and testing shep

Note: earlier versions of the forwarding mechanism relied on
Flume. That dependency is now removed, and Shep will write
direct to HDFS without a Flume intermediary.

Prerequisites
-------------

### Hadoop

Currently the Hadoop version used is 0.20.203.0

You can download it from one of the mirror sites listed [here][hadoop-download].
And see the [Hadoop documentation][] for instructions on installing and more.

[hadoop-download]:http://www.apache.org/dyn/closer.cgi?path=hadoop/core/hadoop-0.20.203.0/hadoop-0.20.203.0rc1.tar.gz
[Hadoop documentation]:http://hadoop.apache.org/common/docs/r0.20.203.0/

### Splunk

Currently the Splunk version used is 4.2.4

You can download it from [here][splunk-download]
And see the [Splunk documentation][] for instructions on installing and more.

[Splunk documentation]:http://docs.splunk.com/Documentation/Splunk/latest/User
[splunk-download]:http://www.splunk.com/download


Building from Source
--------------------

### Requirements

All you need is:

* Java JDK 6

The build should work on both MacOSX and Linux.

Environment setup:

	% source setjavaenv

Building Shep:

	% ./buildit.sh

or

	% ant

If everything goes well, you should see the message

	BUILD SUCCESSFUL

### Eclipse Users

You'll need to build once, before you can use Eclipse
This .eclipse.templates directory contains templates for generating Eclipse files to configure
Eclipse for shep development.


### Coding Conventions

The Shep code base is to follow the standard java conventions:
http://java.sun.com/docs/codeconv/html/CodeConvTOC.doc.html

Getting Started
---------------

Set the `JAVA_HOME` environment variable

Make sure that you can run `ssh localhost` without having to enter a password*

Cd into to the splunk-shep directory and run the following:

Put a .tgz packaged Splunk in the put-splunk-tgz-here directory**
{'fraction_non_alphanumeric': 0.10937186433875176, 'fraction_numerical': 0.014449127031908489, 'mean_word_length': 4.024193548387097, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 8, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '584186', 'n_tokens_mistral': 1563, 'n_tokens_neox': 1412, 'n_words': 574}