<commit_before>druid-dumbo, the druid batch config generator
=============================================

When you start to use [batch ingestion](https://github.com/metamx/druid/wiki/Batch-ingestion),
you'll quickly notice, you will need to edit the batch config for each run.

Also, it's unreliable and needs lots of fiddling.

dumbo.rb actually checks your hdfs against your s3 and computes what's needed.

The easiest way to use dumbo is via environment variables:

 * DRUID_DATASOURCE - set it to your druid datasource 
 * DRUID_S3_BUCKET - the s3 bucket to look into
 * DRUID_S3_PREFIX - the s3 prefix to observe
 * DRUID_S3_HOST - optional, set to 's3-eu-west-1.amazonaws.com' if you use an EU bucket (strongly recommended for EU people)
 * AMAZON_ACCESS_KEY_ID - your s3 keys
 * AMAZON_SECRET_ACCESS_KEY - your s3 secrets

Start by creating a `importer.template` based on `importer.template.example`.

Once you got that, try:

```
./dumbo.rb
CLASSPATH= hadoop_config_path:druid_indexer_selfcontained_jar_path com.metamx.druid.indexer.HadoopDruidIndexerMain
java -cp $CLASSPATH ./druidimport.conf
```

Caveats:
<commit_msg>Fixed java call<commit_after>druid-dumbo, the druid batch config generator
=============================================

When you start to use [batch ingestion](https://github.com/metamx/druid/wiki/Batch-ingestion),
you'll quickly notice, you will need to edit the batch config for each run.

Also, it's unreliable and needs lots of fiddling.

dumbo.rb actually checks your hdfs against your s3 and computes what's needed.

The easiest way to use dumbo is via environment variables:

 * DRUID_DATASOURCE - set it to your druid datasource 
 * DRUID_S3_BUCKET - the s3 bucket to look into
 * DRUID_S3_PREFIX - the s3 prefix to observe
 * DRUID_S3_HOST - optional, set to 's3-eu-west-1.amazonaws.com' if you use an EU bucket (strongly recommended for EU people)
 * AMAZON_ACCESS_KEY_ID - your s3 keys
 * AMAZON_SECRET_ACCESS_KEY - your s3 secrets

Start by creating a `importer.template` based on `importer.template.example`.

Once you got that, try:

```
./dumbo.rb
CLASSPATH= hadoop_config_path:druid_indexer_selfcontained_jar_path
java -cp $CLASSPATH com.metamx.druid.indexer.HadoopDruidIndexerMain ./druidimport.conf 
```

Caveats:

{'fraction_non_alphanumeric': 0.11214128035320088, 'fraction_numerical': 0.008830022075055188, 'mean_word_length': 4.81025641025641, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1476076', 'n_tokens_mistral': 766, 'n_tokens_neox': 739, 'n_words': 248}