<commit_before># Humboldt

Humboldt provides a tool-set on top of Rubydoop to run Hadoop jobs
effortlessly both locally and on Amazon EMR. There is also some sugar
added on top of the Rubydoop DSL.

## Sugar

### Type converters

Humboldt adds a number of type converters:

* binary - `String` - `Hadoop::Io::BytesWritable`
* encoded - `String` - MessagePack encoded `Hadoop::Io::BytesWritable`
* text - `String` - `Hadoop::Io::Text`
* json - `Hash` - `Hadoop::Io::Text`
* long - `Integer` - `Hadoop::Io::LongWritable`
* none - `nil` - `Hadoop::Io::NullWritable`

Use them like so:

```ruby
class Mapper < Humboldt::Mapper
  input :long, :json
  # ...
end
```

### Combine input files

Hadoop does not perform well with many small input files, since each
file is handled by its own map task, by default. Humboldt bundles an
input format which combines files. Due to a bug in Hadoop 1.0.3 (and
other versions), Hadoop 2.2.0 is required to run this for input files
on S3, see
[this bug](https://issues.apache.org/jira/browse/MAPREDUCE-1806).

Example usage:

```ruby
Rubydoop.configure do |input_paths, output_path|
  job 'my job' do
    input input_paths, format: :combined_text
    set 'mapreduce.input.fileinputformat.split.maxsize', 32 * 1024 * 1024

    # ...
  end
end
```

`mapreduce.input.fileinputformat.split.maxsize` controls the maximum
size of an input split.

## Development setup

Download Hadoop and set up the classpath using

    $ rake setup

The default is Hadoop 1.0.3. Specify a hadoop release by setting
`$HADOOP_RELEASE`, e.g.

    $ HADOOP_RELEASE=hadoop-2.2.0/hadoop-2.2.0 rake setup

Run the tests with

    $ rake spec

## Release a new gem

Bump the version number in `lib/humboldt/version.rb`, run `rake gem:release`.

# Copyright

© 2014 Burt AB, see LICENSE.txt (BSD 3-Clause).<commit_msg>Add a note in the readme about secondary sort<commit_after># Humboldt

Humboldt provides a tool-set on top of Rubydoop to run Hadoop jobs
effortlessly both locally and on Amazon EMR. There is also some sugar
added on top of the Rubydoop DSL.

## Sugar

### Type converters

Humboldt adds a number of type converters:

* binary - `String` - `Hadoop::Io::BytesWritable`
* encoded - `String` - MessagePack encoded `Hadoop::Io::BytesWritable`
* text - `String` - `Hadoop::Io::Text`
* json - `Hash` - `Hadoop::Io::Text`
* long - `Integer` - `Hadoop::Io::LongWritable`
* none - `nil` - `Hadoop::Io::NullWritable`

Use them like so:

```ruby
class Mapper < Humboldt::Mapper
  input :long, :json
  # ...
end
```

### Combine input files

Hadoop does not perform well with many small input files, since each
file is handled by its own map task, by default. Humboldt bundles an
input format which combines files. Due to a bug in Hadoop 1.0.3 (and
other versions), Hadoop 2.2.0 is required to run this for input files
on S3, see
[this bug](https://issues.apache.org/jira/browse/MAPREDUCE-1806).

Example usage:

```ruby
Rubydoop.configure do |input_paths, output_path|
  job 'my job' do
    input input_paths, format: :combined_text
    set 'mapreduce.input.fileinputformat.split.maxsize', 32 * 1024 * 1024

    # ...
  end
end
```

`mapreduce.input.fileinputformat.split.maxsize` controls the maximum
size of an input split.

### Secondary sort

Example usage:

A common mapreduce pattern when you need to count uniques is secondary sort, which can be quite a pain to implement. Humboldt makes it really easy, all you need to do say which indexes to partition and group by:

```ruby
Rubydoop.configure do |input_paths, output_path|
  job 'my job' do
    # ...

    secondary_sort 0, 10

    # ...
  end
end
```

See the API documentation for `Rubydoop::JobDefinition#secondary_sort` for more information on how use it.

## Development setup

Download Hadoop and set up the classpath using

    $ rake setup

The default is Hadoop 1.0.3. Specify a hadoop release by setting
`$HADOOP_RELEASE`, e.g.

    $ HADOOP_RELEASE=hadoop-2.2.0/hadoop-2.2.0 rake setup

Run the tests with

    $ rake spec

## Release a new gem

Bump the version number in `lib/humboldt/version.rb`, run `rake gem:release`.

# Copyright

© 2014 Burt AB, see LICENSE.txt (BSD 3-Clause).
{'fraction_non_alphanumeric': 0.10590778097982709, 'fraction_numerical': 0.017531219980787705, 'mean_word_length': 3.7928653624856157, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2664247', 'n_tokens_mistral': 1484, 'n_tokens_neox': 1365, 'n_words': 517}