<commit_before>## Summingbird [![Build Status](https://secure.travis-ci.org/twitter/summingbird.png)](http://travis-ci.org/twitter/summingbird)

Streaming MapReduce on [Storm](https://github.com/nathanmarz/storm) and [Scalding](https://github.com/twitter/scalding).

## Maven

Storehaus modules are available on maven central. The current groupid and version for all modules is, respectively, `"com.twitter"` and  `0.1.0-SNAPSHOT`.

Current published artifacts are

* `summingbird-core_2.9.2`
* `summingbird-core_2.10`
* `summingbird-batch_2.9.2`
* `summingbird-batch_2.10`
* `summingbird-client_2.9.2`
* `summingbird-client_2.10`
* `summingbird-kryo_2.9.2`
* `summingbird-kryo_2.10`
* `summingbird-storm_2.9.2`
* `summingbird-storm_2.10`
* `summingbird-scalding_2.9.2`
* `summingbird-scalding_2.10`
* `summingbird-builder_2.9.2`
* `summingbird-builder_2.10`

The suffix denotes the scala version.<commit_msg>readme udpate<commit_after>## Summingbird [![Build Status](https://secure.travis-ci.org/twitter/summingbird.png)](http://travis-ci.org/twitter/summingbird)

Summingbird is a library that lets you write MapReduce programs that look like native Scala or Java collection transformations and execute them on a number of well-known distributed MapReduce platforms, including [Storm](https://github.com/nathanmarz/storm) and [Scalding](https://github.com/twitter/scalding).

While a word-counting aggregation in pure Scala might look like this:

```scala
  def wordCount(source: Iterable[String], store: MutableMap[String, Long]) =
    source.flatMap { sentence =>
      toWords(sentence).map(_ -> 1L)
    }.foreach { case (k, v) => store.update(k, store.get(k) + v) }
```

Counting words in Summingbird looks like this:

```scala
  def wordCount[P <: Platform[P]]
    (source: Producer[P, String], store: P#Store[String, Long]) =
      source.flatMap { sentence =>
        toWords(sentence).map(_ -> 1L)
      }.sumByKey(store)
```

The logic is exactly the same, and the code is almost the same. The main difference is that you can execute the Summingbird program in "batch mode" (using [Scalding](https://github.com/twitter/scalding)), in "realtime mode" (using [Storm](https://github.com/nathanmarz/storm)), or on both Scalding and Storm in a hybrid batch/realtime mode that offers your application very attractive fault-tolerance properties.

Summingbird provides you with the primitives you need to build rock solid production systems.

## Community and Documentation

To learn more and find links to tutorials and information around the web, check out the [Summingbird Wiki](https://github.com/twitter/summingbird/wiki).

The latest ScalaDocs are hosted on Summingbird's [Github Project Page](http://twitter.github.io/summingbird).

Discussion occurs primarily on the Summingbird mailing list. To join the mailing list, email [<EMAIL>](<EMAIL>). The same address is used for posting once you've joined. Issues should be reported on the GitHub issue tracker. Simpler issues appropriate for first-time contributors looking to help out are tagged "newbie".

Follow [@summingbird](https://twitter.com/summingbird) on Twitter for updates.

## Maven

Summingbird modules are published on maven central. The current groupid and version for all modules is, respectively, `"com.twitter"` and  `0.1.0-SNAPSHOT`.

Current published artifacts are

* `summingbird-core_2.9.3`
* `summingbird-core_2.10`
* `summingbird-batch_2.9.3`
* `summingbird-batch_2.10`
* `summingbird-client_2.9.3`
* `summingbird-client_2.10`
* `summingbird-kryo_2.9.3`
* `summingbird-kryo_2.10`
* `summingbird-storm_2.9.3`
* `summingbird-storm_2.10`
* `summingbird-scalding_2.9.3`
* `summingbird-scalding_2.10`
* `summingbird-builder_2.9.3`
* `summingbird-builder_2.10`

The suffix denotes the scala version.
{'fraction_non_alphanumeric': 0.12496690495101932, 'fraction_numerical': 0.024357956049774955, 'mean_word_length': 4.921630094043887, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 10, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3762826', 'n_tokens_mistral': 1299, 'n_tokens_neox': 1247, 'n_words': 340}