<commit_before>![](logo.png)

PigPen is a map-reduce abstraction for Clojure. It is written on top of [Apache Pig](http://pig.apache.org/), but you don't need to know much about Pig to use it.

Goals of PigPen:

  * A map-reduce query abstraction that's as similar as possible to clojure.core. The idea is to feel like you're working with local Clojure data when running a map-reduce job.
  * Allow for iterative development of scripts using standard unit testing
  * Map-reduce using a dynamic language (no types, but it actually works)

PigPen is __not__ a Clojure wrapper for writing Pig scripts - the resulting scripts are not intended for human consumption.


# Really, yet another map-reduce language?

### If you know Clojure, you already know pigpen

The primary goal of PigPen is to take language out of the equation. PigPen operators are designed to be as close as possible to the Clojure equivalents. And there are no sidecar files for UDFs - they're right inline where they should be.

Here's the proverbial word count:

    (require '[pigpen.core :as pig])

    (defn word-count [lines]
      (->> lines
        (pig/mapcat #(-> % first
                       (clojure.string/lower-case)
                       (clojure.string/replace #"[^a-zA-Z0-9\s]" "")
                       (clojure.string/split #"\s+")))
        (pig/group-by identity)
        (pig/map (fn [[word occurrences]] [word (count occurrences)]))))

As you can see, this is just the word count logic. We don't have to conflate external concerns, like where our data is coming from or going.


# Will it compose?

Yep - PigPen queries are written as function compositions - data in, data out. Write it once and avoid the copy & paste routine.

Here we use word-count to make a PigPen query:

    (defn word-count-query [input output]
      (->>
        (pig/load-tsv input)
        (word-count)
        (pig/store-tsv output)))
<commit_msg>Fix readme typos<commit_after>![](logo.png)

PigPen is a map-reduce abstraction for Clojure. It is written on top of [Apache Pig](http://pig.apache.org/), but you don't need to know much about Pig to use it.

Goals of PigPen:

  * A map-reduce query abstraction that's as similar as possible to clojure.core. The idea is to feel like you're working with local Clojure data when running a map-reduce job.
  * Allow for iterative development of scripts using standard unit testing
  * Map-reduce using a dynamic language (no types, but it actually works)

PigPen is __not__ a Clojure wrapper for writing Pig scripts - the resulting scripts are not intended for human consumption.


# Really, yet another map-reduce language?

### If you know Clojure, you already know PigPen

The primary goal of PigPen is to take language out of the equation. PigPen operators are designed to be as close as possible to the Clojure equivalents. And there are no sidecar files for UDFs - they're right inline where they should be.

Here's the proverbial word count:

    (require '[pigpen.core :as pig])

    (defn word-count [lines]
      (->> lines
        (pig/mapcat #(-> % first
                       (clojure.string/lower-case)
                       (clojure.string/replace #"[^a-zA-Z0-9\s]" "")
                       (clojure.string/split #"\s+")))
        (pig/group-by identity)
        (pig/map (fn [[word occurrences]] [word (count occurrences)]))))

As you can see, this is just the word count logic. We don't have to conflate external concerns, like where our data is coming from or going.


# Will it compose?

Yep - PigPen queries are written as function compositions - data in, data out. Write it once and avoid the copy & paste routine.

Here we use word-count to make a PigPen query:

    (defn word-count-query [input output]
      (->>
        (pig/load-tsv input)
        (word-count)
        (pig/store-tsv output)))

{'fraction_non_alphanumeric': 0.08639084048920115, 'fraction_numerical': 0.0010408534998698933, 'mean_word_length': 2.9184505606523956, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6828109', 'n_tokens_mistral': 1178, 'n_tokens_neox': 1078, 'n_words': 506}