<commit_before>Pydoop for Dumbo Users
======================

Pydoop is not the only way to write Hadoop applications in Python. The
"built-in" solutions are Hadoop Streaming with Python executables and
Jython. The main disadvantages with these approaches are the
following:

#. Streaming does not provide a true API: the developer writes a
   mapper and a reducer script that communicate with the framework via
   standard input/output. The programming style is therefore rather
   awkward, especially in the case of reducers, where developers must
   manually handle key switching. More importantly, there is no way to
   write a Python RecordReader, RecordWriter or Partitioner. Before
   Hadoop 0.21, Streaming had the additional limitation of only being
   able to process UTF-8 text records

#. Jython is a Java implementation of the Python language: the
   standard C implementation, in cases where ambiguity may arise, is
   referred to as "cPython". Although this approach gives access to
   the full programming API, it is limited by the fact that cPython
   modules (including part of the standard library) cannot be used.

As a consequence, few Python programmers use Streaming or Jython
directly. A popular alternative is offered by `Dumbo
<http://klbostee.github.com/dumbo>`_, a programming framework built as
a wrapper around Streaming that allows for a more natural API-like
coding style and also includes facilities to make job running
easier. Its author, `<NAME>
<http://users.ugent.be/~klbostee/>`_, is also the author of the patch
that lifted the aforementioned UTF-8 restriction from Streaming.

This chapter is meant both as a guide for Dumbo programmers wishing to
port their code to Pydoop and as a comparison between the two for
Python programmers in search of the solution that best fits their
MapReduce programming needs.

The following sections refer to Dumbo version 0.21.


Counting IPs from an Apache Access Log
--------------------------------------

The ``examples`` directory includes a Pydoop reimplementation of
`Dumbo's tutorial example
<http://wiki.github.com/klbostee/dumbo/short-tutorial>`_, an
application for generating a list of the IPs that occur more
frequently in an `Apache access log
<http://httpd.apache.org/docs/1.3/logs.html#common>`_. Complete
documentation for the tutorial is included in K. Bosteels, `Fuzzy
techniques in the usage and construction of comparison measures for
music objects <http://users.ugent.be/~klbostee/thesis.pdf>`_, PhD
thesis, Ghent University, 2009.

The Dumbo MapReduce code for the basic example is::

  def mapper(key,value):
    yield value.split(" ")[0], 1
    
  def reducer(key,values):
    yield key, sum(values)
    <commit_msg>pydoop: docs: stated more clearly pydoop's advantages with respect to dumbo<commit_after>Pydoop for Dumbo Users
======================

Pydoop is not the only way to write Hadoop applications in Python. The
"built-in" solutions are Hadoop Streaming with Python executables and
Jython. The main disadvantages with these approaches are the
following:

#. Streaming does not provide a true API: the developer writes a
   mapper and a reducer script that communicate with the framework via
   standard input/output. The programming style is therefore rather
   awkward, especially in the case of reducers, where developers must
   manually handle key switching. More importantly, there is no way to
   write a Python RecordReader, RecordWriter or Partitioner. Before
   Hadoop 0.21, Streaming had the additional limitation of only being
   able to process UTF-8 text records

#. Jython is a Java implementation of the Python language: the
   standard C implementation, in cases where ambiguity may arise, is
   referred to as "cPython". Although this approach gives access to
   the full programming API, it is limited by the fact that cPython
   modules (including part of the standard library) cannot be used.

As a consequence, few Python programmers use Streaming or Jython
directly. A popular alternative is offered by `Dumbo
<http://klbostee.github.com/dumbo>`_, a programming framework built as
a wrapper around Streaming that allows for a more natural API-like
coding style and also includes facilities to make job running
easier. Its author, `<NAME>els
<http://users.ugent.be/~klbostee/>`_, is also the author of the patch
that lifted the aforementioned UTF-8 restriction from
Streaming. However, writing Python components other than the mapper,
reducer and combiner is not possible, and there is no HDFS
API. Pydoop, on the other hand, gives you almost complete access to
MapReduce components (you can write a Python RecordReader,
RecordWriter and Partitioner) and to HDFS without adding much
complexity. As an example, in this section we will show how to rewrite
Dumbo's tutorial example (Dumbo version 0.21) in Pydoop.


Counting IPs from an Apache Access Log
--------------------------------------

The ``examples`` directory includes a Pydoop reimplementation of
`Dumbo's tutorial example
<http://wiki.github.com/klbostee/dumbo/short-tutorial>`_, an
application for generating a list of the IPs that occur more
frequently in an `Apache access log
<http://httpd.apache.org/docs/1.3/logs.html#common>`_. Complete
documentation for the tutorial is included in <NAME>, `Fuzzy
techniques in the usage and construction of comparison measures for
music objects <http://users.ugent.be/~klbostee/thesis.pdf>`_, PhD
thesis, Ghent University, 2009.

The Dumbo MapReduce code for the basic example is::

  def mapper(key,value):
    yield value.split(" ")[0], 1
    
  def reducer(key,values):
    yield key, sum(values)
    
{'fraction_non_alphanumeric': 0.07609081234480312, 'fraction_numerical': 0.005675771550195104, 'mean_word_length': 4.178145087235996, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2609310', 'n_tokens_mistral': 1547, 'n_tokens_neox': 1438, 'n_words': 758}