<commit_before>.. _Tut3_BasicImp:

***********
Tutorial #3
***********

.. _writing-your-own:

Writing Your Own Mrs Implimentation:
====================================

The wordcount implementation that we have been working with throught the
previous three tutorials shows a Mrs program in it's most simple form. In this
tutorial, we will try to explain the Basic format for a Mrs MapReduce program
and some of the options for a more complex implimentation.

The Basic format for a Mrs MapReduce program looks something like this ::

    class Mrs-Program(mrs.MapReduce):
    
        def map(key, value):
            yield newkey, newvalue
      
        def reduce(key, values):
            yield newvalue

    if __name__ == '__main__':
        import mrs
        mrs.main(Mrs-Program)

Here we create a class that extends the mapreduce.py file in MRS_HOME/mrs, and
overrides only the required map and reduce functions. The last section of code
(the "if __name__ == '__main__':" section) just passes the program to Mrs
to run.

For example, in the map function in the wordcount program, we take in a chunk of
text as the starting value, break it up into words and yield each word as a new
key with a value of one (found one instance of that word). Then the reduce
function sums all values associated with a particular key (word), and yields
the result.

But what if, instead to reading in just one textfile and counting the words as
we've been doing, you had thousands of files that you wanted to count? This is a
little more complicated, so you would probably want to override the input_data()
function from mapreduce.py, by adding something like the following to your the
wordcount program. (See wordcount2.py in the examples folder.) ::

    def input_data(self, job):
        if len(self.args) < 2:
            print >>sys.stderr, "Requires input(s) and an output."
            return None
        inputs = []
        f = open(self.args[0])
        for line in f:
            inputs.append(line[:-1])
        return job.file_data(inputs)

Now, you can just pass in a single file containing the path to each text file on
a line of the input file, and it will read them all in. (See input_files in the
example folder.)

In the examples folder we have included a file called mapreduce-parse.py as an
example of a much more complex implimentation where there was a need to extend
several of the functions.



Defining More Complex Map Functions:
------------------------------------

Let's look at how to create map functions in more detail. The simple WordCount
mapper is: ::

    def mapper(key, value):
        for word in value.split():<commit_msg>s/implementation/program/ in the docs<commit_after>.. _Tut3_BasicImp:

***********
Tutorial #3
***********

.. _writing-your-own:

Writing Your Own Mrs Program:
====================================

The wordcount program that we have been working with throught the
previous three tutorials shows a Mrs program in it's most simple form. In this
tutorial, we will try to explain the Basic format for a Mrs MapReduce program
and some of the options for a more complex program.

The Basic format for a Mrs MapReduce program looks something like this ::

    class Mrs-Program(mrs.MapReduce):

        def map(key, value):
            yield newkey, newvalue

        def reduce(key, values):
            yield newvalue

    if __name__ == '__main__':
        import mrs
        mrs.main(Mrs-Program)

Here we create a class that extends the mapreduce.py file in MRS_HOME/mrs, and
overrides only the required map and reduce functions. The last section of code
(the "if __name__ == '__main__':" section) just passes the program to Mrs
to run.

For example, in the map function in the wordcount program, we take in a chunk of
text as the starting value, break it up into words and yield each word as a new
key with a value of one (found one instance of that word). Then the reduce
function sums all values associated with a particular key (word), and yields
the result.

But what if, instead to reading in just one textfile and counting the words as
we've been doing, you had thousands of files that you wanted to count? This is a
little more complicated, so you would probably want to override the input_data()
function from mapreduce.py, by adding something like the following to your the
wordcount program. (See wordcount2.py in the examples folder.) ::

    def input_data(self, job):
        if len(self.args) < 2:
            print >>sys.stderr, "Requires input(s) and an output."
            return None
        inputs = []
        f = open(self.args[0])
        for line in f:
            inputs.append(line[:-1])
        return job.file_data(inputs)

Now, you can just pass in a single file containing the path to each text file on
a line of the input file, and it will read them all in. (See input_files in the
example folder.)

In the examples folder we have included a file called mapreduce-parse.py as an
example of a much more complex program where there was a need to extend
several of the functions.



Defining More Complex Map Functions:
------------------------------------

Let's look at how to create map functions in more detail. The simple WordCount
mapper is: ::

    def mapper(key, value):
        for word in value.split():
{'fraction_non_alphanumeric': 0.09003397508493771, 'fraction_numerical': 0.0022650056625141564, 'mean_word_length': 3.1173271173271173, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1642571', 'n_tokens_mistral': 1414, 'n_tokens_neox': 1369, 'n_words': 750}