<commit_before>= A !MapReduce service =
The original !MapReduce framework was designed and developed by Google to support large-scale distributed computing on large amounts of closely-networked computers.  The main drive for the !MapReduce framework stem from the intrinsic benefits of functional programming, more specifically the map() and reduce() functions.

There are 3 main steps in a !MapReduce run.

 1. Map step: data received by the worker is tossed into a map() function that will emit key-value pairs based on each aliquot of data.

 2. Partition (or shuffle) step: The generated key-value pairs from map() are hashed by its key to a particular worked and sent off.  If a key ends up on a particular work at the end of the partition phase, it is required that all other key-value pairs with the same key also be on the same host in order for the reduce step to work properly.

 3. Reduce step: The collected key-value pairs are collected into a key, value_1, value_2, ..., value_n list and can do computation to reduce the expanded data down to a useful conclusion.

This computation can be performed on an arbitrary number of worker machines, making it scalable to nearly all sizes of computing clusters!  !MapReduce frameworks often have a primary node delegating tasks to workers, organizing and distributing the initial data, and acting as a mediator in fault situations.

More detail on the use of !MapReduce is available from [http://labs.google.com/papers/mapreduce.html the original Google paper] or an open-source implementation of !MapReduce in Java: [http://hadoop.apache.org/core/ Hadoop].
[[BR]]
[[BR]]
-----
'''These assignments are a work-in-progress, please e-mail our developer list for more information! '''
[[TOC(inline)]]
[[BR]]
[[BR]]
== Stage 1: A simple implementation of !MapReduce ==

To start out, students will make the base framework of a !MapReduce application: a primary node that delegates a task to its workers (also called peers), and a single peer that takes the data given to it by the primary, performs a map pass and a reduce pass on the data, and eventually passes that data back to the primary for post-processing.  This assignment will require the need to implement a simple message-passing protocol, and abstract the functionality of the map() and reduce() methods.

Sample Assignment: [wiki:EducationalAssignments/SimpleMapReduce Simple MapReduce Assignment]
<commit_msg>Revision 10 of RepyMapReduce<commit_after>= A !MapReduce service =
The original !MapReduce framework was designed and developed by Google to support large-scale distributed computing on large amounts of closely-networked computers.  The main drive for the !MapReduce framework stem from the intrinsic benefits of functional programming, more specifically the map() and reduce() functions.

There are 3 main steps in a !MapReduce run.

 1. Map step: data received by the worker is tossed into a map() function that will emit key-value pairs based on each aliquot of data.

 2. Partition (or shuffle) step: The generated key-value pairs from map() are hashed by its key to a particular worked and sent off.  If a key ends up on a particular work at the end of the partition phase, it is required that all other key-value pairs with the same key also be on the same host in order for the reduce step to work properly.

 3. Reduce step: The collected key-value pairs are collected into a key, value_1, value_2, ..., value_n list and can do computation to reduce the expanded data down to a useful conclusion.

This computation can be performed on an arbitrary number of worker machines, making it scalable to nearly all sizes of computing clusters!  !MapReduce frameworks often have a primary node delegating tasks to workers, organizing and distributing the initial data, and acting as a mediator in fault situations.

More detail on the use of !MapReduce is available from [http://research.google.com/archive/mapreduce-osdi04.pdf Google's 2004 OSDI paper] or an open-source implementation of !MapReduce in Java: [http://hadoop.apache.org/core/ Hadoop].
[[BR]]
[[BR]]
-----
'''These assignments are a work-in-progress, please e-mail our developer list for more information! '''
[[TOC(inline)]]
[[BR]]
[[BR]]
== Stage 1: A simple implementation of !MapReduce ==

To start out, students will make the base framework of a !MapReduce application: a primary node that delegates a task to its workers (also called peers), and a single peer that takes the data given to it by the primary, performs a map pass and a reduce pass on the data, and eventually passes that data back to the primary for post-processing.  This assignment will require the need to implement a simple message-passing protocol, and abstract the functionality of the map() and reduce() methods.

Sample Assignment: [wiki:EducationalAssignments/SimpleMapReduce Simple MapReduce Assignment]

{'fraction_non_alphanumeric': 0.059076923076923075, 'fraction_numerical': 0.0045128205128205125, 'mean_word_length': 4.598163030998852, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2083629', 'n_tokens_mistral': 1208, 'n_tokens_neox': 1157, 'n_words': 731}