<jupyter_start><jupyter_text>Simple distributed wordcount with MapReduce Check that file `file.txt` exists, view size.<jupyter_code>!ls -hal file.txt<jupyter_output><empty_output><jupyter_text>Copy file to HDFS<jupyter_code>!hdfs dfs -put -f file.txt<jupyter_output><empty_output><jupyter_text>Erase `result` folder.<jupyter_code>!hdfs dfs -rm -R result 2>/dev/null<jupyter_output><empty_output><jupyter_text>Run the bash wordcount command `wc` in parallel on the distributed file.<jupyter_code>!mapred streaming \
  -input file.txt \
  -output result \
  -mapper /bin/cat \
  -reducer /usr/bin/wc<jupyter_output><empty_output><jupyter_text>Check result of MapReduce job<jupyter_code>!hdfs dfs -cat result/part*<jupyter_output><empty_output><jupyter_text>Check that the word count is correct by comparing with `wc` on local host (warning: do not try with too large files).<jupyter_code>!wc file.txt<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10883620689655173, 'fraction_numerical': 0.0010775862068965517, 'mean_word_length': 6.741666666666666, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '65273', 'license': 'MIT', 'n_tokens_mistral': 344, 'n_tokens_neox': 340, 'n_words': 77, 'path': 'simplest_mapreduce_bash_wordcount.ipynb', 'repo_name': 'alhaol/big_data', 'size': '928'}