<jupyter_start><jupyter_text>5-MapReduceThis tutorial demonstrates how to use Map and Reduce to count the number of atoms in a structure.<jupyter_code>from pyspark.sql import SparkSession
from mmtfPyspark.io import mmtfReader<jupyter_output><empty_output><jupyter_text>Configure Spark<jupyter_code>spark = SparkSession.builder.appName("5-MapReduce").getOrCreate()<jupyter_output><empty_output><jupyter_text>Read PDB structures<jupyter_code>path = "../resources/mmtf_full_sample"

pdb = mmtfReader.read_sequence_file(path)<jupyter_output><empty_output><jupyter_text>MapUse a lambda expression to get the number of atoms for each entry.The variable t represents a tuple (PDB ID, mmtfStructure)* t[0]: PDB ID* t[1]: mtfStructure<jupyter_code>num_atoms = pdb.map(lambda t: t[1].num_atoms)<jupyter_output><empty_output><jupyter_text>Print the number of atoms for 10 entries<jupyter_code>num_atoms.take(10)<jupyter_output><empty_output><jupyter_text>ReduceUse the reduce method with a summation function defined as a lambda expression<jupyter_code>num_atoms.reduce(lambda a, b: a+b)
spark.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09651474530831099, 'fraction_numerical': 0.00804289544235925, 'mean_word_length': 9.467289719626168, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '22079', 'license': 'Apache-2.0', 'n_tokens_mistral': 409, 'n_tokens_neox': 389, 'n_words': 83, 'path': '3-mmtf-pyspark/5-MapReduce.ipynb', 'repo_name': 'ashokbiotech/mmtf-workshop-2018', 'size': '1119'}