<jupyter_start><jupyter_text>1-Inputmmtf-pyspark operates on 3D structures in the compressed binary MMTF file format.Info about MMTF:* [Website](http://mmtf.rcsb.org/index.html)* [Format paper](https://doi.org/10.1371/journal.pcbi.1005575)* [Compression paper](https://doi.org/10.1371/journal.pone.0174846)* [Specification](https://github.com/rcsb/mmtf/blob/master/spec.md)Protein Data Bank structures are available in two MMTF data representations:* full * All atom representation  * 0.001Å coordinate precision, 0.01 B-factor and occupancy precision* reduced * C-alpha atoms only for polypeptides  * P-backbone atoms only for polynucleotides  * All atom representation for all other residue types  * 0.1Å coordinate precision, 0.1 B-factor and occupancy precision.  Import pyspark and mmtfPyspark<jupyter_code>from pyspark import SparkContext
from pyspark.sql import SparkSession
from mmtfPyspark.io import mmtfReader<jupyter_output><empty_output><jupyter_text>Configure Spark<jupyter_code>spark = SparkSession.builder.master("local[4]").appName("1-Input").getOrCreate()
sc = spark.sparkContext
sc.defaultParallelism<jupyter_output><empty_output><jupyter_text>Download StructuresFor a small list of PDB entries (10s to 100), the download methods are the quickest way to import structures. Here we download a list of 4 structure in the full representation.<jupyter_code>pdbids = ['1LQ9','1LXJ','4XPX','1P1J']
structures = mmtfReader.download_full_mmtf_files(pdbids, sc)<jupyter_output><empty_output><jupyter_text>Structures are represented as keyword-value pairs (tuples):* key: structure identifier (e.g., PDB ID)* value: MmtfStructure (structure data)We can print the keys and values using the collect() method. Note, that the structures are loaded in an arbritray order. You cannot rely on the order of structures.<jupyter_code>structures.keys().collect()
structures.values().collect()<jupyter_output><empty_output><jupyter_text>Spark represents these keyword-value pairs as Resilient Distributed Datasets (RDDs), which are a fault-tolerant collection of elements that can be operated on in parallel. To see how the dataset was distributed, we can print the number of partitions.<jupyter_code>structures.getNumPartitions()<jupyter_output><empty_output><jupyter_text>Reading structures from an MMTF Hadoop Sequence FileNext, we read PDB structures from a local copy of an MMTF Hadoop Sequence file. For the following examples to work, the MMTF_FULL and MMTF_REDUCED environment variables need to be set. See installation instructions for details. If you have long list (1000s) of PDB IDs, you can read the list of structures from a local copy of the MMTF Hadoop Sequence file,however, it's very inefficent for a few structures, e.g, in the example below.<jupyter_code>path = "../resources/mmtf_reduced_sample/"
structures = mmtfReader.read_sequence_file(path,sc, pdbids)<jupyter_output><empty_output><jupyter_text>Let's print the keys again and see how long this takes. You can see that Spark loads the data only when and if it's required.<jupyter_code>structures.keys().collect()<jupyter_output><empty_output><jupyter_text>Now, let's read a sample of the PDB archive from the MMTF Hadoop Sequence file<jupyter_code>structures = mmtfReader.read_sequence_file(path, sc).cache()<jupyter_output><empty_output><jupyter_text>There are 9756 structures in the sample file<jupyter_code>%%time
structures.count()<jupyter_output>CPU times: user 16.6 ms, sys: 0 ns, total: 16.6 ms
Wall time: 12.4 s<jupyter_text>About data flow and caching in SparkNow, let's count the number of structures again. Should this be faster this time since we already loaded the entire PDB? Not necessarily, the data from the Hadoop Sequence file are streamed through parallel threads. If you need the data again, they need to be reloaded from scratch, unless they are cached. See .cache() method call after reading the MMTF Hadoop Sequence file.Remove the .cache() method call, run this notebook again and compare the time it takes to count the number of structures.<jupyter_code>%%time
structures.count()<jupyter_output>CPU times: user 11.4 ms, sys: 518 µs, total: 11.9 ms
Wall time: 1.85 s<jupyter_text>Reading the whole PDB from MMTF-Hadoop Sequence files In this workshop we use a sample set of the PDB with about 10,000 structures.To use the entire PDB, the MMTF_FULL and MMTF_REDUCED environment variables must to be set. See mmtf-pypark [installation instructions](https://github.com/sbl-sdsc/mmtf-pysparkinstallation) for details.  Read whole PDB in the full (all atom) representationWe commented this lines below, since we are using a smaller sample of the PDB for the tutorials. To use the whole PDB, the MMTF_FULL and MMTF_REDUCED environment variables need to be set to the `full` and `reduced` MMTF Hadoop Sequence file locations. See [installation instructions](https://github.com/sbl-sdsc/mmtf-pysparkhadoop-sequence-files) for details.<jupyter_code># %%time
# pdb_full = mmtfReader.read_full_sequence_file(sc);
# pdb_full.count()<jupyter_output><empty_output><jupyter_text>Read whole PDB in the reduced representation<jupyter_code># %%time
# pdb_reduced = mmtfReader.read_reduced_sequence_file(sc);
# pdb_reduced.count()<jupyter_output><empty_output><jupyter_text>Very Important: Stop Spark!!!It is very important to run the notebook all the way to the spark.stop() statement to terminate Spark. Otherwise you may end up running multiple instances of Spark that will interfere with each other.<jupyter_code>spark.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08110549132947976, 'fraction_numerical': 0.015895953757225433, 'mean_word_length': 6.256880733944954, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '251574', 'license': 'Apache-2.0', 'n_tokens_mistral': 1782, 'n_tokens_neox': 1625, 'n_words': 605, 'path': '3-mmtf-pyspark/1-Input.ipynb', 'repo_name': 'NicholasAKovacs/mmtf-workshop', 'size': '5536'}