<jupyter_start><jupyter_text>Secondary Structure Segment DemoThis demo shows how to get a dataset of secondary structure segment Imports<jupyter_code>from pyspark import SparkConf, SparkContext
from mmtfPyspark.io import mmtfReader
from mmtfPyspark.mappers import StructureToPolymerChains
from mmtfPyspark.filters import ContainsLProteinChain
from mmtfPyspark.datasets import secondaryStructureSegmentExtractor<jupyter_output><empty_output><jupyter_text>Configure Spark<jupyter_code>conf = SparkConf().setMaster("local[*]") \
                      .setAppName("secondaryStructureSegmentDemo")
sc = SparkContext(conf = conf)<jupyter_output><empty_output><jupyter_text>Download protein (1STP) Note: Need to use SparkContext as parameter to download Mmtf files<jupyter_code>pdb = mmtfReader.download_mmtf_files(['1STP'], sc).cache()<jupyter_output><empty_output><jupyter_text>Map protein to polymer chains and apply LProteinChain filter<jupyter_code>pdb = pdb.flatMap(StructureToPolymerChains()) \
         .filter(ContainsLProteinChain())<jupyter_output><empty_output><jupyter_text>Extract secondary structure element 'E'<jupyter_code>segmentLength = 25
ds = secondaryStructureSegmentExtractor.get_dataset(pdb, segmentLength)

ds.show(50, False)<jupyter_output>+----------------+-------------------------+-------+-------+
|structureChainId|sequence                 |labelQ8|labelQ3|
+----------------+-------------------------+-------+-------+
|1STP.A          |DPSKDSKAQVSAAEAGITGTWYNQL|C      |C      |
|1STP.A          |PSKDSKAQVSAAEAGITGTWYNQLG|H      |H      |
|1STP.A          |SKDSKAQVSAAEAGITGTWYNQLGS|H      |H      |
|1STP.A          |KDSKAQVSAAEAGITGTWYNQLGST|H      |H      |
|1STP.A          |DSKAQVSAAEAGITGTWYNQLGSTF|H      |H      |
|1STP.A          |SKAQVSAAEAGITGTWYNQLGSTFI|C      |C      |
|1STP.A          |KAQVSAAEAGITGTWYNQLGSTFIV|E      |E      |
|1STP.A          |AQVSAAEAGITGTWYNQLGSTFIVT|E      |E      |
|1STP.A          |QVSAAEAGITGTWYNQLGSTFIVTA|E      |E      |
|1STP.A          |VSAAEAGITGTWYNQLGSTFIVTAG|E      |E      |
|1STP.A          |SAAEAGITGTWYNQLGSTFIVTAGA|E      |E      |
|1STP.A          |AAEAGITGTWYNQLGSTFIVTAGAD|T      |C      |
|1STP.A          |AEAGITGTWYNQLGSTFIVTAGADG|T      |C      |
|1STP.A          |EAGITG[...]<jupyter_text>Terminate Spark<jupyter_code>sc.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13938618925831203, 'fraction_numerical': 0.00937766410912191, 'mean_word_length': 2.977966101694915, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '95475', 'license': 'Apache-2.0', 'n_tokens_mistral': 905, 'n_tokens_neox': 867, 'n_words': 121, 'path': 'docs/_static/demos/datasets/SecondaryStructureSegmentDemo.ipynb', 'repo_name': 'sbliven/mmtf-pyspark', 'size': '2346'}