<jupyter_start><jupyter_text>2-JoiningDatasetsThis tutorial shows how to identify drug molecules in the PDB by joining two datasets: 1. Drug information from DrugBank2. Ligand information from RCSB PDB<jupyter_code>from pyspark.sql import SparkSession
from mmtfPyspark.datasets import customReportService, drugBankDataset
from mmtfPyspark.structureViewer import view_binding_site<jupyter_output><empty_output><jupyter_text>Configure Spark<jupyter_code>spark = SparkSession.builder.appName("2-JoiningDatasets").getOrCreate()<jupyter_output><empty_output><jupyter_text>Download open DrugBank datasetDownload a dataset of drugs from [DrugBank](https://www.drugbank.ca) and filter out any drugs that do not have an InChIKey. [InChIKeys](https://en.wikipedia.org/wiki/International_Chemical_Identifier) are unique identifiers for small molecules. DrugBank provides more [detailed datasets](https://github.com/sbl-sdsc/mmtf-pyspark/blob/master/mmtfPyspark/datasets/drugBankDataset.py), e.g., subset of approved drugs, but a DrugBank username and password is required. For this tutorial we use the open DrugBank dataset.<jupyter_code>drugs = drugBankDataset.get_open_drug_links()
drugs = drugs.filter("StandardInChIKey IS NOT NULL").cache()
drugs.toPandas().head(5)<jupyter_output><empty_output><jupyter_text>NOTE: RCSB PDB web services have been depreciated!New functionality needs to be developed to enable this example.  Download ligand annotations from RCSB PDB Here we use [RCSB PDB web services](http://dx.doi.org/10.1093/nar/gkq1021) to download InChIKeys and molecular weight for ligands in the PDB (this step can be slow!).We filter out entries without an InChIKey and low molecular weight ligands using SQL syntax.<jupyter_code># ligands = customReportService.get_dataset(["ligandId","InChIKey","ligandMolecularWeight"])

# ligands = ligands.filter("InChIKey IS NOT NULL AND ligandMolecularWeight > 300").cache()

# ligands.toPandas().head(10)<jupyter_output><empty_output><jupyter_text>Find drugs in PDBBy [joining](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=joinpyspark.sql.DataFrame.join) the two datasets on the InChIKey, we get the intersection between the two datasets.<jupyter_code># ligands = ligands.join(drugs, ligands.InChIKey == drugs.StandardInChIKey)<jupyter_output><empty_output><jupyter_text>Keep only unique ligands per structure Here we [drop](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=joinpyspark.sql.DataFrame.dropDuplicates) rows with the same structureId and ligandId.<jupyter_code># ligands = ligands.dropDuplicates(["structureId","ligandId"]).cache()<jupyter_output><empty_output><jupyter_text>Keep only essential columns<jupyter_code># ligands = ligands.select("structureId","ligandId","chainId","Commonname")
# ligands.toPandas().head(10)<jupyter_output><empty_output><jupyter_text>Visualize drug binding sites  Extract id columns as lists (required for visualization)<jupyter_code># pdb_ids = ligands.select("structureId").rdd.flatMap(lambda x: x).collect()
# ligand_ids = ligands.select("ligandId").rdd.flatMap(lambda x: x).collect()
# chain_ids = ligands.select("chainId").rdd.flatMap(lambda x: x).collect()<jupyter_output><empty_output><jupyter_text>Disable scrollbar for the visualization below<jupyter_code>#%%javascript 
#IPython.OutputArea.prototype._should_scroll = function(lines) {return false;}<jupyter_output><empty_output><jupyter_text>Show binding site residues within 4.5 A from the drug molecule<jupyter_code># view_binding_site(pdb_ids, ligand_ids, chain_ids, distance=4.5);
spark.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10664094791953706, 'fraction_numerical': 0.0071645081289611464, 'mean_word_length': 8.029850746268657, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 42, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '195344', 'license': 'Apache-2.0', 'n_tokens_mistral': 1209, 'n_tokens_neox': 1136, 'n_words': 262, 'path': '4-mmtf-pyspark-advanced/2-JoiningDatasets.ipynb', 'repo_name': 'sbl-sdsc/mmtf-workshop-2018', 'size': '3629'}