<jupyter_start><jupyter_text>Notebook for extracting relevant info. about cardiovascular drugs extracted from DrugBank XML (related entities, name, synonyms, etc.) and subsequently cleansing the data<jupyter_code>import json
import os
import _pickle
import xml.etree.ElementTree as ET
from lxml import etree
from nltk import flatten
from importlib import reload
os.chdir(r'C:\Users\ttran\OneDrive\Desktop\COVID-CDV-DATA\covidii_KG')
import parse_xml
from parse_xml import *
import cleanse_data
from cleanse_data import *
import parse_web
from parse_web import *<jupyter_output><empty_output><jupyter_text>Reload modules and re-import classes when needed<jupyter_code>reload(parse_xml)
from parse_xml import *
reload(cleanse_data)
from cleanse_data import *
reload(parse_web)
from parse_web import *
tree = ET.parse('fulldb21.xml')
root = tree.getroot()<jupyter_output><empty_output><jupyter_text>Previously, we extracted a list of CV drugs from the XML file. We saved a list of the corresponding DrugBank accession numbers/IDs of those drugs as a python file called 'cv_drugs_dbid'. We will use these IDs to extract all the relevant info. about the drug using the ParseXML class.<jupyter_code>with open("cv_drugs_dbid", 'rb') as f:
    cv_drugs_dbid = _pickle.load(f)
DATA = []

for ele in root:
    if ParseXML.getID(ele) in cv_drugs_dbid:
        data = {}
        ID = ParseXML.getID(ele)
        name = ParseXML.getName(ele) 
        synonyms = ParseXML.getSynonyms(ele)
        description = ParseXML.getDescription(ele)
        category = ParseXML.getCategory(ele)
        atccode = ParseXML.getATCCode(ele)
        indication = ParseXML.getIndication(ele)
        carriers = ParseXML.getEntities(ele,'carriers')
        targets = ParseXML.getEntities(ele,'targets')
        transporters = ParseXML.getEntities(ele,'transporters')
        enzymes = ParseXML.getEntities(ele,'enzymes')
        pathways = ParseXML.getPathways(ele)
    
        data.update({"drugbank_id": ID,\
                "name": name,\
                "synonyms":synonyms,\
                "descriptions": description,\
                "categories": category,\
                "ATC code": atccode,\
                "indication": indication,\
                "carriers": carriers,\
                "targets":targets,\
                "transporters":transporters,\
                "enzymes":enzymes,\
                "pathways": pathways})    
        DATA.append(data)
DATA[0]<jupyter_output><empty_output><jupyter_text>Get list of index from DATA where the drug does not interact with any entity (i.e. the value of 'targets', 'carriers', 'enzymes', and 'transporters' is an empty list). These will be removed.<jupyter_code>oCleanseData = CleanseData(DATA)
index_list = oCleanseData.getDrugIndexList()
print(index_list)<jupyter_output>[221, 233, 235, 238, 249, 254, 267, 273, 274, 276, 279, 289, 301, 305, 311, 314, 317, 319, 320, 323, 324, 326, 328, 329, 331, 332, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 376, 377, 378, 379, 381, 382, 384, 385, 386, 387, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 421, 424, 425, 426, 428, 429, 430, 431, 432, 433, 434, 435, 439, 440, 441, 442, 447, 448, 449]<jupyter_text>Get list of index from DATA where the drug contains at least one entity with a 'Null' UniProt ID. These entities will be removed and the drugs interacting exclusively with these entities will also be removed. The values for each dictionary represent the names of ALL the entities (not just the entity with the null UniProt ID) for that particular index.<jupyter_code>nulluid_index_info = oCleanseData.getNullUIDIndexInfo()
nulluid_index_info<jupyter_output><empty_output><jupyter_text>Remove entities with 'Null' UniProt ID<jupyter_code>DATA = oCleanseData.removeNullUIDEntity()
DATA[134]
# Entity named 'Bile acids' (does not have a UniProt ID) is gone.<jupyter_output><empty_output><jupyter_text>Remove drugs that interact exclusively with entities with 'Null' UniProt ID<jupyter_code># re-initialize object with updated data
oCleanseData = CleanseData(DATA)

# remove drugs
DATA = oCleanseData.removeDrugs()
# DATA now has 322 instead of 450 drugs.
len(DATA)<jupyter_output><empty_output><jupyter_text>Make a list (uid_list) of unique UniProt IDs from DATA<jupyter_code>uid_list = []
for drug in DATA:
        for entity in ['targets','enzymes','transporters','carriers']:
            for ent in drug[entity]:
                uid = ent['uniprot_id']
                if not uid in uid_list:
                    uid_list.append(uid)<jupyter_output><empty_output><jupyter_text>Get list (dup_uids) of Uniprot IDs that correspond to multiple entities (i.e. duplicate UniProt IDs) from DATA<jupyter_code># re-initialize object with updated data
oCleanseData = CleanseData(DATA)

dup_uids = oCleanseData.getDupUIDs()
dup_uids<jupyter_output><empty_output><jupyter_text>Make dictionaries with the names, DrugBank IDs, and UniProt IDs as keys. The dictionary with the UniProt IDs as the keys will only contain non-duplicate UniProt IDs. These dictionaries will make extracting info. much easier downstream.<jupyter_code>ent_name = []
ent_dbid = []
ent_uid = []
name_dict = {}
dbid_dict = {}
uid_dict = {}

for drug in DATA:
    for entity in ['targets','carriers','transporters','enzymes']:
        for ent in drug[entity]:
            if not ent['name'] in ent_name:
                ent_name.append(ent['name'])
                name_dict.update({ent['name']:{'drugbank_id':ent['drugbank_id'],\
                                                     'uniprot_id':ent['uniprot_id']}})
            if not ent['drugbank_id'] in ent_dbid:
                ent_dbid.append(ent['drugbank_id'])
                dbid_dict.update({ent['drugbank_id']:{'name':ent['name'],\
                                                     'uniprot_id':ent['uniprot_id']}})
            if not ent['uniprot_id'] in ent_uid and not \
            ent['uniprot_id'] in list(dup_uids.keys()):
                ent_uid.append(ent['uniprot_id'])
                uid_dict.update({ent['uniprot_id']:{'name':ent['name'],\
                                                     'drugbank_id':ent['drugbank_id']}})<jupyter_output><empty_output><jupyter_text>Make a list (ent2check) containing the DrugBank IDs of all the duplicate UniProt IDs. Using the ParseWeb class (contains methods for web scraping), we will check whether each ID in the list corresponds to a protein or protein group. If the latter, then we will find the protein members of the group in the DrugBank webpage corresponding to that group. The entries corresponding to the protein group in DATA will be replaced with the members.<jupyter_code>ent2check = flatten(list(dup_uids.values()))
ent2check = list(set(ent2check))<jupyter_output><empty_output><jupyter_text>From the ent2check list, make a sub-list (protein_dbids) containing only the DrugBank IDs corresponding to proteins, not protein groups. Then, create a list (ent_list) containg dictionaries with each corresponding to a protein group in the ent2check list and info. about its name, DrugBank ID, and members.<jupyter_code>ent_list = []
protein_dbids = []

for ent in ent2check:
    oParseWeb = ParseWeb('https://go.drugbank.com/bio_entities/' + ent,\
                            ent)
    html = oParseWeb.getHTML()
    kind = oParseWeb.getKind(html)
    if kind=='protein':
        protein_dbids.append(ent)

for ent in ent2check:
    oParseWeb = ParseWeb('https://go.drugbank.com/bio_entities/' + ent,\
                            ent)
    html = oParseWeb.getHTML()
    memberDict = oParseWeb.getMembers(html,dup_uids,uid_list,protein_dbids,\
                                      dbid_dict,uid_dict)
    if memberDict!=None:
        ent_list.append(memberDict)
ent_list<jupyter_output><empty_output><jupyter_text>Make a list (ent_dbid) containing the DrugBank IDs of the protein groups.<jupyter_code>ent_dbid = []
for i in ent_list:
    ent_dbid.append(i['drugbank_id'])<jupyter_output><empty_output><jupyter_text>Merge entities with the same UniProt ID<jupyter_code># before merge
DATA[21]['targets']
DATA = oCleanseData.mergeDuplicateUIDs(ent_dbid,ent_list)
# after merge
DATA[21]['targets']
# re-initialize object with updated data
oCleanseData = CleanseData(DATA)

# call getDupUIDs method again to see if the duplicate Uniprot IDs are gone
dup_uids = oCleanseData.getDupUIDs()
# no more duplicate UniProt IDs!
dup_uids<jupyter_output><empty_output><jupyter_text>Export cleansed data as json file<jupyter_code>with open("cvdrug_ent_pw.json", 'w') as f:
    json.dump(DATA,f)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10248868778280543, 'fraction_numerical': 0.04479638009049774, 'mean_word_length': 3.7202349172450613, 'pattern_counts': {'":': 12, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 7}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '170965', 'license': 'MIT', 'n_tokens_mistral': 3229, 'n_tokens_neox': 2720, 'n_words': 820, 'path': '02_KG_DataPrep_DrugEntDrugPathway/02_KG_DataPrep_DrugEntDrugPathway.ipynb', 'repo_name': 'CaseOLAP/covid-cvd-knowledgegraph', 'size': '8840'}