<jupyter_start><jupyter_text>The purpose of this notebook is to complete a data cleaning workflow from start to finish in order to validate the core functionality our package TO DO:- Add in complete PubChem data- Write PubChem function- Organize code modules & tests- Clean up/finish writing tests- Write main script wrapper function<jupyter_code># imports

from core import *
from cpd_info import *
from mol_sim_copy import *

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Step 1 Generate dataframe of all current kegg enzymes from zipped text file<jupyter_code># create_kegg_df()

enzyme_df = create_kegg_df('../datasets/KEGG_enzyme_db_entries.gz', 'enzyme')
print(enzyme_df.shape)
enzyme_df.head()<jupyter_output>(7524, 16)<jupyter_text>Step 2 Down select promiscuous enzymes from master enzyme dataframe<jupyter_code># select_promiscuous_enzymes()

prom_df = select_promiscuous_enzymes(enzyme_df)
print(prom_df.shape)
prom_df.head()<jupyter_output>(549, 4)<jupyter_text>Step 3 Combine substrates and products to account for reversible reactions<jupyter_code># combine_substrates_products()
    
combo_df = combine_substrates_products(prom_df)
print(combo_df.shape)
combo_df.head()<jupyter_output>(549, 2)<jupyter_text>Step 4 Expand dataframe so that each row is a unique enzyme-product pairThere are multiple compounds in the 'product' field of the dataframe. This function parses each of those, and for each provides a new enzyme-product pair.<jupyter_code># explode_dataframe()

exploded_df = explode_dataframe(combo_df, parse_compound_ids, 'product', ['entry'])
print(exploded_df.shape)
exploded_df.head()<jupyter_output>(3697, 2)<jupyter_text>Step 5 Remove cofactors from dataframeWe have curated a list of 37 common cofactors and reactant molecules that participate in reactions, but are not products that we care to train our model on. These steps removes cofactor data from our master dataset.<jupyter_code># remove_cofactors()

cofactors_df = pd.read_csv('../datasets/cofactor_list.csv')
clean_df = remove_cofactors(exploded_df, 'product', cofactors_df, 'CPD')
print(clean_df.shape)
clean_df.head()<jupyter_output>(2144, 2)<jupyter_text>Step 6 Curate negative datasetSo far our curated dataset includes only examples of enzyme-product pairs that are known to react. In order to train our model, we need to include negative examples of enzyme-product pairs not expected to react. This function artifically pairs enzymes and products that are not known to react, and selects a subsample of these negative pairs to include in the master dataset<jupyter_code># create_negative_matches()

pos_df, neg_df = create_negative_matches(clean_df, 'entry', 'product')
neg_df = neg_df.sample(frac=0.01)

# negative data

print(neg_df.shape)
neg_df.head()
# positive data

print(pos_df.shape)
pos_df.head()<jupyter_output>(2144, 3)<jupyter_text>Step 7 Get SMILES strings for each product from the PubChem databaseThe RDKit chemistry package requires SMILES strings as an input. The Kegg database does not store SMILES strings for compounds. This step joins a previously curated dataset of SMILES strings into our master dataset<jupyter_code># read in smiles data

smiles_df = pd.read_csv('../datasets/df_cleaned_kegg_with_smiles.csv')
smiles_df = smiles_df.drop_duplicates(subset='SMILES')
print(smiles_df.shape)
smiles_df.head()
# add SMILES strings to positive data

pos_df = pd.merge(pos_df, smiles_df, how='inner', left_on='product', right_on='KEGG')
pos_df = pos_df.drop(columns=['Unnamed: 0', 'KEGG', 'CID', 'entry'])
pos_df = pos_df[pos_df['SMILES'] != 'none']
pos_df = pos_df.reset_index(drop=True)
print(pos_df.shape)
pos_df.head()
# add SMILES strings to negative data

neg_df = pd.merge(neg_df, smiles_df, how='inner', left_on='product', right_on='KEGG')
neg_df = neg_df.drop(columns=['Unnamed: 0', 'KEGG', 'CID', 'entry'])
neg_df = neg_df[neg_df['SMILES'] != 'none']
neg_df = neg_df.reset_index(drop=True)
print(neg_df.shape)
neg_df.head()<jupyter_output>(5992, 5)<jupyter_text>Step 8 Pre-process negative and positive datasets to remove rows with only 1 enzymePositive and negative data must be processed separately, as the average molecular distances will be calculated separately for each dataset<jupyter_code># remove_single_cpd_rows()
# positive data

pos_master_df = remove_single_cpd_rows(pos_df, 'enzyme', 'SMILES')
pos_master_df = pos_master_df.reset_index(drop=True)
print(pos_master_df.shape)
pos_master_df.head()
# negative data

neg_master_df = remove_single_cpd_rows(neg_df, 'enzyme', 'SMILES')
neg_master_df = neg_master_df.reset_index(drop=True)
print(neg_master_df.shape)
neg_master_df.head()<jupyter_output>(5992, 5)<jupyter_text>Step 9 Calculate molecular distances between products of the same enzymePositive and negative data must be calculated separately because the calculated molecular distance is an average of the distances between all products of any one enzyme. The model is based on the fact that we expect these distances to be closer for reactive enzyme-product pairs than for non-reactive enzyme-product pairs<jupyter_code># calculate_dist()
# positive data

pos_dist_df = calculate_dist(pos_master_df)
print(pos_dist_df.shape)
pos_dist_df.head()
# negative data

neg_dist_df = calculate_dist(neg_master_df)
print(neg_dist_df.shape)
neg_dist_df.head()
# concatenate negative & positive data for further processing

combined_df = pd.concat((pos_dist_df, neg_dist_df), axis=0)
print(combined_df.shape)
combined_df.head()<jupyter_output>(7652, 8)<jupyter_text>Step 10 Get dummy variables to represent enzyme classWe expect that many enzyme properties could be predictive features for this model. Enzyme class should encapsulate many of these features at a high level.<jupyter_code># binarize_enzyme_class()

master_df = binarize_enzyme_class(combined_df, 'entry')
master_df_df = master_df.reset_index(drop=True)
print(master_df.shape)
master_df.head()<jupyter_output>(7652, 15)<jupyter_text>Step 11 Add in compound features with RDKitThis step uses the RDKit packages to generate descriptive features of the reaction product compounds<jupyter_code>master_df = create_cpd_info(master_df)
print(master_df.shape)
master_df.head()
# write master dataframe to csv

# master_df.to_csv('../datasets/MASTER_DF.csv', index=False)<jupyter_output><empty_output><jupyter_text>Out of curiosity: Examine average molecular distance distributions for negative and positive dataOn first glance, it appears that our hypothesis is correct in that the distributions of average molecular distances are qualitatively different between the positive and negative datasets<jupyter_code># look at distributions of distances for positive and negative data

#fig, axes = plt(1, 1, figsize=(9, 5))
fig, axes = plt.plot(1, 1, figsize=(9, 5))

pos = sns.distplot(pos_dist_df['dist'], bins=90, kde=True, norm_hist=True, ax=axes[0], color= 'red')
axes[0].set_title('Avg. Tanimoto Similarity to Positive Prediction Products')

neg = sns.distplot(neg_dist_df['dist'], bins=30, kde=True, norm_hist=False, ax=axes[0])
axes[0].set_title('Avg. Tanimoto Similarity to Negative Prediction Products')

for axis in axes:
    axis.set_xlim([0.0, 1.0])
    
fig.savefig('output')
# look at distributions of distances for positive and negative data
pos_dist = pd.Series(pos_dist_df['dist'], name="Avg. Tanimoto Similarity")
neg_dist = pd.Series(neg_dist_df['dist'], name="Avg. Tanimoto Similarity")

fig = sns.distplot(pos_dist, bins=75, kde=True,
                   kde_kws={"color": "r", "lw": 2, "label": "Positive Prediction Products"},
                   hist_kws={"histtype": "bar", "linewidth": 3, "alpha": 0.5, "color": "r"})

fig.set_xlim([0.0, 1.0])
sns.distplot(neg_dist, bins=25, kde=True,
                   kde_kws={"color": "b", "lw": 2, "label": "Negative Prediction Products"},
                   hist_kws={"histtype": "bar", "linewidth": 3, "alpha": 0.5, "color": "b"})
fig.set_title('Normed Distribution of Avg. Similarity to Products ')
plt.savefig('output')
fig.savefig<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09884943709018928, 'fraction_numerical': 0.013361375726834096, 'mean_word_length': 6.527001862197393, 'pattern_counts': {'":': 14, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '75068', 'license': 'MIT', 'n_tokens_mistral': 2784, 'n_tokens_neox': 2597, 'n_words': 781, 'path': 'deprecated/notebooks/data_curation_workflow - Copy.ipynb', 'repo_name': 'metamoles/metamoles', 'size': '8083'}