<jupyter_start><jupyter_text>Data curation of HTR3A protein data from ExCAPE-DB Data  Scope of the tutorial*   Input data from Excape dataset for HTR3A protein target*   AMPL will be used to accomplish the following steps:   *   Standardize SMILES string  *   Clean the data (look for duplicates, average the assay data, cluster the compounds etc.)  *   Curate the data  *   Carry out some Exploratory Data Analysis (Chemical space exploration; PCA etc.)   Time on COLAB-Pro ( ~ 3 minutes)  Protein target (HTR3A) information The Target specific data was downloaded from https://zenodo.org/record/173258.X_44kuhKi70Please refer to the ExCAPE-DB publication (https://pubmed.ncbi.nlm.nih.gov/28316655/) for details about the database  Here are some details about HTR3A gene (taken from RefSeq NCBI)*   Proteins belongs to GPCR superfamily*   HTR3a is a receptor for Serotonin, a biogenic hormone that functions as a neurotransmitter*   HTR3A (also the name of the gene) encodes the subunit of the type 3 receptor for neurotransmitter *   Herteromeric combination of subunit A and B (HTR3B) is needed for full function. *   Different alternately spliced transcript variant forms for this gene are available.   Diseases associated with HTR3A include Irritable Bowel Syndrome and Motion Sickness.  Basic information about HTR3A gene:**Present in** Chromosome 11**Exon count:** 10mRNA and protein information*   NM_000869.6 → NP_000860.3 *   NM_001161772.3 → NP_001155244.1*   NM_213621.4 → NP_998786.3   Before you begin, make sure you close all other COLAB notebooks.   Change Runtime settingsIf you have access to COLAB-Pro (commercial/not-free), please change your runtime settings to use GPU and high-memory,```Runtime --> Change Runtime Type --> GPU with high-RAM```If you are not a paid COLAB-Pro customer, you can still choose GPU, with standard-RAM.<jupyter_code>!date # starting time<jupyter_output>Fri Sep 17 10:56:49 UTC 2021<jupyter_text>Install AMPL<jupyter_code>! pip install rdkit-pypi
! pip install --pre deepchem

import deepchem
# print(deepchem.__version__)
! pip install umap
! pip install llvmlite==0.34.0  --ignore-installed
! pip install umap-learn
! pip install molvs
! pip install bravado
import deepchem as dc

# get the Install AMPL_GPU_test.sh
!wget 'https://raw.githubusercontent.com/ATOMconsortium/AMPL/master/atomsci/ddm/examples/tutorials/config/install_AMPL_GPU_test.sh'

# run the script to install AMPL
! chmod u+x install_AMPL_GPU_test.sh
! ./install_AMPL_GPU_test.sh<jupyter_output><empty_output><jupyter_text>Exploring target activity data from selected database Data source = ExcapeDB Target activity = HTR3A<jupyter_code># We temporarily disable warnings for demonstration.
# FutureWarnings and DeprecationWarnings are present from some of the AMPL 
# dependency modules.
import warnings
warnings.filterwarnings('ignore')
import json
import requests
import sys
#
# Import AMPL libraries
# 
import atomsci.ddm.utils.data_curation_functions as dcf
import atomsci.ddm.utils.curate_data as curate_data
import atomsci.ddm.pipeline.diversity_plots as dp
import atomsci.ddm.pipeline.chem_diversity as cd
# Additional python libraries
import pandas as pd
import numpy as np
import getpass, os<jupyter_output><empty_output><jupyter_text>Select a target to work with  (e.g. PDE2A, KCNH2, SCNA5)<jupyter_code>target_name='HTR3A'
ofile=target_name+'_excape_curated.csv'
import io
url = 'https://raw.githubusercontent.com/ATOMconsortium/AMPL/master/atomsci/ddm/examples/tutorials/datasets/Excape_HTR3A.tsv'
download = requests.get(url).content
# Reading the downloaded content and turning it into a pandas dataframe
data = pd.read_csv(io.StringIO(download.decode('utf-8')), sep='\t', header=0 )<jupyter_output><empty_output><jupyter_text>define data locations   Load previously saved raw data<jupyter_code>data.head()
data.rename( columns={ "pXC50" : "PXC50", "Ambit_InchiKey" : "compound_id" }, inplace=True)
data.head()
# import os
# working_dir=tmp_dir
curated_file=ofile
data['SMILES']
# Generate SMILES, InChI keys for dataset with curation and structure modules.
# RDkit modules are used to process the SMILES strings

data['rdkit_smiles'] = data['SMILES'].apply(curate_data.base_smiles_from_smiles)
# data['inchi_key'] = data['smiles'].apply(struct_utils.smiles_to_inchi_key)
print(data.shape)
print(data['rdkit_smiles'].value_counts()[0:10])
# import os
# working_dir=tmp_dir
# ofile=target_name+'_excape_curated.csv.'+username

# column: Response values column
column = 'PXC50'

# tolerance: Percentage of individual respsonse values allowed to different from 
# the average to be included in averaging
tolerance = 10

# list_bad_duplicates: Print structures with bad duplicates
list_bad_duplicates = 'Yes'

# max_std: Maximum allowed standard deviation for computed average response value
# NOTE: In this example, we set this value very high to disable this feature
max_std = 1

# compound_id: Compound ID column
compound_id = 'compound_id'

# smiles_col: SMILES column
smiles_col = 'rdkit_smiles'

# column that will be averaged ; smiles_col; used to find the replicates
# 

curated_df = curate_data.average_and_remove_duplicates(column, tolerance, 
                                                       list_bad_duplicates, 
                                                       data, max_std, 
                                                       compound_id=compound_id, 
                                                       smiles_col=smiles_col)
# curated_file = os.path.join(working_dir, ofile)
## discard out of range values
curated_df=curated_df[curated_df['VALUE_NUM_mean']>2]
curated_df.to_csv(curated_file, index=False)<jupyter_output>Bad duplicates removed from dataset
Dataframe size (643, 17)
List of 'bad' duplicates removed
                     compound_id    PXC50  ...   Perc_Var  VALUE_NUM_std
36   BSNKYWSMUAGMDO-UHFFFAOYNA-N  6.57025  ...  11.638406       1.223846
37   BSNKYWSMUAGMDO-UHFFFAOYNA-N  8.30103  ...  11.638406       1.223846
227  JBVRKZZSTULWTO-OAHLLOKONA-N  9.30103  ...  15.893248       1.803851
228  JBVRKZZSTULWTO-OAHLLOKONA-N  6.75000  ...  15.893248       1.803851
548  WLINCFBAKQHMPZ-OAHLLOKONA-N  8.19382  ...   9.661653       1.020935
549  WLINCFBAKQHMPZ-OAHLLOKONA-N  6.75000  ...   9.661653       1.020935

[6 rows x 5 columns]

Dataset de-duplicated
Dataframe size (602, 17)
New column created with averaged values:  VALUE_NUM_mean<jupyter_text>Note this includes the compounds discarded by measurement variation and out of range values<jupyter_code>curated_df['Remove_BadDuplicate'].value_counts()
compound_id='rdkit_smiles'
reject=data[~data[compound_id].isin(curated_df[compound_id])]
reject<jupyter_output><empty_output><jupyter_text>save rejected molecules for further examination<jupyter_code>print("Number of discarded compounds",reject[compound_id].nunique())

ofile=target_name+'.reject_check.csv'
reject.to_csv(ofile,index=False)
import matplotlib.pyplot as plt
import seaborn as sns
fig, ax = plt.subplots()

# This will be our before curation
before=data["PXC50"]

# current curated dataframe will not include any out of range values
after=curated_df["PXC50"]

sns.distplot(before, bins=range(0, 12, 1), ax=ax, kde=False, label='before')
sns.distplot(after, bins=range(0, 12, 1), ax=ax, kde=False, label='after')
plt.legend()
ax.set_xlim([0, 12])
column = 'PXC50'
num_bins = 20
title = target_name
units = 'NM'
filepath = ""
data=curated_df

curate_data.summarize_data(column, num_bins, title, units, filepath, data)
!date #ending time
curated_df.head()
curated_df.shape
!date<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08588714980928581, 'fraction_numerical': 0.03827436538208602, 'mean_word_length': 4.518142235123367, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 30, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '172668', 'license': 'MIT', 'n_tokens_mistral': 2759, 'n_tokens_neox': 2473, 'n_words': 712, 'path': 'atomsci/ddm/examples/tutorials/02_Explore_Data_ExcapeDB_curation.ipynb', 'repo_name': 'vgutta/AMPL', 'size': '7603'}