<jupyter_start><jupyter_text>DeepPurpose Deep Dive Tutorial 1: Training a Drug-Target Interaction Model from Scratch [@KexinHuang5](https://twitter.com/KexinHuang5)In this tutorial, we take a deep dive into DeepPurpose and show how it builds a drug-target interaction model from scratch. Agenda:- Part I: Overview of DeepPurpose and Data- Part II: Drug Target Interaction Prediction    - DeepPurpose Framework    - Applications to Drug Repurposing and Virtual Screening    - Pretrained Models    - Hyperparameter Tuning    - Model Robustness EvaluationLet's start!<jupyter_code>from DeepPurpose import utils, dataset
from DeepPurpose import DTI as models
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Part I: Overview of DeepPurpose and DataDrug-target interaction measures the binding of drug molecules to the protein targets. Accurate identification of DTI is fundamental for drug discovery and supports many downstream tasks. Among others, drug screening and repurposing are two main applications based on DTI. Drug screening helps identify ligand candidates that can bind to the protein of interest, whereas drug repurposing finds new therapeutic purposes for existing drugs. Both tasks could alleviate the costly, time-consuming, and labor-intensive process of synthesis and analysis, which is extremely important, especially in the cases of hunting effective and safe treatments for COVID-19.DeepPurpose is a pytorch-based deep learning framework that is initiated to provide a simple but powerful toolkit for drug-target interaction prediction and its related applications. We see many exciting recent works in this direction, but to leverage these models, it takes lots of efforts due to the esoteric instructions and interface. DeepPurpose is designed to make things as simple as possible using a unified framework.DeepPurpose uses an encoder-decoder framework. Drug repurposing and screening are two applications after we obtain DTI models. The input to the model is a drug target pair, where drug uses the simplified molecular-input line-entry system (SMILES) string and target uses the amino acid sequence. The output is a score indicating the binding activity of the drug target pair. Now, we begin talking about the data format expected.(**Data**) DeepPurpose takes into an array of drug's SMILES strings (**d**), an array of target protein's amino acid sequence (**t**), and an array of label (**y**), which can either be binary 0/1 indicating interaction outcome or a real number indicating affinity value. The input drug and target arrays should be paired, i.e. **y**\[0\] is the score for **d**\[0\] and **t**\[0\].Besides transforming into numpy arrays through some data wrangling on your own, DeepPurpose also provides two ways to help data preparation. The first way is to read from local files. For example, to load drug target pairs, we expect a file.txt where each line is a drug SMILES string, followed by a protein sequence, and an affinity score or 0/1 label:```CC1=C...C4)N MKK...LIDL 7.365``` \```CC1=C...C4)N QQP...EGKH 4.999```Then, we use ```dataset.read_file_training_dataset_drug_target_pairs``` to load it.<jupyter_code>X_drugs, X_targets, y = dataset.read_file_training_dataset_drug_target_pairs('./toy_data/dti.txt')
print('Drug 1: ' + X_drugs[0])
print('Target 1: ' + X_targets[0])
print('Score 1: ' + str(y[0]))<jupyter_output>Drug 1: CC1=C2C=C(C=CC2=NN1)C3=CC(=CN=C3)OCC(CC4=CC=CC=C4)N
Target 1: MKKFFDSRREQGGSGLGSGSSGGGGSTSGLGSGYIGRVFGIGRQQVTVDEVLAEGGFAIVFLVRTSNGMKCALKRMFVNNEHDLQVCKREIQIMRDLSGHKNIVGYIDSSINNVSSGDVWEVLILMDFCRGGQVVNLMNQRLQTGFTENEVLQIFCDTCEAVARLHQCKTPIIHRDLKVENILLHDRGHYVLCDFGSATNKFQNPQTEGVNAVEDEIKKYTTLSYRAPEMVNLYSGKIITTKADIWALGCLLYKLCYFTLPFGESQVAICDGNFTIPDNSRYSQDM<KEY>[...]<jupyter_text>Many method researchers want to test on benchmark datasets such as KIBA/DAVIS/BindingDB, DeepPurpose also provides data loaders to ease preprocessing. For example, we want to load the DAVIS dataset, we can use ```dataset.load_process_DAVIS```. It will download, preprocess to the designated data format. It supports label log-scale transformation for easier regression and also allows label binarization given a customized threshold.<jupyter_code>X_drugs, X_targets, y = dataset.load_process_DAVIS(path = './data', binary = False, convert_to_log = True, threshold = 30)
print('Drug 1: ' + X_drugs[0])
print('Target 1: ' + X_targets[0])
print('Score 1: ' + str(y[0]))<jupyter_output>Beginning Processing...
Beginning to extract zip file...
Default set to logspace (nM -> p) for easier regression
Done!
Drug 1: CC1=C2C=C(C=CC2=NN1)C3=CC(=CN=C3)OCC(CC4=CC=CC=C4)N
Target 1: MKKFFDSRREQGGSGLGSGSSGGGGSTSGLGSGYIGRVFGIGRQQVTVDEVLAEGGFAIVFLVRTSNGMKCALKRMFVNNEHDLQVCKREIQIMRDLSGHKNIVGYIDSSINNVSSGDVWEVLILMDFCRGGQVVNLMNQRLQTGFTENEVLQIFCDTCEAVARLHQCKTPIIHRDLKVENILLHDRGHYVLCDFGSATNKFQNPQTEGVNAVEDEIKKYTTLSYRAPEMVNLYSGKIITTKADIWALGCLLYKLCYFTLPFGESQVAICDGNFTIPDNSRYSQDMHCLIRYMLEPDPDKRPDIYQVSYFSFKLLKKECPIPNVQNSPIPAKLPEPVKASEAAAKKTQPKARLTDPIPTTETSIAPRQRPKAGQTQPNPGILPIQPALTPRKRATVQPPPQAAGSSNQPGLLASVPQPKPQAPPSQPLPQTQAKQPQAPPTPQQTPSTQAQGLPAQAQATPQHQQQLFLKQQQQQQQPPPAQQQPAGTFYQQQQAQTQQFQAVHPATQKPAIAQFPVVSQGGSQQQLMQNFYQQQQQQQQQQQQQQLATALHQQQLMTQQAALQQKPTMAAGQQPQPQPAAAPQPAPAQEPAIQAPVRQQPKVQTTPPPAVQGQKVGSLTPPSSPKTQRAGHRRILSDVTHSAVFGVPASKSTQLLQAAAAEASLNKSKSATTTPSGSPRTSQQNVYNPSEGSTWNPFDDDNFSKLTAEELLNKDFAKLGEGKHPEKLGGSAESLIPGFQSTQGDAFATTSFSAGTAEKRKGGQTVDSGLPLLSVSDPFIPLQVPDAPEKLIEGLKSPDTSLLLPDLLPMT[...]<jupyter_text>For more detailed examples and tutorials of data loading, checkout this [tutorial](./DEMO/load_data_tutorial.ipynb). Part II: Drug Target Interaction Prediction FrameworkDeepPurpose provides a simple framework to conduct DTI research using 8 encoders for drugs and 7 for proteins. It basically consists of the following steps, where each step corresponds to one line of code:- Encoder specification- Data encoding and split- Model configuration generation- Model initialization- Model Training- Model Prediction and Repuposing/Screening- Model Saving and LoadingLet's start with data encoding! (**Encoder specification**) After we obtain the required data format from Part I, we need to prepare them for the encoders. Hence, we first specify the encoder to use for drug and protein. Here we try MPNN for drug and CNN for target.If you find MPNN and CNN are too large for the CPUs, you can try smaller encoders by uncommenting the last line:<jupyter_code>drug_encoding, target_encoding = 'MPNN', 'CNN'
#drug_encoding, target_encoding = 'Morgan', 'Conjoint_triad'<jupyter_output><empty_output><jupyter_text>Note that you can switch encoder just by changing the encoding name above. The full list of encoders are listed [here](https://github.com/kexinhuang12345/DeepPurposeencodings). Here, we are using the message passing neural network encoder for drug and convolutional neural network encoder for protein.(**Data encoding and split**) Now, we encode the data into the specified format, using ```utils.data_process``` function. It specifies train/validation/test split fractions, and random seed to ensure same data splits for reproducibility. This function also support data splitting methods such as ```cold_drug``` and ```cold_protein```, which splits on drug/proteins for model robustness evaluation to test on unseen drug/proteins.The function outputs train, val, test pandas dataframes.<jupyter_code>train, val, test = utils.data_process(X_drugs, X_targets, y, 
                                drug_encoding, target_encoding, 
                                split_method='random',frac=[0.7,0.1,0.2],
                                random_seed = 1)
train.head(1)<jupyter_output>in total: 30056 drug-target pairs
encoding drug...
unique drugs: 68
drug encoding finished...
encoding protein...
unique target sequence: 379
protein encoding finished...
splitting dataset...
Done.<jupyter_text>(**Model configuration generation**) Now, we initialize a model with its configuration. You can modify almost any hyper-parameters (e.g., learning rate, epoch, batch size), model parameters (e.g. hidden dimensions, filter size) and etc in this function. The supported configurations are listed here in this [link](https://github.com/kexinhuang12345/DeepPurpose/blob/e169e2f550694145077bb2af95a4031abe400a77/DeepPurpose/utils.pyL486).For the sake of example, we specify the epoch size to be 3, and set the model parameters to be small so that you can run on both CPUs & GPUs quickly and can proceed to the next steps. For a reference parameters, checkout the notebooks in the DEMO folder.<jupyter_code>config = utils.generate_config(drug_encoding = drug_encoding, 
                         target_encoding = target_encoding, 
                         cls_hidden_dims = [1024,1024,512], 
                         train_epoch = 5, 
                         LR = 0.001, 
                         batch_size = 128,
                         hidden_dim_drug = 128,
                         mpnn_hidden_size = 128,
                         mpnn_depth = 3, 
                         cnn_target_filters = [32,64,96],
                         cnn_target_kernels = [4,8,12]
                        )<jupyter_output><empty_output><jupyter_text>(**Model initialization**) Next, we initialize a model using the above configuration.<jupyter_code>model = models.model_initialize(**config)
model<jupyter_output><empty_output><jupyter_text>(**Model Training**) Next, it is ready to train, using the ```model.train``` function! If you do not have test set, you can just use ```model.train(train, val)```.<jupyter_code>model.train(train, val, test)<jupyter_output>Let's use 1 GPU!
--- Data Preparation ---
--- Go for Training ---
Training at Epoch 1 iteration 0 with loss 30.3964. Total time 0.0 hours
Training at Epoch 1 iteration 100 with loss 0.59583. Total time 0.01277 hours
Validation at Epoch 1 , MSE: 0.79965 , Pearson Correlation: 0.34392 with p-value: 3.23317 , Concordance Index: 0.67805
Training at Epoch 2 iteration 0 with loss 0.62158. Total time 0.02305 hours
Training at Epoch 2 iteration 100 with loss 0.83107. Total time 0.03583 hours
Validation at Epoch 2 , MSE: 0.65400 , Pearson Correlation: 0.46281 with p-value: 1.79746 , Concordance Index: 0.74763
Training at Epoch 3 iteration 0 with loss 0.68138. Total time 0.04611 hours
Training at Epoch 3 iteration 100 with loss 0.73255. Total time 0.05888 hours
Validation at Epoch 3 , MSE: 0.57770 , Pearson Correlation: 0.52944 with p-value: 7.37415 , Concordance Index: 0.77889
Training at Epoch 4 iteration 0 with loss 0.46632. Total time 0.06916 hours
Training at Epoch 4 iteration 100 with loss[...]<jupyter_text>We see that the model will automatically generate and plot the training process, along with the validation result and test result.(**Model Prediction and Repuposing/Screening**) Next, we see how we can predict affinity scores on new data. Suppose the new data is a drug-target pair below.<jupyter_code>X_drug = ['CC1=C2C=C(C=CC2=NN1)C3=CC(=CN=C3)OCC(CC4=CC=CC=C4)N']
X_target = ['MK<KEY>GSTSDAVIEKADVAVESLIPGLEPPVPQRLPSQTESVTSNRTDSLTGEDSLLDCSLLSNPTTDLLEEFAPTAISAPVHKAAEDSNLISGFDVPEGSDKVAEDEFDPIPVLITKNPQGGHSRNSSGSSESSLPNLARSLLLVDQLIDL']
y = [7.365]
X_pred = utils.data_process(X_drug, X_target, y, 
                                drug_encoding, target_encoding, 
                                split_method='no_split')
y_pred = model.predict(X_pred)
print('The predicted score is ' + str(y_pred))<jupyter_output>in total: 1 drug-target pairs
encoding drug...
unique drugs: 1
drug encoding finished...
encoding protein...
unique target sequence: 1
protein encoding finished...
splitting dataset...
do not do train/test split on the data for already splitted data
predicting...
The predicted score is [5.243870735168457]<jupyter_text>We can also do repurposing and screening using the trained model. Basically, for repurposing a set of existing drugs (**r**) for a single new target (*t*), we run the above prediction function after pairing each repurposing drug with the target. Similarly, for screening, we instead have a set of drug-target pairs (**d**, **t**). We wrap the operation into a ```models.repurpose``` and ```models.virtual_screening``` methods.For example, suppose we want to do repurposing from a set of antiviral drugs for a COVID-19 target 3CL protease. The corresponding data can be retrieved using ```dataset``` functions.<jupyter_code>t, t_name = dataset.load_SARS_CoV2_Protease_3CL()
print('Target Name: ' + t_name)
print('Amino Acid Sequence: '+ t)
r, r_name, r_pubchem_cid = dataset.load_antiviral_drugs()
print('Repurposing Drug 1 Name: ' + r_name[0])
print('Repurposing Drug 1 SMILES: ' + r[0])
print('Repurposing Drug 1 Pubchem CID: ' + str(r_pubchem_cid[0]))<jupyter_output>Repurposing Drug 1 Name: Abacavir
Repurposing Drug 1 SMILES: C1CC1NC2=C3C(=NC(=N2)N)N(C=N3)C4CC(C=C4)CO
Repurposing Drug 1 Pubchem CID: 441300<jupyter_text>Now, we can call the ```repurpose``` function. After feeding the necessary inputs, it will print a list of repurposed drugs ranked on its affinity to the target protein. The ```convert_y``` parameter should be set to be ```False``` when the ranking is ascending (i.e. lower value -> higher affinity) due to the log transformation, vice versus.<jupyter_code>y_pred = models.repurpose(X_repurpose = r, target = t, model = model, drug_names = r_name, target_name = t_name, 
                          result_folder = "./result/", convert_y = True)<jupyter_output>repurposing...
in total: 82 drug-target pairs
encoding drug...
unique drugs: 81
drug encoding finished...
encoding protein...
unique target sequence: 1
protein encoding finished...
Done.
predicting...
---------------
Drug Repurposing Result for SARS-CoV2 3CL Protease
+------+----------------------+------------------------+---------------+
| Rank |      Drug Name       |      Target Name       | Binding Score |
+------+----------------------+------------------------+---------------+
|  1   |      Foscarnet       | SARS-CoV2 3CL Protease |      0.14     |
|  2   |      Efavirenz       | SARS-CoV2 3CL Protease |     20.86     |
|  3   |     Rimantadine      | SARS-CoV2 3CL Protease |     34.61     |
|  4   |      Sofosbuvir      | SARS-CoV2 3CL Protease |     81.58     |
|  5   |     Glecaprevir      | SARS-CoV2 3CL Protease |     83.02     |
|  6   |      Remdesivir      | SARS-CoV2 3CL Protease |     89.09     |
|  7   |     Grazoprevir      | SARS-CoV2 3CL Protease |     229.87    |
| [...]<jupyter_text>Now, let's move on to showcase how to do virtual screening. We first load a sample of data from BindingDB dataset.<jupyter_code>t, d = dataset.load_IC50_1000_Samples()<jupyter_output>Downloading...<jupyter_text>We can then use the ```virtual_screening``` function to generate a list of drug-target pairs that have high binding affinities. If no drug/target names are provided, the index of the drug/target list is used instead.<jupyter_code>y_pred = models.virtual_screening(d, t, model)<jupyter_output>virtual screening...
in total: 100 drug-target pairs
encoding drug...
unique drugs: 100
drug encoding finished...
encoding protein...
unique target sequence: 93
protein encoding finished...
Done.
predicting...
---------------
Virtual Screening Result
+------+-----------+-------------+---------------+
| Rank | Drug Name | Target Name | Binding Score |
+------+-----------+-------------+---------------+
|  1   |  Drug 83  |  Target 83  |      8.87     |
|  2   |  Drug 53  |  Target 53  |      7.71     |
|  3   |  Drug 60  |  Target 60  |      7.01     |
|  4   |  Drug 41  |  Target 41  |      6.73     |
|  5   |  Drug 93  |  Target 93  |      6.59     |
|  6   |  Drug 32  |  Target 32  |      6.49     |
|  7   |   Drug 0  |   Target 0  |      6.43     |
|  8   |  Drug 64  |  Target 64  |      6.27     |
|  9   |  Drug 23  |  Target 23  |      6.26     |
|  10  |  Drug 21  |  Target 21  |      6.23     |
checkout ./result/virtual_screening.txt for the whole list<jupyter_text>Saving and loading models are also really easy. The loading function also automatically detects if the model is trained on multiple GPUs. To save a model:<jupyter_code>model.save_model('./tutorial_model')<jupyter_output><empty_output><jupyter_text>To load a saved/pretrained model:<jupyter_code>model = models.model_pretrained(path_dir = './tutorial_model')
model<jupyter_output><empty_output><jupyter_text>We have also provided a list of pretrained model, you can find all available ones under the [list](https://github.com/kexinhuang12345/DeepPurposepretrained-models). For example, to load a MPNN+CNN model pretrained on BindingDB Kd dataset:<jupyter_code>model = models.model_pretrained(model = 'MPNN_CNN_BindingDB')
model<jupyter_output>Beginning Downloading MPNN_CNN_BindingDB Model...
Downloading finished... Beginning to extract zip file...
pretrained model Successfully Downloaded...<jupyter_text>We also provided many more functionalities for DTI research purposes. For example, this [demo](https://github.com/kexinhuang12345/DeepPurpose/blob/master/DEMO/Drug_Property_Pred-Ax-Hyperparam-Tune.ipynb) shows how to use Ax platform to do some latest hyperparameter tuning methods such as Bayesian Optimization on DeepPurpose.Model robustness is very important for DTI task. One way to measure is to see how the model can predict drug or protein that do not exist in the training set, i.e., cold drug/target setting. You can achieve this by modifying the ```split_method``` parameter in the ```data_process``` function:<jupyter_code>X_drugs, X_targets, y = dataset.load_process_DAVIS(path = './data', binary = False, convert_to_log = True, threshold = 30)
train, val, test = utils.data_process(X_drugs, X_targets, y, 
                                drug_encoding, target_encoding, 
                                split_method='cold_drug',frac=[0.7,0.1,0.2],
                                random_seed = 1)<jupyter_output>Beginning Processing...
Beginning to extract zip file...
Default set to logspace (nM -> p) for easier regression
Done!
in total: 30056 drug-target pairs
encoding drug...
unique drugs: 68
drug encoding finished...
encoding protein...
unique target sequence: 379
protein encoding finished...
splitting dataset...
Done.
{'fraction_non_alphanumeric': 0.09465539947322213, 'fraction_numerical': 0.031442054433713784, 'mean_word_length': 3.907108239095315, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 63, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '37224', 'license': 'MIT', 'n_tokens_mistral': 6189, 'n_tokens_neox': 5484, 'n_words': 1901, 'path': 'Tutorial_1_DTI_Prediction.ipynb', 'repo_name': 'williamrobotma/DTI', 'size': '19670'}