<jupyter_start><jupyter_text>Predict ligand profiling using `kissim` using the Karaman datasetIn order to assess the predictive power of `kissim`, we here choose a ligand-centric evaluation. We will compare if `kissim` can predict on- and off-targets determined in ligand profiling studies.- Kinase-kinase distance dataset (use KinMap kinase names): Select kinases from profiling dataset by query ligand- Kinase-ligand profiling dataset (use KinMap kinase names and PKIDB ligand names): Select kinases from distances dataset by the ligand's on-target- Merge both datasets and keep only kinases that have measurements in both datasets- Rank kinases by distances- Calculuate enrichment factors and enrichment plots- Calculate ROC curves<jupyter_code>%load_ext autoreload
%autoreload 2
import pandas as pd
import matplotlib.pyplot as plt

from src import data, evaluation
plt.style.use("seaborn")<jupyter_output><empty_output><jupyter_text>Define parameters<jupyter_code># The only global that can be changed in this notebook
LIGAND_PROFILING_METHOD = "karaman"
KINASE_DISTANCES_METHOD = "kissim"
KINASE_ACTIVITY_CUTOFF = 100
KINASE_ACTIVITY_MAX = True
MIN_N_SHARED_KINASES = 10
MIN_N_SHARED_ACTIVE_KINASES = 3<jupyter_output><empty_output><jupyter_text>Kinase-kinase distances dataset__Criteria__: - Filters out kinases whose names cannot be mapped to KinMap kinase names.<jupyter_code># Full dataset (reported only, not further used)
kissim_df = data.distances.kissim(kinmap_kinases=False)
print(f"Kinase distance matrix shape: {kissim_df.shape}")
# Reduced dataset
kissim_df = data.distances.kissim(kinmap_kinases=True)
print(f"Kinase distance matrix shape: {kissim_df.shape}")
kissim_df.iloc[:5, :5]<jupyter_output>Changed kinase names (unknown names may be discarded - see function docstring):
    kinase.input            kinase.kinmap
68         DCLK1                  DCAMKL1
120       JAK1-b                   JAK1_b
156       MSK1-b                   MSK1_b
175       PI4K2A  unknown (not in KinMap)
176        PI4KB  unknown (not in KinMap)
177       PIK3C3  unknown (not in KinMap)
206       RSK2-b                   RSK2_b
207       RSK3-b                   RSK3_b
232       TYK2-b                   TYK2_b
249        p110a  unknown (not in KinMap)
250        p110d  unknown (not in KinMap)
251        p110g  unknown (not in KinMap)<jupyter_text>Kinase-ligand profiling dataset__Criteria__:- Load dataset with FDA-approved PKIDB ligands only.- Filters out ligands whose names cannot be mapped onto PKIDB ligand names.<jupyter_code># Full dataset (reported only, not further used)
profiling_df = data.profiling.load(LIGAND_PROFILING_METHOD)
print(f"Number of kinases (full dataset): {profiling_df.shape[0]}")
print(f"Number of ligands (full dataset): {profiling_df.shape[1]}")
# Reduced dataset
profiling_df = data.profiling.load(LIGAND_PROFILING_METHOD, pkidb_ligands=True, fda_approved=True)
print(f"Number of kinases (FDA-approved PKIDB ligands only): {profiling_df.shape[0]}")
print(f"Number of ligands (FDA-approved PKIDB ligands only): {profiling_df.shape[1]}")
profiling_df.head()
profiling_df.count().sort_values().plot(kind="barh");<jupyter_output><empty_output><jupyter_text>Ligands' on-targets dataset__Criteria__: - Filters out ligands that are not available in the PKIDB or have no targets in the PKIDB.- Filters out targets whose names cannot be mapped onto KinMap names<jupyter_code>pd.set_option("max_colwidth", 100)
ligand_names = profiling_df.columns
ligand_targets = data.targets.pkidb(ligand_names, fda_approved=False)
ligand_targets<jupyter_output>Changed kinase names (unknown names may be discarded - see function docstring):
   kinase.input            kinase.kinmap
0           nan  unknown (not in KinMap)
3         EPHA2                    EphA2
5          YES1                      YES
7        PDGFRB                   PDGFRb
17        NTRK1                     TRKA
18        CSF1R                      FMS
19       PDGFRA                   PDGFRa
22       PDGFRB                   PDGFRb
24        ERBB2                    ErbB2
30       PDGFRB                   PDGFRb
35       MAPK14                     p38a
36        VEGFA  unknown (not in KinMap)
38         PTK6                      BRK
39          TEK                     TIE2
41       PDGFRB                   PDGFRb
47        CSF1R                      FMS
48       PDGFRA                   PDGFRa
50        NR1I2  unknown (not in KinMap)
52        PDGFD  unknown (not in KinMap)
53          nan  unknown (not in KinMap)
56        ERBB2                    ErbB2<jupyter_text>Ligand-kinase datasetsWe merge now profiling and kinase similarity information into one dataset per ligand-kinase pair (_ligand-kinase dataset_). We only keep ligand-kinase pairs that- have a minimum of `MIN_N_SHARED_KINASES` shared kinases- have a minimum of `MIN_N_SHARED_ACTIVE_KINASES` shared active kinases, while activity is defined <= `KINASE_ACTIVITY_CUTOFF` Select ligand-kinase pairs with enough coverage.<jupyter_code>ligand_kinase_pairs = (
    ligand_targets[["ligand.input", "targets.kinmap"]]
    .explode("targets.kinmap")
    .to_numpy()
    .tolist()
)<jupyter_output><empty_output><jupyter_text>Evaluation metrics<jupyter_code>evaluator = evaluation.LigandVsKinaseEvaluator(
    ligand_kinase_pairs,
    LIGAND_PROFILING_METHOD,
    KINASE_DISTANCES_METHOD,
    KINASE_ACTIVITY_CUTOFF,
    KINASE_ACTIVITY_MAX,
    MIN_N_SHARED_KINASES,
    MIN_N_SHARED_ACTIVE_KINASES,
)
pkidb = data.ligands.pkidb()
ligands = pkidb.set_index("ID")["Type"][list(evaluator.data_dict.keys())].reset_index()
ligands<jupyter_output><empty_output><jupyter_text>Enrichment Enrichment factor An enrichment factor is calculated as follows:$EF_{x\%} = \frac{\frac{n_s}{n}}{\frac{N_s}{N}}$with- $N$   : Number of kinases in total- $N_s$ : Number of kinases in top x% of ranked kinases- $n$   : Number of active kinases in total- $n_s$ : Number of active kinases in top x% of ranked kinases<jupyter_code>evaluator.plot_enrichment_factors()<jupyter_output><empty_output><jupyter_text>Enrichment plotsFor an enrichment plot, the x and y values are calculated as follows:$x = \frac{N_s}{N}$, $y = \frac{n_s}{n}$with- $N$   : Number of kinases in total- $N_s$ : Number of kinases in top x% of ranked kinases- $n$   : Number of active kinases in total- $n_s$ : Number of active kinases in top x% of ranked kinases<jupyter_code>evaluator.plot_enrichment()<jupyter_output><empty_output><jupyter_text>ROC curves- $TPR = \frac{TP}{P} = \frac{TP}{TP + FN}$- $FPR = \frac{FP}{N} = \frac{FP}{FP + TN}$<jupyter_code>evaluator.plot_roc_curves();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07924528301886792, 'fraction_numerical': 0.020075471698113207, 'mean_word_length': 3.417333333333333, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '439740', 'license': 'MIT', 'n_tokens_mistral': 2344, 'n_tokens_neox': 2121, 'n_words': 611, 'path': 'notebooks/006_evaluation/002_profiling_karaman.ipynb', 'repo_name': 'volkamerlab/kissim_app', 'size': '6625'}