<jupyter_start><jupyter_text>Cross-cancer prediction analysis<jupyter_code>import os

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib_venn import venn2
import seaborn as sns

import pancancer_evaluation.config as cfg
import pancancer_evaluation.utilities.analysis_utilities as au<jupyter_output><empty_output><jupyter_text>First, we want to see what happens when we train on a single gene in a single cancer type, and test on other single genes/cancer types. We expect to see good performance on the same gene/cancer type (since we're training and testing on the same data), and we're also hoping to see decent performance on the same gene in different cancer types.<jupyter_code># cross_cancer_dir = os.path.join(cfg.results_dir, 'cross_cancer', 'cross_cancer')
cross_cancer_dir = os.path.join(cfg.repo_root, 'cross_cancer_random_results', 'cross_cancer_2')
cross_cancer_df = au.load_prediction_results_cc(cross_cancer_dir, 'cross_cancer')
print(cross_cancer_df.shape)
cross_cancer_df.head()
# get rows that have the same identifier
cross_cancer_df['train_test_equal'] = (cross_cancer_df.train_identifier ==
                                       cross_cancer_df.test_identifier)
# get rows that have the same gene (possibly different cancer types)
cross_cancer_df['gene_equal'] = (cross_cancer_df.train_identifier.str.split('_', expand=True)[0] ==
                                 cross_cancer_df.test_identifier.str.split('_', expand=True)[0])
sns.set({'figure.figsize': (16, 6)})
fig, axarr = plt.subplots(1, 2)

sns.histplot(data=cross_cancer_df[(cross_cancer_df.signal == 'signal') &
                                  (cross_cancer_df.data_type == 'test')],
             x='aupr', stat='count', hue='train_test_equal', element='step',
             kde=False, ax=axarr[0])
axarr[0].set_xlabel('AUPRC')
axarr[0].set_ylabel('Count')
axarr[0].set_title('Cross-cancer performance, same train/test data highlighted')
sns.histplot(data=cross_cancer_df[(cross_cancer_df.signal == 'signal') &
                                  (cross_cancer_df.data_type == 'test') &
                                  (~cross_cancer_df.train_test_equal)],
             x='aupr', stat='count', hue='gene_equal', element='step',
             kde=False, ax=axarr[1])
axarr[1].set_xlabel('AUPRC')
axarr[1].set_ylabel('Count')
axarr[1].set_title('Cross-cancer performance, same train/test gene highlighted')
count_unique_ids = np.unique(cross_cancer_df.train_identifier).shape[0]
print(count_unique_ids)
heatmap_df, sorted_ids = au.heatmap_from_results(cross_cancer_df, normalize_control=True,
                                                 sort_results=False)
heatmap_df.iloc[:5, :5]
sns.set({'figure.figsize': (15, 8)})
sns.heatmap(heatmap_df, cbar_kws={'label': 'AUPR difference from baseline'}, center=0)
plt.title('Cross-cancer mutation detection, AUPR heatmap')
plt.xlabel('Test identifier')
plt.ylabel('Train identifier')<jupyter_output><empty_output><jupyter_text>We can see that most of the off-diagonal is close to 0 (no improvement against negative control), with the possible exception of TP53 whose classifiers tend to perform well across cancer types. This isn't what we were hoping for (we were hoping to identify new relationships between genes in certain cancer types, but it doesn't look like we're able to do that here, at least for these genes).Next, we want to try training pan-cancer models for a given gene on all but one cancer type, then testing on the held out cancer type, either on the same gene or different genes. The idea is that the pan-cancer models will benefit from additional data, but combining multiple cancers may also wash out cancer type-specific signals.<jupyter_code>pan_cross_cancer_dir = os.path.join(cfg.results_dir, 'pan_cross_cancer', 'pan_cross_cancer')
pan_cross_cancer_df = au.load_prediction_results_cc(pan_cross_cancer_dir, 'pan_cross_cancer')
print(pan_cross_cancer_df.shape)
pan_cross_cancer_df.head()
pan_cross_cancer_df['gene_equal'] = (pan_cross_cancer_df.train_gene ==
                                     pan_cross_cancer_df.test_identifier.str.split('_', expand=True)[0])

sns.set({'figure.figsize': (8, 6)})
sns.histplot(data=pan_cross_cancer_df[(pan_cross_cancer_df.signal == 'signal') &
                                      (pan_cross_cancer_df.data_type == 'test')],
             x='aupr', stat='count', hue='gene_equal', element='step', kde=False)
plt.xlabel('AUPRC')
plt.ylabel('Count')
plt.title('Train pancancer/test cross-cancer, same train/test gene highlighted')
# filter cross-cancer data and make a deep copy
# (this avoids SettingWithCopyError)
heatmap_df, _ = au.heatmap_from_results(pan_cross_cancer_df,
                                        train_pancancer=True,
                                        normalize_control=True,
                                        sorted_ids=sorted_ids)
heatmap_df.iloc[:5, :5]
sns.set({'figure.figsize': (12, 5)})
sns.heatmap(heatmap_df, cbar_kws={'label': 'AUPR difference from baseline'}, center=0)
plt.title('Train pan-cancer/test cross-cancer mutation detection, AUPR heatmap')
plt.xlabel('Test identifier')
plt.ylabel('Train identifier')<jupyter_output><empty_output><jupyter_text>Compared to the last plot, we can see slightly more off-diagonal signal here, but not much. Again, pan-cancer TP53 classifiers seem to perform the best on TP53, but there are a few other interesting examples (KRAS classifier performs well on some BRAF examples, BRAF classifier performs well on NF1/LGG, etc).<jupyter_code># now do this with only Ras/Raf genes in all cancer types
plot_gene_list = ['KRAS', 'HRAS', 'NRAS', 'BRAF', 'TTN']

pan_cross_cancer_dir = os.path.join(cfg.repo_root, 'pan_cross_cancer_results', 'pan_cross_cancer')
pan_cross_cancer_df = au.load_prediction_results_cc(pan_cross_cancer_dir, 'pan_cross_cancer')
print(pan_cross_cancer_df.shape)
pan_cross_cancer_df.head()
# filter cross-cancer data and make a deep copy
# (this avoids SettingWithCopyError)
heatmap_df, _ = au.heatmap_from_results(pan_cross_cancer_df,
                                        plot_gene_list=plot_gene_list,
                                        train_pancancer=True,
                                        normalize_control=True,
                                        sorted_ids=sorted_ids)
print(heatmap_df.shape)
heatmap_df.iloc[:5, :5]
sns.set({'figure.figsize': (12, 4)})
sns.heatmap(heatmap_df, cbar_kws={'label': 'AUPR difference from baseline'}, center=0)
plt.title('Train pan-cancer/test cross-cancer mutation detection, AUPR heatmap')
plt.xlabel('Test identifier')
plt.ylabel('Train identifier')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1040384326677676, 'fraction_numerical': 0.007055997597958264, 'mean_word_length': 4.027924528301887, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '282998', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2177, 'n_tokens_neox': 2003, 'n_words': 516, 'path': '03_cross_cancer_classification/plot_cross_cancer_results.ipynb', 'repo_name': 'greenelab/pancancer-evaluation', 'size': '6661'}