<jupyter_start><jupyter_text>Preprocess TCGA mutation dataAll this does is save the sample info for TCGA somatic mutation data, which is necessary to use mutation information as a predictor (e.g. in survival/prognosis prediction experiments).This data was already preprocessed in http://github.com/greenelab/pancancer and is loaded directly from that repo, so no further preprocessing is necessary.<jupyter_code>import os

import pandas as pd

import mpmp.config as cfg
import mpmp.utilities.data_utilities as du
import mpmp.utilities.tcga_utilities as tu<jupyter_output><empty_output><jupyter_text>Read TCGA Barcode Curation InformationExtract information from TCGA barcodes - `cancer-type` and `sample-type`. See https://github.com/cognoma/cancer-data for more details<jupyter_code>(cancer_types_df,
 cancertype_codes_dict,
 sample_types_df,
 sampletype_codes_dict) = tu.get_tcga_barcode_info()
cancer_types_df.head(2)
sample_types_df.head(2)<jupyter_output><empty_output><jupyter_text>Load and process somatic mutation data<jupyter_code>pancan_data = du.load_pancancer_data(verbose=True)
sample_freeze_df = pancan_data[0]

print(sample_freeze_df.duplicated(['SAMPLE_BARCODE']).sum())
assert (
    sample_freeze_df.duplicated(['SAMPLE_BARCODE']).sum() == 0
)

sample_freeze_df.set_index('SAMPLE_BARCODE', inplace=True)
sample_freeze_df.index.rename('sample_id', inplace=True)
sample_freeze_df.head()<jupyter_output>Loading pan-cancer data from cached pickle file...<jupyter_text>Process TCGA cancer type and sample type info from barcodesSee https://gdc.cancer.gov/resources-tcga-users/tcga-code-tables/tissue-source-site-codes for more details.<jupyter_code># get sample info and save to file

tcga_id = tu.get_and_save_sample_info(sample_freeze_df,
                                      sampletype_codes_dict,
                                      cancertype_codes_dict,
                                      training_data='mutation')

print(tcga_id.shape)
tcga_id.head()
# get cancer type counts and save to file
cancertype_count_df = (
    pd.DataFrame(tcga_id.cancer_type.value_counts())
    .reset_index()
    .rename({'index': 'cancertype', 'cancer_type': 'n ='}, axis='columns')
)

file = os.path.join(cfg.sample_info_dir, 'tcga_mutation_sample_counts.tsv')
cancertype_count_df.to_csv(file, sep='\t', index=False)

cancertype_count_df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09617807643847123, 'fraction_numerical': 0.0016799664006719867, 'mean_word_length': 5.139175257731959, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '209027', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 817, 'n_tokens_neox': 779, 'n_words': 162, 'path': '00_download_data/1G_preprocess_mutations.ipynb', 'repo_name': 'greenelab/mpmp', 'size': '2381'}