<jupyter_start><jupyter_text>Loading Required LibrariesFor this experiment we are employing the following libraries:* **util** to download raw data sets;* **collections** to compute some group metrics like count, sum, and mean;* **zipfile** to unzip raw data sets;* **pandas**  to read, process and visualize tabular data; * **numpy** no some matrix and algebric operations;* **os** to file system search; and<jupyter_code>from util import download_file_from_google_drive as dffgd
from collections import Counter
from zipfile import ZipFile

import pandas as pd
import numpy as np
import os<jupyter_output><empty_output><jupyter_text>Downloading Required Data SetsAll collect data are from the study CoMMpass℠ (Relating Clinical Outcomes in MM to Personal Assessment of Genetic Profile) organized by the Multiple Myeloma Research Foundation (MMRF) avaliable at https://research.themmrf.org. For convenience, we saved our data set into Google Drive. If they do not exist in our local folder, the process below download that automatically.<jupyter_code>BASE_PATH = "data/mmrf"

#
# Downloading files is they do not exist
#
if not os.path.exists(BASE_PATH):
    
    if not os.path.exists('data'):
        os.mkdir('data')
    
    if not os.path.exists('data/gene_fpkm.txt'):
        dffgd('1zou_wHWzDCy488-cqbIzgSLRdNykEEZx', 'data/gene_data.zip')

    dffgd('1tDDV75TEriiL_JwJ8aSquAj-N4eJrf1J', 'data/mmrf.zip')
    
    for file_path in ['data/mmrf.zip', 'data/gene_data.zip']:
        
        with ZipFile(file_path, 'r') as zipObj:
            #
            # Extract all the contents of zip file in current directory
            #
            zipObj.extractall('data')
        
        os.remove(file_path)<jupyter_output><empty_output><jupyter_text>Loading and Merging Data SetsThe script merge all collected data into a unique pandas data frame keeping only patients with related to the following threatment classes:* IMIDs-based* Bortezomib-based* Carfilzomib-based* Combined bortezomib/IMIDs-based* Combined IMIDs/carfilzomib-basedWe exclude observations associated to missing *therapy_first_line* and *iss* variables, which describe the first therapy applied to the Multiple Myeloma (MM) patient, and the cance stange, respectively. We remove redundant or leakage variables:* progression_free_survivel_status* therapy_first_line_most_common* therapy_first_line_starting_treatment* days_to_overall_survival* disease_status* overall_survival_status<jupyter_code>#
# Starting data processing
#
df = None

for root, directories, files in os.walk(BASE_PATH):
    for file in files:
        if '.tsv' in file:
            
            tmp = pd.read_csv(os.path.join(root, file), sep='\t', index_col='ID')
            
            tmp = tmp[~tmp.index.duplicated(keep='first')]
            
            if tmp.shape[0] > 400:
            
                try:
                    df  = tmp if df is None else df.join(tmp, how='outer')
                except Exception as e:
                    pass

for c in ['progression_free_survivel_status', 'therapy_first_line_most_common', 
          'therapy_first_line_starting_treatment', 'days_to_overall_survival',
          'disease_status', 'overall_survival_status']:     
    del df[c]
                
classes = ['IMIDs-based', 'Bortezomib-based', 'Carfilzomib-based', 
           'Combined bortezomib/IMIDs-based']

print('\n\nRaw data set composed by {} rows and {} columns\n'.format(df.shape[0], df.shape[1]))

#
#
#
count_therapy_class_all = pd.DataFrame(dict(Counter(df['therapy_first_line_class'])),index=['count']).T

count_therapy_all = pd.DataFrame(dict(Counter(df['therapy_first_line'])), index=['count']).T

#
#
#
#df = df.loc[df['therapy_first_line_class'].isin(classes),:]

df = df.loc[~df.index.duplicated(keep='first')]

#df = df.loc[~df['therapy_first_line'].isnull()]

#df = df.loc[~df['iss'].isnull()]

df.to_csv('data/input.tsv', sep='\t', index=True)

display(df.iloc[:8,:8])

print('\n\nFiltered data set composed by {} rows and {} columns'.format(df.shape[0], df.shape[1]))
(df['days_to_disease_progression']/30.).describe()<jupyter_output><empty_output><jupyter_text>Counting Patient per Therapy ClassThe following script counts the patients associated to each considered therapy class. We consider three counts criterium:1. All data composed by missing and no missing data2. Exclusion of all observations associated to missing *days_to_disease_progression*3. Exclusion of all observations associated to missing *best_response_first_line*<jupyter_code>count_therapy_class_rep = pd.DataFrame(dict(Counter(
    df.loc[~df['days_to_disease_progression'].isnull()]['therapy_first_line_class'])), 
             index=['days_to_disease_progression']).T.join(pd.DataFrame(dict(
    Counter(df.loc[~df['best_response_first_line'].isnull()]['therapy_first_line_class'])),
                        index=['best_response_first_line']).T)

count_therapy_class_all.join(count_therapy_class_rep).fillna(0)<jupyter_output><empty_output><jupyter_text>Couting Patient per First Line Therapy Now we count patients per first line therapy, grouping results in the same way used for first line therapy class above.<jupyter_code>count_therapy_rep = pd.DataFrame(dict(Counter(
    df.loc[~df['days_to_disease_progression'].isnull()]['therapy_first_line'])),
                                 index=['days_to_disease_progression']).T.join(
    pd.DataFrame(dict(Counter(df.loc[~df['best_response_first_line'].isnull()]['therapy_first_line'])),
                 index=['best_response_first_line']).T)

count_therapy_all.join(count_therapy_rep).fillna(0)<jupyter_output><empty_output><jupyter_text>Response VariablesHere we define two response variables:* **best_response_first_line_class** is computed from *best_response_first_line* variable that describes how well the patient responde to the applied therapy in a qualitative way. We split patient in two groups generated by the following responses:    * **Low Risk**: Stringent Complete Response, and Complete Response    * **Hight Risk**: Very Good Partial Response, Partial Response, Stable Disease, Progressive Disease * **days_to_disease_progression** is computed from *days_to_disease_progression* variable that describe the amount of days until the disease progression. We split patients into two groups:    * **Low Risk**: patients with days_to_disease_progression >= 18 months    * **High Risk**: patients with days_to_disease_progression < 18 months<jupyter_code>df['best_response_first_line'].unique()
groups = (['Stringent Complete Response', 'Complete Response'], 
          ['Very Good Partial Response', 'Partial Response', 'Stable Disease', 'Progressive Disease'])

'''
groups2 = (['Stringent Complete Response', 'Complete Response', 'Very Good Partial Response'], 
           ['Partial Response', 'Stable Disease', 'Progressive Disease'])

groups3 = (['Stringent Complete Response'], 
           ['Complete Response', 'Very Good Partial Response', 'Partial Response', 'Stable Disease', 'Progressive Disease'])

groups4 = (['Stringent Complete Response', 'Complete Response', 'Very Good Partial Response', 'Partial Response'], 
           ['Stable Disease', 'Progressive Disease'])
'''

# split response variable and drop it from iss and fish variables
    
df['response_best_response_first_line'] = \
    df['best_response_first_line'].apply(lambda x: np.nan if pd.isnull(x) else (1 if x in groups[0] else 0))

'''
df['response_best_response_first_line2'] = \
    df['best_response_first_line'].apply(lambda x: np.nan if pd.isnull(x) else (1 if x in groups2[0] else 0))

df['response_best_response_first_line3'] = \
    df['best_response_first_line'].apply(lambda x: np.nan if pd.isnull(x) else (1 if x in groups3[0] else 0))

df['response_best_response_first_line4'] = \
    df['best_response_first_line'].apply(lambda x: np.nan if pd.isnull(x) else (1 if x in groups4[0] else 0))
'''

###################################################

del df['best_response_first_line']

df['response_days_to_disease_progression'] = \
    df['days_to_disease_progression'].apply(
        lambda x: np.nan if np.isnan(x) or pd.isnull(x) or x is None else (
            0 if x < df['days_to_disease_progression'].median() else 1))

del df['days_to_disease_progression']

df['response_days_to_first_response'] = df['days_to_first_response'].apply(
    lambda x: np.nan if np.isnan(x) or pd.isnull(x) or x is None else (
        0 if x < df['days_to_first_response'].median() else 1))

del df['days_to_first_response']

df['response_best_response_and_days_to_first_therapy'] = df['response_best_response_first_line'] * df['response_days_to_first_response']

df[['response_best_response_first_line', 'response_days_to_disease_progression', 
    # 'response_best_response_first_line2', 'response_best_response_first_line3',
    'response_days_to_first_response', 'response_best_response_and_days_to_first_therapy']].head(10)
from collections import Counter
Counter(df['response_best_response_first_line'].dropna())<jupyter_output><empty_output><jupyter_text>Describing Qualitative VariablesDescribing qualitative variable filtring by dtype =='object'. Count unique values and present them as a list.<jupyter_code>qualitative_counts = {'variable': [], 'unique_values': [], 'unique_count': []}

for c in df:
    if df[c].dtype == 'object':
        uniquies = df[c].unique().tolist()
        qualitative_counts['variable'].append(c)
        qualitative_counts['unique_values'].append(', '.join([str(u) for u in uniquies]))
        qualitative_counts['unique_count'].append(len(uniquies))

pd.DataFrame(qualitative_counts).set_index('variable').head(8)<jupyter_output><empty_output><jupyter_text>Cleaning Qualitative VariablesReplacing *'Not Recorded* and *Unknown* by *numpy.nan* for all qualitaive variables. Replace ISS values by corresponding numeric ones.<jupyter_code>iss_dict = {'Stage I': 1, 'Stage II': 2, 'Stage III': 3}

df['iss'] = df['iss'].apply(lambda x: iss_dict.get(x, np.nan))

for c in df:
    if df[c].dtype == 'object':
        df[c] = df[c].apply(lambda v: np.nan if v in ('Not Recorded', 'Unknown') else v)<jupyter_output><empty_output><jupyter_text>Exporting New Data SetWe save our new data set to *data/input.tsv* formmated as colunar file split by tabs (\t).<jupyter_code>df.to_csv('data/input.tsv', sep='\t', index=True)
df.shape<jupyter_output><empty_output><jupyter_text>Basic Statistics<jupyter_code>from collections import Counter
gender = dict(Counter(df['gender'].dropna()))
gender['Male'] / (gender['Male'] + gender['Female'])
df.agg(['mean', 'max', 'min']).T<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12363465160075329, 'fraction_numerical': 0.006026365348399247, 'mean_word_length': 4.997176736307171, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 43, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '350625', 'license': 'Apache-2.0', 'n_tokens_mistral': 3432, 'n_tokens_neox': 3297, 'n_words': 837, 'path': '00_00_merge_and_clean.ipynb', 'repo_name': 'urielcaire/mult', 'size': '10620'}