<jupyter_start><jupyter_text>Predicting Lung Cancer Survival Time predict the survival time of a patient (remaining days to live) from one three-dimensional CT scan (grayscale image) and a set of pre-extracted quantitative imaging features, as well as clinical data  $\color{green}{\text{ Importing Libraries }}$<jupyter_code>import pandas as pd
import numpy as np
import missingno
from sklearn.preprocessing import LabelEncoder , OneHotEncoder
from lifelines.utils import concordance_index
from lifelines import CoxPHFitter, WeibullAFTFitter , LogNormalAFTFitter, LogLogisticAFTFitter , PiecewiseExponentialRegressionFitter
from lifelines.utils import k_fold_cross_validation
from IPython.display import HTML<jupyter_output><empty_output><jupyter_text>$\color{red}{\text{ Exploring Data}}$  X Train   $\color{Blue}{\text {Features Exploration }}$<jupyter_code>train_path = '/home/zahrael97/Downloads/challenge file/xtrain/data_Q0G7b5t'
#Radiomic Training data 
df_Radiomics_train = pd.read_csv(train_path + '/features/radiomics.csv',skiprows=[0,2], header=[0],index_col=0)
#Clinical Training data 
df_clinical_train = pd.read_csv(train_path + '/features/clinical_data.csv',index_col=0)
df_clinical_train.head() #the 5 first lines<jupyter_output><empty_output><jupyter_text>What missing values are there?<jupyter_code>missingno.matrix(df_clinical_train, figsize = (30,10))
#We will use the age feature after filling the NaN values 
df_clinical_train = df_clinical_train.fillna(df_clinical_train.mean())
missingno.matrix(df_clinical_train, figsize = (30,10))<jupyter_output><empty_output><jupyter_text>We will ignore the Histology feature<jupyter_code>df_clinical_train.age.plot.hist()
df_clinical_train.Tstage.plot.hist()
df_clinical_train.Nstage.plot.hist()<jupyter_output><empty_output><jupyter_text>Feature EncodingNow we will encode the SourceDataset feature so it will be  ready to be used with our machine learning models.<jupyter_code>encoder = LabelEncoder()
encoder.fit(df_clinical_train["SourceDataset"] )
df_clinical_train["SourceDataset"]= encoder.transform(df_clinical_train["SourceDataset"])
df_Radiomics_train.head()
df_Radiomics_train.isnull().sum().sum()<jupyter_output><empty_output><jupyter_text>Features Used - Tumor sphericity, a measure of the roundness of the shape of the tumor region relative to a sphere, regardless its dimensions (size)- The tumor's surface to volume ratio is a measure of the compactness of the tumor, related to its size  - The tumor's maximum 3d diameter The biggest diameter measurable from the tumor volume - The dataset of origin- The N-tumoral stage grading of the tumor describing nearby (regional) lymph nodes involved- The tumor's joint entropy, specifying the randomness in the image pixel values- age- Tstage- The tumor's inverse different, a measure of the local homogeneity of the tumor-  The tumor's inverse difference moment is another measurement of the local homogeneity of the tumor<jupyter_code>df_X_train = pd.concat([df_Radiomics_train[["original_shape_Sphericity", "original_shape_SurfaceVolumeRatio", "original_shape_Maximum3DDiameter","original_firstorder_Entropy",
"original_glcm_Id","original_glcm_Idm"]],df_clinical_train[["SourceDataset","Nstage","Tstage","age"]]], axis=1)<jupyter_output><empty_output><jupyter_text>y Train<jupyter_code>ytrain_path = '/home/zahrael97/Downloads/challenge file/ytrain'
df_y = pd.read_csv( ytrain_path+ '/output_VSVxRFU.csv', index_col=0)
p0 = df_y.loc[202]
print("p0.Event", p0.Event)
print("p0.SurvivalTime", p0.SurvivalTime)
df_y.head()
df_y.SurvivalTime.plot.hist()
import matplotlib.pyplot as plt
import seaborn as sns

# Visualise the counts of SibSp and the distribution of the values
# against Survived
#plot_count_dist(df_y, 
#                bin_df=df_bin, 
#                label_column='SurvivalTime', 
#                target_column='Event', 
#                figsize=(10, 5))<jupyter_output><empty_output><jupyter_text>$\color{red}{\text {Start Building Machine Learning Models}}$Now our data has been manipulating and converted to numbers, we can run the machine learning Models.  CoxPHFitter<jupyter_code>cph = CoxPHFitter().fit(pd.concat([df_X_train, df_y], axis = 1),duration_col = 'SurvivalTime', event_col='Event') 
cph.print_summary()<jupyter_output><empty_output><jupyter_text>Although using Cox proportional hazard model has come to the fore by most researchers in medical and cancer fields, results of accelerated failure-time models have often been more valid and have had minor bias since these models have better fitness in similar conditions due to a specific statistical distribution for the survival time and their not having need to PH assumption. Accelerated failure-time models will also be reliable alternatives to Cox proportional hazard model where this assumption is not made. In addition, accelerated failure-time models may offer some benefits. Based on asymptotic results, accelerated failure-time models lead to more efficient parameters than proportional hazard model. With a reduction in sample size, relative efficiency may further change in favor of accelerated failure-time models. When empirical information is adequate, accelerated failure-time models can prepare some insights into the form of the baseline hazard.From paper : A Comparison between Accelerated Failure-time and Cox Proportional Hazard Models in Analyzing the Survival of Gastric Cancer Patients $\color{red}{\text { Let's try it out}}$  WeibullAFTFitter<jupyter_code>Wei = WeibullAFTFitter().fit(pd.concat([df_X_train, df_y], axis = 1),duration_col = 'SurvivalTime', event_col='Event') 
Wei.print_summary()<jupyter_output><empty_output><jupyter_text>LogNormalAFTFitter<jupyter_code>log = LogNormalAFTFitter().fit(pd.concat([df_X_train, df_y], axis = 1),duration_col = 'SurvivalTime', event_col='Event') 
log.print_summary()<jupyter_output><empty_output><jupyter_text>LogLogisticAFTFitter<jupyter_code>logistic = LogLogisticAFTFitter().fit(pd.concat([df_X_train, df_y], axis = 1),duration_col = 'SurvivalTime', event_col='Event') 
logistic.print_summary<jupyter_output><empty_output><jupyter_text>Concordance_index<jupyter_code>df = pd.concat([df_X_train, df_y], axis = 1)
cox_cv_result = k_fold_cross_validation(cph, df, duration_col='SurvivalTime', event_col='Event', k=5)
print('CoxPHFitter C-index(cross-validation) = ', np.mean(cox_cv_result))
cox_cv_result = k_fold_cross_validation(Wei, df, duration_col='SurvivalTime', event_col='Event', k=5)
print('WeibullAFTFitter C-index(cross-validation) = ', np.mean(cox_cv_result))
cox_cv_result = k_fold_cross_validation(log, df, duration_col='SurvivalTime', event_col='Event', k=5)
print('LogNormalAFTFitter C-index(cross-validation) = ', np.mean(cox_cv_result))
cox_cv_result = k_fold_cross_validation(logistic, df, duration_col='SurvivalTime', event_col='Event', k=5)
print(' C-index(cross-validation) = ', np.mean(cox_cv_result))<jupyter_output>C-index(cross-validation) =  0.6790343487530152<jupyter_text>$\color{red}{\text{ Submission}}$  X Test<jupyter_code>test_path = '/home/zahrael97/Downloads/challenge file/xtest/data_9Cbe5hx'
df_Radiomics_test = pd.read_csv(test_path +'/features/radiomics.csv',skiprows=[0,2], header=[0],index_col=0)
df_clinical_test = pd.read_csv(test_path + '/features/clinical_data.csv',index_col=0)
#fill the NaN values of age 
df_clinical_test = df_clinical_test.fillna(df_clinical_test.mean())
encoder = LabelEncoder()
encoder.fit(df_clinical_test["SourceDataset"])
df_clinical_test["SourceDataset"] = encoder.transform(df_clinical_test["SourceDataset"])
df_X_test = pd.concat([df_Radiomics_test[["original_shape_Sphericity", "original_shape_SurfaceVolumeRatio", "original_shape_Maximum3DDiameter","original_firstorder_Entropy",
"original_glcm_Id","original_glcm_Idm"]],df_clinical_test[["SourceDataset","Nstage","Tstage","age"]]], axis=1)
#df = pd.concat([df_X_train, df_y], axis = 1)
#concordance_index(df['SurvivalTime'], -cph.predict_partial_hazard(df), df['Event'])<jupyter_output><empty_output><jupyter_text>Lifetime prediction<jupyter_code>lifetime_prediction = log.predict_expectation(df_X_test)
lifetime_prediction.columns = ['SurvivalTime']
lifetime_prediction.index.names = ['PatientID']
lifetime_prediction['Event'] = 'NaN'
def create_download_link(title = "Download CSV file", filename = "data.csv"):  
    html = '<a href={filename}>{title}</a>'
    html = html.format(title=title,filename=filename)
    return HTML(html)

lifetime_prediction.to_csv('submission.csv')

create_download_link(filename='submission.csv')<jupyter_output><empty_output><jupyter_text>$\color{black}{\text{ Still working on it }}$  Exploring Images data<jupyter_code>archive = np.load(train_path + '/images/patient_003.npz')
scan = archive['scan']
mask = archive['mask']
#print(scan.shape)
import matplotlib.pyplot as plt
from matplotlib.pyplot import imshow
from PIL import Image
#plt.figure()
#plt.ion()

#archive.shape
#imshow(img)
#plt.imshow(Image.fromarray(scan, 'RGBA'))
#plt.imshow(Image.fromarray(scan, 'CMYK'))
#plt.imshow(Image.fromarray(mask, 'LA'))
plt.imshow(Image.fromarray(scan, 'LA'))
def logprediction(y_true_times) :
    y_true_times.columns = ['SurvivalTime']
    y_true_times.index.names = ['PatientID']
    y_true_times['Event'] = 0
    return(y_pred)<jupyter_output><empty_output><jupyter_text>Challenge C-index<jupyter_code>from metrics_t9gbvr2 import cindex<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11003614714012333, 'fraction_numerical': 0.00850520944078248, 'mean_word_length': 7.791588785046729, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '179176', 'license': 'MIT', 'n_tokens_mistral': 3210, 'n_tokens_neox': 3021, 'n_words': 689, 'path': 'code/Data Exploration + Lifeline.ipynb', 'repo_name': 'zahrael97/Predicting-lung-cancer-survival-time', 'size': '9406'}