<jupyter_start><jupyter_text>Select patients for CFR model: Split patients in train, val and test sets<jupyter_code>import os
import glob
import numpy as np
import pandas as pd

pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
cfr_data_root = os.path.normpath('/mnt/obi0/andreas/data/cfr')
pet_data_dir = os.path.normpath('/mnt/obi0/phi/pet/pet_cfr')
meta_date = '200617'
meta_dir = os.path.join(cfr_data_root, 'metadata_'+meta_date)
print(meta_dir)
match_view_filename = 'pet_match365_diff_files_'+meta_date+'.parquet'
files_cfr = pd.read_parquet(os.path.join(meta_dir, match_view_filename))

print('Total number of patients      {}'.format(len(files_cfr.mrn.unique())))
print('Total number of echo studies  {}'.format(len(files_cfr.study.unique())))
print('Total number of PET studies   {}'.format(len(files_cfr.petmrn_identifier.unique())))
print('Total number of echos         {}'.format(len(files_cfr.filename.unique())))

files_cfr.head(2)<jupyter_output>Total number of patients      3268
Total number of echo studies  6460
Total number of PET studies   3603
Total number of echos         307566<jupyter_text>Filter data sets: GLOBAL and NON-DEFECT variables   Global variables Notebook 3/17/2020: global_pet_cfrFile used: pet_cfr_petdata_02_26_2020_withperfandseg7.xlsxAs described above, 2871 after excluding CABG, transplant, and those with missing perfusion dataFile used: post_2018_pets_with_clinical_cfr_all.csvAfter excluding CABG, transplant, and missing CFR values, 167 PETs with any perfusionMerge:After combining, 3038 PETsFile saved as pets_with_echos_global_all.parquetUse notes:NOTE- there are petmrn_identifiers that have two rows- the one with post_2018==0 is the one to use, and the one with post_2018==1 should be excludedVariables to use: rest_global_mbf, stress_global_mbf, global_cfr_calcOther variables- myocardial_perfusion, segmental data, perfusion data, TID, gated SPECT results, calcium score, height, weightFor "cleaner" data, would exclude those with post-2018==1 Update 6/14/2020 Created a revised version of the above that includes CABG cases (from pre-2018), and excludes post-2018 cases that are already in the pre-2018 dataset (n=7). Total 3718 rows. File at /mnt/obi0/phi/pet/pet_cfr/pets_with_echos_global_all_withcabg.parquetUpdate 6/17/2020:Created a revised version that has a column tracer_obi for the tracer used. Also, the 7 post-2018 duplicate cases are excluded so there are are 3031 studies.File used- aiCohort_withPerfusion_addRadiopharm.xlsxThe ammonia cases before 7/25/2011 have the value 'listed as ammonia', and the rubidium cases after 7/25/2011 have the value 'listed as rubidium. For the remaining discrepant values (i.e. missing values, FDG, sestamibi), the tracer was assumed to be the tracer in use at the time.Rubidium 1,740, ammonia 1,276, listed as ammonia 4, listed as rubidium 2File saved as /mnt/obi0/phi/pet/pet_cfr/pets_with_echos_global_all_withtracer.parquet<jupyter_code>global_pet_file = 'pets_with_echos_global_all_withtracer.parquet'
global_pet = pd.read_parquet(os.path.join(pet_data_dir, global_pet_file))
global_pet = global_pet.astype({'pet_date': 'datetime64[ns]'})
print(f'PET studies: {len(global_pet.petmrn_identifier.unique())}')
print(os.path.join(pet_data_dir, global_pet_file))
global_pet.head()
global_pet[global_pet.petmrn_identifier=='1414556_2018-10-30']
# Exclude files without frame_time
file_cfr_meta = files_cfr.loc[~files_cfr.frame_time.isnull()]

# Add echo data to the pet studies (inner join, to keep only keys in both dataframes)
global_pet_echo = global_pet.merge(file_cfr_meta, on = ['mrn', 'pet_date', 'petmrn_identifier'], how='inner')
print(f'PET studies:  {len(global_pet_echo.petmrn_identifier.unique())}')
print(f'ECHO studies: {len(global_pet_echo.study.unique())}')
print(f'All data:    patients:     {len(file_cfr_meta.mrn.unique())}')
print(f'All data:    PET studies:  {len(file_cfr_meta.petmrn_identifier.unique())}')
print(f'All data:    ECHO studies: {len(file_cfr_meta.study.unique())}')
print(f'All data:    videos:       {len(file_cfr_meta.filename.unique())}')
print()
print(f'Global data: patients:     {len(global_pet_echo.mrn.unique())}')
print(f'Global data: PET studies:  {len(global_pet_echo.petmrn_identifier.unique())}')
print(f'Global data: ECHO studies: {len(global_pet_echo.study.unique())}')
print(f'Global data: videos:       {len(global_pet_echo.filename.unique())}')

excluded_pet_studies = len(file_cfr_meta.petmrn_identifier.unique()) - len(global_pet_echo.petmrn_identifier.unique())
print(f'Excluded PET studies: {excluded_pet_studies}')
# Find out which echo studies do not have a4c views
# Get all studies WITH a4c views
a4c_study_list = list(global_pet_echo[global_pet_echo.max_view=='a4c'].study.unique())
print(len(a4c_study_list))

# Remove all studies from global list that have a4c views
global_pet_echo_no_a4c = global_pet_echo[~global_pet_echo.study.isin(a4c_study_list)]
print(f'Number of ECHO studies without a4c view: {len(global_pet_echo_no_a4c.study.unique())}')
print(f'For this number of patients:             {len(global_pet_echo_no_a4c.mrn.unique())}')
print(f'With this number of videos:              {len(global_pet_echo_no_a4c.filename.unique())}')
# Exclusions and filters
# A4C VIEW
global_pet_echo_a4c = global_pet_echo.loc[global_pet_echo.max_view=='a4c']
global_pet_echo_a4c = global_pet_echo_a4c.loc[~global_pet_echo_a4c.frame_time.isnull()]
global_a4c_pre18 = global_pet_echo_a4c.loc[global_pet_echo_a4c['post-2018']==0]

print(f'Global data: patients:     {len(global_pet_echo.mrn.unique())}')
print(f'Global data: PET studies:  {len(global_pet_echo.petmrn_identifier.unique())}')
print(f'Global data: ECHO studies: {len(global_pet_echo.study.unique())}')
print(f'Global data: videos:       {len(global_pet_echo.filename.unique())}')
print()
print('After a4c and post-2018 filters:')
print(f'Global data: patients:     {len(global_a4c_pre18.mrn.unique())}')
print(f'Global data: PET studies:  {len(global_a4c_pre18.petmrn_identifier.unique())}')
print(f'Global data: ECHO studies: {len(global_a4c_pre18.study.unique())}')
print(f'Global data: videos:       {len(global_a4c_pre18.filename.unique())}')
print()
print(f'Lost {len(global_pet_echo_a4c.filename.unique())-len(global_a4c_pre18.filename.unique())} due to post-2018 filtering.')
# Global_pet_echo table with the variables to use (drop rows with na in any of those variables)
global_pet_echo.head(2)
global_pet_variables_target = ['rest_global_mbf', 
                               'stress_global_mbf', 
                               'global_cfr_calc', 
                               'post-2018', 
                               'tracer_obi']

global_pet_variables = global_pet_variables_target.copy()
global_pet_variables.extend(list(files_cfr.columns))

global_pet_echo_variables = global_a4c_pre18[global_pet_variables].dropna(subset=global_pet_variables_target,
                                                                          axis=0)

print(f'Global data: patients:     {len(global_pet_echo_variables.mrn.unique())}')
print(f'Global data: PET studies:  {len(global_pet_echo_variables.petmrn_identifier.unique())}')
print(f'Global data: ECHO studies: {len(global_pet_echo_variables.study.unique())}')
print(f'Global data: videos:       {len(global_pet_echo_variables.filename.unique())}')
print(f'Tracer values:             {len(global_pet_echo_variables.tracer_obi.unique())}')
# Complete list of unique petmrn_identifier
petmrn_identifier_list = list(global_pet.petmrn_identifier.unique())
petmrn_identifier_set = list(set(petmrn_identifier_list))
# Let's filter Rahuls list of missing echos
missing_echo = pd.read_parquet(os.path.join(meta_dir, 'mrn_pet_missing_echo_file.parquet'))
print(f'petmrn_identifier in original list: {len(missing_echo.petmrn_identifier.unique())}')
missing_echo.tail()
missing_echo_filtered = missing_echo[missing_echo.petmrn_identifier.isin(petmrn_identifier_set)].\
                        drop(columns=['pet_measurement'], axis=1).reset_index(drop=True)
print(f'petmrn_identifier in filtered list:   {len(missing_echo_filtered.petmrn_identifier.unique())}')
missing_no_echo_date = missing_echo_filtered.loc[missing_echo_filtered.echo_date.isnull()]
print(f'petmrn_identifier without echo dates: {len(missing_no_echo_date.petmrn_identifier.unique())}')

missing_echo_filtered_file = 'mrn_pet_missing_echo_file_filtered.parquet'
missing_echo_filtered.to_parquet(os.path.join(meta_dir, missing_echo_filtered_file))<jupyter_output>petmrn_identifier in original list: 229
petmrn_identifier in filtered list:   72
petmrn_identifier without echo dates: 2<jupyter_text>Split the patients in train, validate and test sets Although each view might have a little different patient pupulation distribution, because not all views are in each study. However, we want the same MRNS in each data set and for all views so that we can directly compare the performance of the algorithm for the same patients. We can expand the data frame above to add the splits.<jupyter_code>def patientsplit(patient_list):

    train_test_split = 0.85
    train_eval_split = 0.90

    # Take a test set from all patients
    patient_list_train = np.random.choice(patient_list,
                                          size = int(np.floor(train_test_split*len(patient_list))),
                                          replace = False)
    patient_list_test = list(set(patient_list).difference(patient_list_train))
    train_test_intersection = set(patient_list_train).intersection(set(patient_list_test)) # This should be empty
    print('Intersection of patient_list_train and patient_list_test:', train_test_intersection)

    # Further separate some patients for evaluation from the training list
    patient_list_eval = np.random.choice(patient_list_train,
                                         size = int(np.ceil((1-train_eval_split)*len(patient_list_train))),
                                         replace = False)

    patient_list_train = set(patient_list_train).difference(patient_list_eval)
    train_eval_intersection = set(patient_list_train).intersection(set(patient_list_eval))
    print('Intersection of patient_list_train and patient_list_eval:', train_eval_intersection)

    # Show the numbers
    print('total patients:', len(patient_list))
    print()
    print('patients in set:', np.sum([len(patient_list_train),
                                     len(patient_list_eval),
                                     len(patient_list_test)]))
    print()
    print('patients in train:', len(patient_list_train))
    print('patients in eval:', len(patient_list_eval))
    print('patients in test:', len(patient_list_test))

    return patient_list_train, patient_list_eval, patient_list_test
dataset = global_pet_echo_variables
dataset_filename = 'global_pet_echo_dataset_'+meta_date+'.parquet'
global_pet_echo_variables.head()

# Get a patient list
patient_list = list(dataset.sample(frac=1).mrn.unique())
patient_list_train, patient_list_eval, patient_list_test = patientsplit(patient_list)

patient_split = {'train': patient_list_train,
                 'eval': patient_list_eval,
                 'test': patient_list_test}

print('Patient IDs in train:', len(patient_split['train']))
print('Patient IDs in eval:', len(patient_split['eval']))
print('Patient IDs in test:', len(patient_split['test']))

print()

print('Intersection train-test:', set(patient_split['train']).intersection(set(patient_split['test'])))
print('Intersection train-eval:', set(patient_split['train']).intersection(set(patient_split['eval'])))
print('Intersection eval-test:', set(patient_split['eval']).intersection(set(patient_split['test'])))
# Add mode column to files_cfr
split_list = []
for dset in patient_split.keys():
    dset_mrn_list = list(patient_split[dset])
    split_list.append(pd.DataFrame({'mrn': dset_mrn_list,
                                    'dset_mode': [dset]*len(dset_mrn_list)}))

split_df = pd.concat(split_list, ignore_index = True)

dataset_split = dataset.merge(right = split_df, on = 'mrn', how = 'left').\
                sample(frac = 1).\
                reset_index(drop = True)
print(f'TOTAL patients:     {len(dataset_split.mrn.unique())}')
print(f'TOTAL PET studies:  {len(dataset_split.petmrn_identifier.unique())}')
print(f'TOTAL ECHO studies: {len(dataset_split.study.unique())}')
print(f'TOTAL videos:       {len(dataset_split.filename.unique())}')

dset_list = ['train', 'eval', 'test']
for dset in dset_list:
    df_dset = dataset_split[dataset_split.dset_mode==dset]
    print()
    print(f'patients in {dset}:     {len(df_dset.mrn.unique())}')
    print(f'PET studies in {dset}:  {len(df_dset.petmrn_identifier.unique())}')
    print(f'ECHO studies in {dset}: {len(df_dset.study.unique())}')
    print(f'videos in {dset}:       {len(df_dset.filename.unique())}')
# Load actual video files after TFR conversion
tfr_dir = os.path.join(cfr_data_root, 'tfr_200617', 'cfr')
train_files = sorted(glob.glob(os.path.join(tfr_dir, 'cfr_a4c_train_200617_*.parquet')))
train_files_failed = sorted(glob.glob(os.path.join(tfr_dir, 'cfr_a4c_train_200617_*.failed')))

eval_files = sorted(glob.glob(os.path.join(tfr_dir, 'cfr_a4c_eval_200617_*.parquet')))
test_files = sorted(glob.glob(os.path.join(tfr_dir, 'cfr_a4c_test_200617_*.parquet')))

train_df = pd.concat([pd.read_parquet(file) for file in train_files])
eval_df = pd.concat([pd.read_parquet(file) for file in eval_files])
test_df = pd.concat([pd.read_parquet(file) for file in test_files])

df_tf = pd.concat([train_df, eval_df, test_df]).reset_index(drop=True)

# Lets look at the failed training files
train_failed_df = pd.concat([pd.read_parquet(file) for file in train_files_failed])
print((train_failed_df.err.unique()))

dset_list = ['train', 'eval', 'test']
for dset in dset_list:
    df_dset = df_tf[df_tf.dset_mode==dset]
    print()
    print(f'patients in {dset}:     {len(df_dset.mrn.unique())}')
    print(f'PET studies in {dset}:  {len(df_dset.petmrn_identifier.unique())}')
    print(f'ECHO studies in {dset}: {len(df_dset.study.unique())}')
    print(f'videos in {dset}:       {len(df_dset.filename.unique())}')
train_failed_df[(train_failed_df.err=='deltaXY') & (train_failed_df.deltaX<0)].shape
# Let's look at the video tracer numbers
tracer_list = dataset_split.tracer_obi.unique()
print(tracer_list)
for tracer in tracer_list:
    df_tracer = dataset_split[dataset_split.tracer_obi==tracer]
    for dset in dset_list:
        df_dset = df_tracer[df_tracer.dset_mode==dset]
        print()
        print(f'{dset}: patients for {tracer}:     {len(df_dset.mrn.unique())}')
        print(f'{dset}: PET studies for {tracer}:  {len(df_dset.petmrn_identifier.unique())}')
        print(f'{dset}: ECHO studies for {tracer}: {len(df_dset.study.unique())}')
        print(f'{dset}: videos for {tracer}:       {len(df_dset.filename.unique())}')
dataset_split.columns
# Prepare the final set that we will use for writing TFR files. We don't want any rows with NAs in some columns.
col_set = ['rest_global_mbf', 'rest_global_mbf', 'global_cfr_calc', 'tracer_obi', 
           'pet_measurement', 'sum_views', 'dset_mode']
dataset_split_tfr = dataset_split.dropna(subset = col_set, axis=0)

print('Dropped {} rows.'.format(dataset_split.shape[0]-dataset_split_tfr.shape[0]))

# Add some other information that we need and shuffle the whole thing
dataset_split_tfr = dataset_split_tfr.assign(rate = np.round(1/dataset_split_tfr.frame_time*1e3, decimals = 1))
dataset_split_tfr = dataset_split_tfr.assign(dur = dataset_split_tfr.frame_time*1e-3*dataset_split_tfr.number_of_frames)

dataset_split_tfr = dataset_split_tfr.sample(frac = 1)

dataset_split.loc[~dataset_split.index.isin(dataset_split_tfr.index)]
print(list(dataset_split_tfr.tracer_obi.unique()))
dataset_split_tfr.to_parquet(os.path.join(meta_dir, dataset_filename))
print('Saved to file:', dataset_filename)
print(dataset_split_tfr.shape)
dataset_split_tfr.head()
minframes = 40
max_frame_time = 33.34
print('Minimum frames:   {}'.format(minframes))
print('Maximum frame_time: {}'.format(max_frame_time))
minduration = max_frame_time*minframes*1e-3
print('Minimum duration: {}'.format(minduration))

#minframes = int(np.ceil(minrate*minduration))

maxrows = dataset_split_tfr.shape[0]
dataset_disqualified = dataset_split_tfr[(dataset_split_tfr.frame_time > max_frame_time) | (dataset_split_tfr.dur < minduration)]
dataset_qualified = dataset_split_tfr[(dataset_split_tfr.frame_time <= max_frame_time) & (dataset_split_tfr.dur >= minduration)]

n_videos = len(dataset_split_tfr.filename.unique())
n_disqualified = len(dataset_disqualified.filename.unique())
n_qualified = len(dataset_qualified.filename.unique())

print()
print('Total videos: {}'.format(n_videos))
print('Disqualified videos {}, fraction:{:.1f}'.format(n_disqualified, 
                                                     np.round(n_disqualified/n_videos*100), decimals=1))
print('Qualified videos {}, fraction:{:.1f}'.format(n_qualified, 
                                                     np.round(n_qualified/n_videos*100), decimals=1))
# Report train val test numbers that qualify....
tracer_list = dataset_split.tracer_obi.unique()
print(tracer_list)
for dset in dset_list:
    df_dset = dataset_qualified[dataset_qualified.dset_mode==dset]
    print()
    print(f'{dset}: patients for TOTAL:     {len(df_dset.mrn.unique())}')
    print(f'{dset}: PET studies for TOTAL:  {len(df_dset.petmrn_identifier.unique())}')
    print(f'{dset}: ECHO studies for TOTAL: {len(df_dset.study.unique())}')
    print(f'{dset}: videos for TOTAL:       {len(df_dset.filename.unique())}')
    
    for tracer in tracer_list:
        df_tracer = df_dset[df_dset.tracer_obi==tracer]
        print()
        print(f'{dset}: patients for {tracer}:     {len(df_dset.mrn.unique())}')
        print(f'{dset}: PET studies for {tracer}:  {len(df_dset.petmrn_identifier.unique())}')
        print(f'{dset}: ECHO studies for {tracer}: {len(df_dset.study.unique())}')
        print(f'{dset}: videos for {tracer}:       {len(df_dset.filename.unique())}')<jupyter_output>['ammonia' 'rubidium' 'listed as rubidium' 'listed as ammonia']

train: patients for TOTAL:     1391
train: PET studies for TOTAL:  1455
train: ECHO studies for TOTAL: 2050
train: videos for TOTAL:       6225

train: patients for ammonia:     1391
train: PET studies for ammonia:  1455
train: ECHO studies for ammonia: 2050
train: videos for ammonia:       6225

train: patients for rubidium:     1391
train: PET studies for rubidium:  1455
train: ECHO studies for rubidium: 2050
train: videos for rubidium:       6225

train: patients for listed as rubidium:     1391
train: PET studies for listed as rubidium:  1455
train: ECHO studies for listed as rubidium: 2050
train: videos for listed as rubidium:       6225

train: patients for listed as ammonia:     1391
train: PET studies for listed as ammonia:  1455
train: ECHO studies for listed as ammonia: 2050
train: videos for listed as ammonia:       6225

eval: patients for TOTAL:     157
eval: PET studies for TOTAL:  163
eval: ECHO studies for[...]
{'fraction_non_alphanumeric': 0.1196269473245454, 'fraction_numerical': 0.02068462460271974, 'mean_word_length': 5.047258979206049, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '497188', 'license': 'MIT', 'n_tokens_mistral': 6880, 'n_tokens_neox': 6417, 'n_words': 1366, 'path': 'notebooks/cfr_datasets_patientsplit_global.ipynb', 'repo_name': 'awerdich/werdich-cfr', 'size': '19193'}