<jupyter_start><jupyter_text>Patient Overlap and Data LeakagePatient overlap in medical data is a part of a more general problem in machine learning called **data leakage**.  To identify patient overlap in this notebook, we'll check to see if a patient's ID appears in both the training set and the test set. We should also verify that we don't have patient overlap in the training and validation sets, which is what we'll do here.Below is a simple example showing how we can check for and remove patient overlap in our training and validations sets.  Import necessary packages<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import os
import seaborn as sns
sns.set()<jupyter_output><empty_output><jupyter_text>Read in the data from a csv fileFirst, we'll read in our training and validation datasets from csv files. Running the next two cells to read these csvs into `pandas` dataframes.<jupyter_code># Read csv file containing training data
train_df = pd.read_csv("nih/train-small.csv")
# Print first 5 rows
print(f'There are {train_df.shape[0]} rows and {train_df.shape[1]} columns in the training dataframe')
train_df.head()<jupyter_output>There are 1000 rows and 16 columns in the training dataframe<jupyter_text>Read csv file containing validation data<jupyter_code>valid_df = pd.read_csv("nih/valid-small.csv")
# Print first 5 rows
print(f'There are {valid_df.shape[0]} rows and {valid_df.shape[1]} columns in the validation dataframe')
valid_df.head()<jupyter_output>There are 109 rows and 16 columns in the validation dataframe<jupyter_text>Extract and compare the PatientId columns from the train and validation setsBy running the next four cells we will do the following:1. Extract patient IDs from the train and validation sets2. Convert these arrays of numbers into `set()` datatypes for easy comparison3. Identify patient overlap in the intersection of the two sets  Extract patient id's for the training set<jupyter_code>ids_train = train_df.PatientId.values<jupyter_output><empty_output><jupyter_text>Extract patient id's for the validation set<jupyter_code>ids_valid = valid_df.PatientId.values<jupyter_output><empty_output><jupyter_text>Create a "set" datastructure of the training & Test set id's to identify unique id's<jupyter_code>ids_train_set = set(ids_train)
print(f'There are {len(ids_train_set)} unique Patient IDs in the training set')

ids_valid_set = set(ids_valid)
print(f'There are {len(ids_valid_set)} unique Patient IDs in the validation set')<jupyter_output>There are 928 unique Patient IDs in the training set
There are 97 unique Patient IDs in the validation set<jupyter_text>Identify patient overlap by looking at the intersection between the sets<jupyter_code>patient_overlap = list(ids_train_set.intersection(ids_valid_set))
n_overlap = len(patient_overlap)
print(f'There are {n_overlap} Patient IDs in both the training and validation sets')
print('')
print(f'These patients are in both the training and validation datasets:')
print(f'{patient_overlap}')<jupyter_output>There are 11 Patient IDs in both the training and validation sets

These patients are in both the training and validation datasets:
[20290, 27618, 9925, 10888, 22764, 19981, 18253, 4461, 28208, 8760, 7482]<jupyter_text>Identify rows (indices) of overlapping patients and remove from either the train or validation setRunning the next two cells to do the following:1. Create lists of the overlapping row numbers in both the training and validation sets. 2. Drop the overlapping patient records from the validation set (could also choose to drop from train set)  Creating lists of the overlapping row numbers in both the training and validation sets<jupyter_code>train_overlap_idxs = []
valid_overlap_idxs = []
for idx in range(n_overlap):
    train_overlap_idxs.extend(train_df.index[train_df['PatientId'] == patient_overlap[idx]].tolist())
    valid_overlap_idxs.extend(valid_df.index[valid_df['PatientId'] == patient_overlap[idx]].tolist())
    
print(f'These are the indices of overlapping patients in the training set: ')
print(f'{train_overlap_idxs}')
print(f'These are the indices of overlapping patients in the validation set: ')
print(f'{valid_overlap_idxs}')<jupyter_output>These are the indices of overlapping patients in the training set: 
[306, 186, 797, 98, 408, 917, 327, 913, 10, 51, 276]
These are the indices of overlapping patients in the validation set: 
[104, 88, 65, 13, 2, 41, 56, 70, 26, 75, 20, 52, 55]<jupyter_text>Dropping the overlapping patient records from the validation set (could also choose to drop from train set)<jupyter_code>train_overlap_idxs = []
valid_overlap_idxs = []
for idx in range(n_overlap):
    train_overlap_idxs.extend(train_df.index[train_df['PatientId'] == patient_overlap[idx]].tolist())
    valid_overlap_idxs.extend(valid_df.index[valid_df['PatientId'] == patient_overlap[idx]].tolist())
    
print(f'These are the indices of overlapping patients in the training set: ')
print(f'{train_overlap_idxs}')
print(f'These are the indices of overlapping patients in the validation set: ')
print(f'{valid_overlap_idxs}')<jupyter_output>These are the indices of overlapping patients in the training set: 
[306, 186, 797, 98, 408, 917, 327, 913, 10, 51, 276]
These are the indices of overlapping patients in the validation set: 
[104, 88, 65, 13, 2, 41, 56, 70, 26, 75, 20, 52, 55]<jupyter_text>Drop the overlapping rows from the validation set<jupyter_code>valid_df.drop(valid_overlap_idxs, inplace=True)<jupyter_output><empty_output><jupyter_text>Checking that everything worked as planned by rerunning the patient ID comparison between train and validation sets.When we run the next two cells we should see that there are now fewer records in the validation set and that the overlap problem has been removed!   Extract patient id's for the validation set<jupyter_code>ids_valid = valid_df.PatientId.values
# Create a "set" datastructure of the validation set id's to identify unique id's
ids_valid_set = set(ids_valid)
print(f'There are {len(ids_valid_set)} unique Patient IDs in the validation set')<jupyter_output>There are 86 unique Patient IDs in the validation set<jupyter_text>Identify patient overlap by looking at the intersection between the sets<jupyter_code>patient_overlap = list(ids_train_set.intersection(ids_valid_set))
n_overlap = len(patient_overlap)
print(f'There are {n_overlap} Patient IDs in both the training and validation sets')<jupyter_output>There are 0 Patient IDs in both the training and validation sets
{'fraction_non_alphanumeric': 0.07426650366748166, 'fraction_numerical': 0.029798288508557456, 'mean_word_length': 6.370495495495495, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '490056', 'license': 'MIT', 'n_tokens_mistral': 2147, 'n_tokens_neox': 1915, 'n_words': 768, 'path': 'Notebooks/Handling Patient Overlap & Data Leakage.ipynb', 'repo_name': 'kampaitees/Deep-Learning-for-Medical-Purposes', 'size': '6544'}