<jupyter_start><jupyter_text>Basic Analysis of Breast Cancer Statistics in Mimic-III Database OverviewIn this exercise, we will perform some basic analysis of breast cancer patient data in the MIMIC-III Database. InitializationFor this analysis, we'll be using `pandas` for data loading and dataframe manipulation; `matplotlib` for visualizations; and `warnings` to suppress an ugly, unavoidable warning that a `pandas`/`numpy` bug throws.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import warnings 

# all data has been de-identified.

# ignore global numpy warnings that are thrown by pandas
warnings.simplefilter(action='ignore', category=FutureWarning)<jupyter_output><empty_output><jupyter_text>DataWe'll be using five data files from the MIMIC-III database to load diagnoses, procedures, patient data, prescription data, and ICD9 code data. We're specifying specific `usecols` values for our prescriptions because there are data type inconsistencies in other (unused) columns that generate warnings.Once we've loaded our data files, we'll take a quick look at each dataframe so we know what we're dealing with.<jupyter_code>diagDF = pd.read_csv("/home/cs52700/mimiciii/DIAGNOSES_ICD.csv", index_col=0)
admitDF = pd.read_csv("/home/cs52700/mimiciii/ADMISSIONS.csv", usecols=['ROW_ID', 'HADM_ID', 'ADMITTIME'], index_col=0)
procDF = pd.read_csv("/home/cs52700/mimiciii/PROCEDURES_ICD.csv", index_col=0)
patDF  = pd.read_csv("/home/cs52700/mimiciii/PATIENTS.csv", index_col=0)
diagCodeDF = pd.read_csv("/home/cs52700/mimiciii/D_ICD_DIAGNOSES.csv", index_col=0)
procCodeDF = pd.read_csv("/home/cs52700/mimiciii/D_ICD_PROCEDURES.csv", index_col=0)
scripDF = pd.read_csv("/home/cs52700/mimiciii/PRESCRIPTIONS.csv", usecols=['ROW_ID', 'SUBJECT_ID', 'DRUG'], index_col=0)<jupyter_output><empty_output><jupyter_text>Note that here we are paring down our patient data to only include patients with ICD9 codes that match breast cancer. We also want to merge this with our admissions data so we can associate dates with each diagnosis.<jupyter_code>bcDF = diagDF[diagDF['ICD9_CODE'].isin(["1749", "1759", "19881", "V103"])]

bcDF = pd.merge(bcDF, admitDF, on='HADM_ID')

bcDF['ADMITTIME'] = bcDF['ADMITTIME'].astype('datetime64[ns]')

bcDF.head()

procDF['ICD9_CODE'] = procDF['ICD9_CODE'].astype('str')
procDF.head()

procCodeDF['ICD9_CODE'] = procCodeDF['ICD9_CODE'].astype('str')
procCodeDF.head()

patDF['DOB'] = patDF['DOB'].astype('datetime64[ns]')

patDF.head()
scripDF = scripDF[['SUBJECT_ID', 'DRUG']]
scripDF.head()<jupyter_output><empty_output><jupyter_text>Once we've loaded and prepared our data, we can merge our breast cancer diagnosis records with our patient data. This allows us to associate our anonymized patient DOB (year of birth) with our diagnoses.There's some gloriously ugly data transformation going on here. Here's what we're doing:1. Calculate a timedelta value based on admit time minus date of birth1. Convert this value directly to a string. This will give us a timedelta in the format 'xxxxx days, yyyyyyy'1. Split this string on the first space that appears. This will give us the total number of days only.1. convert this value once more, this time to a float1. Divide this float by 365.25 (taking leap year into account!)1. Finally, convert this float to an int. This conversion automatically rounds down, which is exactly the behavior we want.When it's all said and done, we end up with the patient's age at time of hospital admission! It's an ugly kludge, sure, but it *works*, and I'm fine with that.<jupyter_code>mergeDF = pd.merge(bcDF, patDF, on='SUBJECT_ID')
mergeDF['EVENT_AGE'] = mergeDF['ADMITTIME'] - mergeDF['DOB']
mergeDF['EVENT_AGE'] = mergeDF['EVENT_AGE'].astype('str')
mergeDF['EVENT_AGE'] = mergeDF['EVENT_AGE'].str.split(" ", 1, expand=True)
mergeDF['EVENT_AGE'] = mergeDF['EVENT_AGE'].astype('float')
mergeDF['EVENT_AGE'] = mergeDF['EVENT_AGE'] / 365.25
mergeDF['EVENT_AGE'] = mergeDF['EVENT_AGE'].astype('int')


mergeDF.head()<jupyter_output><empty_output><jupyter_text>Now, we can drop duplicates and group our patients by minimum event age, then by year-of-birth, counting the total number of unique patient IDs at the end. Then, we'll draw a histogram of this data.<jupyter_code>ageDF = mergeDF[['SUBJECT_ID', 'EVENT_AGE']]
ageDF = ageDF.drop_duplicates()
ageDF = ageDF.groupby(['SUBJECT_ID'], as_index=False).min()
ageDF = ageDF.groupby(['EVENT_AGE'], as_index=False).count()

ageDF.head()
plt.title("Breast Cancer Patients by Age at First Reported Diagnosis (Raw)")
plt.xlabel("Age")
plt.ylabel("Total patients")
plt.bar(ageDF['EVENT_AGE'], ageDF['SUBJECT_ID'])
plt.rcParams['figure.figsize'] = [16, 10]
plt.show()

# Jupyter bug: you need to manually re-run this plot after running the full notebook to make it the right size.
# Sorry.<jupyter_output><empty_output><jupyter_text>Looking at our histogram, it becomes immediately evident that there are some inconsistent entries. For example, we have impressive numbers of breast cancer patients who were first diagnosed nearly 300 years before they were born. While medical science may some day be able to provide our descendants with such astonishingly foresighted diagnoses, we know for a fact that 17th century medical science decidedly could **not** achieve this amazing feat.Let's eliminate this bad data.<jupyter_code>corAgeDF = ageDF[ageDF['EVENT_AGE'] > 0];

corAgeDF.head()

plt.title("Breast Cancer Patients by Age at First Reported Diagnosis (corrected)")
plt.xlabel("Age")
plt.ylabel("Total Patients")
plt.bar(corAgeDF['EVENT_AGE'], corAgeDF['SUBJECT_ID'])
plt.rcParams['figure.figsize'] = [16, 10]
plt.show()<jupyter_output><empty_output><jupyter_text>...and there we have it: breast cancer incidence rates graphed by age of a patient's first reported breast cancer ICD9 code!Let's do a few more. Let's look at associated conditions, too!<jupyter_code>bcPatientsDF = mergeDF[['SUBJECT_ID']]
bcPatientsDF = bcPatientsDF.drop_duplicates()

bcConditionsDF = pd.merge(bcPatientsDF, diagDF, on='SUBJECT_ID')
bcConditionsDF = bcConditionsDF[['ICD9_CODE', 'SUBJECT_ID']]


bcConditionsDF = bcConditionsDF[~bcConditionsDF['ICD9_CODE'].isin(["1749", "1759", "19881", "V103"])]
bcConditionsDF = bcConditionsDF.groupby(['ICD9_CODE'], as_index=False).count()
bcConditionsDF = bcConditionsDF.sort_values(by=['SUBJECT_ID'], ascending=False)
bcConditionsDF = bcConditionsDF.head(10)

bcConditionDefinitionsDF = pd.merge(bcConditionsDF, diagCodeDF, on='ICD9_CODE')
bcConditionDefinitionsDF = bcConditionDefinitionsDF.rename(columns={"SUBJECT_ID": "PATIENT_COUNT"})
bcConditionDefinitionsDF


plt.title("Top 10 Conditions Associated with Breast Cancer Patients")
plt.pie(bcConditionDefinitionsDF['PATIENT_COUNT'], labels=bcConditionDefinitionsDF['LONG_TITLE'], autopct='%1.1f%%')
plt.rcParams['figure.figsize'] = [16, 10]
plt.show()<jupyter_output><empty_output><jupyter_text>Next up: associated procedures!<jupyter_code>bcProceduresDF = pd.merge(bcPatientsDF, procDF, on='SUBJECT_ID')
bcProceduresDF = bcProceduresDF[['ICD9_CODE', 'SUBJECT_ID']]


bcProceduresDF = bcProceduresDF[~bcProceduresDF['ICD9_CODE'].isin(["1749", "1759", "19881", "V103"])]
bcProceduresDF = bcProceduresDF.groupby(['ICD9_CODE'], as_index=False).count()
bcProceduresDF = bcProceduresDF.sort_values(by=['SUBJECT_ID'], ascending=False)
bcProceduresDF = bcProceduresDF.head(10)


bcProceduresDF = pd.merge(bcProceduresDF, procCodeDF, on='ICD9_CODE')
bcProceduresDF = bcProceduresDF.rename(columns={"SUBJECT_ID": "PATIENT_COUNT"})
bcProceduresDF.head(10)

plt.title("Top 10 Procedures Associated with Breast Cancer Patients")
plt.pie(bcProceduresDF['PATIENT_COUNT'], labels=bcProceduresDF['LONG_TITLE'], autopct='%1.1f%%')
plt.rcParams['figure.figsize'] = [16, 10]
plt.show()<jupyter_output><empty_output><jupyter_text>...and finally, associated drug prescriptions!<jupyter_code>bcDrugsDF = pd.merge(bcPatientsDF, scripDF, on='SUBJECT_ID')
bcDrugsDF = bcDrugsDF[['SUBJECT_ID', 'DRUG']]


bcDrugsDF = bcDrugsDF.groupby(['DRUG'], as_index=False).count()
bcDrugsDF = bcDrugsDF.sort_values(by=['SUBJECT_ID'], ascending=False)
bcDrugsDF = bcDrugsDF.head(10)


bcDrugsDF = bcDrugsDF.rename(columns={"SUBJECT_ID": "PATIENT_COUNT"})
bcDrugsDF.head(10)

plt.title("Top 10 Prescribed Drugs Associated with Breast Cancer Patients")
plt.pie(bcDrugsDF['PATIENT_COUNT'], labels=bcDrugsDF['DRUG'], autopct='%1.1f%%')
plt.rcParams['figure.figsize'] = [16, 10]
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1252358490566038, 'fraction_numerical': 0.020636792452830188, 'mean_word_length': 7.274146341463415, 'pattern_counts': {'":': 3, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '444110', 'license': 'MIT', 'n_tokens_mistral': 3060, 'n_tokens_neox': 2778, 'n_words': 753, 'path': 'Assignment 3 - Breast Cancer Analysis.ipynb', 'repo_name': 'ThomasDarby/DataScienceEdgeDetection', 'size': '8480'}