<jupyter_start><jupyter_text>EDA is open-ended, and it is up to you to decide how to look at different ways to slice and dice your data. A good starting point is to look at the requirements for the FDA documentation in the final part of this project to guide (some) of the analyses you do. This EDA should also help to inform you of how pneumonia looks in the wild. E.g. what other types of diseases it's commonly found with, how often it is found, what ages it affects, etc. Note that this NIH dataset was not specifically acquired for pneumonia. So, while this is a representation of 'pneumonia in the wild,' the prevalence of pneumonia may be different if you were to take only chest x-rays that were acquired in an ER setting with suspicion of pneumonia.  Perform the following EDA:* The patient demographic data such as gender, age, patient position,etc. (as it is available)* The x-ray views taken (i.e. view position)* The number of cases including:     * number of pneumonia cases,    * number of non-pneumonia cases* The distribution of other diseases that are comorbid with pneumonia* Number of disease per patient * Pixel-level assessments of the imaging data for healthy & disease states of interest (e.g. histograms of intensity values) and compare distributions across diseases.Note: use full NIH data to perform the first a few EDA items and use `sample_labels.csv` for the pixel-level assassements.  Also, **describe your findings and how will you set up the model training based on the findings.**<jupyter_code>## Below is some helper code to read data for you.
## Load NIH data
all_xray_df = pd.read_csv('/data/Data_Entry_2017.csv')
all_xray_df.sample(10)
## Load 'sample_labels.csv' data for pixel level assessments
sample_df = pd.read_csv('sample_labels.csv')
sample_df.sample(10)
## EDA
# Todo 

## Check for null values

all_xray_df.info()
all_xray_df.columns
all_xray_df.shape<jupyter_output><empty_output><jupyter_text>Patient Gender<jupyter_code>all_xray_df['Patient Gender'].value_counts().plot(kind='bar')
all_xray_df['Patient Gender'].describe()<jupyter_output><empty_output><jupyter_text>Patient Age<jupyter_code>from scipy.stats import norm
sns.distplot(all_xray_df['Patient Age'], hist=True, bins=25, fit=norm, kde=False)
all_xray_df['Patient Age'].describe()<jupyter_output><empty_output><jupyter_text>Patient View Position<jupyter_code>all_xray_df['View Position'].value_counts().plot(kind='bar')
all_xray_df['View Position'].describe()<jupyter_output><empty_output><jupyter_text>Analysis 1 1. After plotting patient's age and gender, we can clearly see that majority of the data consists of male samples and average age is around 47.2. By analysing patient view postion we have majority cases taken as PA. and the questiom is what is PA and AP. 3. PA X-ray: An X-ray picture in which the beams pass from back-to-front (posteroanterior). As opposed to an AP (anteroposterior) image in which the rays pass through the body from front-to-back.4. AP, X-ray: An X-ray picture in which the beams pass from front-to-back (anteroposterior). As opposed to a PA (posteroanterior) film in which the rays pass through the body from back-to-front.**Definition source:** [medicinenet](https://www.medicinenet.com/ap_x-ray/definition.htm)  Number of Pneumonia VS Non Pneumonia Cases Here we are splitting up the 'Finding Labels' column into one additional column per disease (e.g. one for 'Cardiomegaly', one for 'Emphysema', etc.) and put a binary flag in that column to indicate the presence of the disease.<jupyter_code>## Pre processing on dataframe

all_labels = np.unique(list(chain(*all_xray_df['Finding Labels'].map(lambda x: x.split('|')).tolist())))
all_labels = [x for x in all_labels if len(x)>0]
print('All Labels ({}): {}'.format(len(all_labels), all_labels))
for c_label in all_labels:
    if len(c_label)>1: # leave out empty labels
        all_xray_df[c_label] = all_xray_df['Finding Labels'].map(lambda finding: 1.0 if c_label in finding else 0)
all_xray_df.sample(5)
pos_cnt = all_xray_df['Patient ID'][all_xray_df['Pneumonia'] == 1].value_counts().sum()
neg_cnt = all_xray_df['Patient ID'][all_xray_df['Pneumonia'] == 0].value_counts().sum()
print('Pneumonia Cases: %s and Non-Pneumonia Cases: %s' % (pos_cnt, neg_cnt))<jupyter_output>Pneumonia Cases: 1431 and Non-Pneumonia Cases: 110689<jupyter_text>Analyzing Pneumonia Cases  Gender Distribution<jupyter_code>plt.figure(figsize=(10,5))
all_xray_df[all_xray_df['Pneumonia'] == 1]['Patient Gender'].value_counts().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Age Distribution<jupyter_code>plt.figure(figsize=(10,6))
all_xray_df[all_xray_df['Pneumonia'] == 1]['Patient Age'].value_counts().plot(kind='hist', bins=25, rwidth=0.8)
all_xray_df[all_xray_df['Pneumonia'] == 1]['Patient Age'].value_counts().describe()<jupyter_output><empty_output><jupyter_text>Analysis 21. Here we can clearly see that, we have 1431 cases of pneumonia and 110689 cases of non-pneumonia.2. On analyzing pneumonia cases we can clearly see that we have majority male cases and with average age of patient suffering from pneumonia being 16 years.3. We can clearly see that there is a huge imbalance in tha data set, we have very few examples of positive cases which can create a hindrance while trainng our model. We can adopt following well known techniques to deal with imbalance situations: -    3.1 Random Undersampling and Oversampling - here what we do is, we remove samples from the majority class (under-sampling) and/or adding more examples from the minority class (over-sampling).    3.2 Undersampling and Oversampling using imbalanced-learn - imbalanced-learn(imblearn) is a Python Package to tackle the curse of imbalanced datasets. It provides a variety of methods to undersample and oversample like Undersampling using Tomek Links, Oversampling using SMOTE etc.    3.3 Changing Evaluation Metrics**Imbalance Handling Source:** [kdnuggets.com](https://www.kdnuggets.com/2020/01/5-most-useful-techniques-handle-imbalanced-datasets.html:~:text=Random%20Undersampling%20and%20Oversampling&text=A%20widely%20adopted%20and%20perhaps,class%20(over%2Dsampling).)   The distribution of other diseases that are comorbid with pneumonia<jupyter_code>all_xray_df[all_labels].sum().plot(kind='bar')
plt.figure(figsize=(20,10))
all_xray_df[all_xray_df['Pneumonia'] == 1]['Finding Labels'].value_counts()[:30].plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Analysis 3Here after plotting the distribution of occurence of other diseases with pneumonia, we can see that most commonly occuring diseases with pneumonia are Infiltration, Edemal, Atelectasis and consolidation.  Number of disease per patient<jupyter_code>all_xray_df.head()
print(all_labels)
all_xray_df.info()
subset_df = all_xray_df.iloc[:,12:27].copy()
subset_df.head()
subset_df.drop('No Finding', axis=1, inplace=True)
subset_df.head()
all_xray_df['Disease Count'] = subset_df.sum(axis=1)
all_xray_df.head()
all_xray_df['Disease Count'].value_counts()
plt.figure(figsize=(8,8))
plt.hist(all_xray_df['Disease Count'], rwidth=0.8)
plt.xlabel('Number of Diseases')
plt.ylabel('Patients Count')<jupyter_output><empty_output><jupyter_text>Analysis 4From the above manipulations to dataframe, we have calculated a new column named **Disease Count** and we can clearly see that there are 30963 patients with one disease, 14306 patients with two diseases, and 4856 with three diseases respectively  Pixel-level assessments of the Imaging Data<jupyter_code>## Let's examine one DICOM file and its contents 

dcm = pydicom.dcmread('test1.dcm')
dcm
## Let's do some exploratory work before about how to extract these attributes

print('Patient ID: ',dcm.PatientID)
print('Patient Age: ',dcm.PatientAge)
print('Patient Sex: ',dcm.PatientSex)
print('Modality: ',dcm.Modality)
print('Study Description: ',dcm.StudyDescription)
print('Rows in Image: ',dcm.Rows)
print('Columns in Image: ',dcm.Columns)
## Now we create dataframe from dicom files with necessary attributes

dicom_files = glob('*.dcm')

dicom_data = []

for file in dicom_files: 
    dcm = pydicom.dcmread(file)
    attr = [dcm.PatientID, dcm.PatientAge, dcm.PatientSex, dcm.Modality, dcm.StudyDescription,
             dcm.Rows, dcm.Columns]
    dicom_data.append(attr)
dicom_data
dicom_df = pd.DataFrame(dicom_data, columns=['PatientID','PatientAge','PatientSex','Modality',
                                        'StudyDescription','Rows','Columns'])
dicom_df.head()
def plot_distribution(dcm, finding):
    img_arr = dcm.pixel_array
    
    ## normalize image
    mean, std = np.mean(img_arr), np.std(img_arr)
    img_arr = (img_arr - mean)/std
    
    ## plotting image and its pixel intensity distribution
    
    fig = plt.figure()
    fig.set_figwidth(20)
    
    ## Plotting Image
    
    sub1 = fig.add_subplot(1,2,1)
    sub1.set_title('X-Ray Image with ' + finding)
    sub1.imshow(img_arr, cmap='gray')
    
    ## Plotting Pixel Intensity
    
    sub2 = fig.add_subplot(1,2,2)
    sub2.set_title('Pixel Intensity Distribution of Image with ' + finding)
    plt.hist(img_arr.ravel(), bins=256)
    
    
for file in dicom_files: 
    dcm = pydicom.dcmread(file)
    plot_distribution(dcm, dcm.StudyDescription)<jupyter_output><empty_output><jupyter_text>Note: use full NIH data to perform the first a few EDA items and use sample_labels.csv for the pixel-level assassements.<jupyter_code>sample_df.head()
all_labels = np.unique(list(chain(*sample_df['Finding Labels'].map(lambda x: x.split('|')).tolist())))
all_labels = [x for x in all_labels if len(x)>0]
print('All Labels ({}): {}'.format(len(all_labels), all_labels))
for c_label in all_labels:
    if len(c_label)>1: # leave out empty labels
        sample_df[c_label] = sample_df['Finding Labels'].map(lambda finding: 1.0 if c_label in finding else 0)
sample_df.sample(5)
!ls /data/
def plot_images(img_path):
    
    fig = plt.figure()
    fig.set_figwidth(20)
    
    img = plt.imread(path)
    sub1 = fig.add_subplot(1,2,1)
    sub1.set_title('X-Ray Image with Pneumonia')
    sub1.imshow(img, cmap='gray')
    
    sub2 = fig.add_subplot(1,2,2)
    sub2.set_title('Pixel Intensity Distribution of Image with Pneumonia')
    plt.hist(img.ravel(), bins=256, density=True)
for i, row in sample_df[sample_df['Pneumonia'] == 1].iterrows():
    path = os.path.join("/data/images_002/images/", row['Image Index'])
    if not os.path.exists(path):
        #print('Invalid Path..')
        continue
    else:
        plot_images(path)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10709542781255929, 'fraction_numerical': 0.016125972301271108, 'mean_word_length': 5.268133174791914, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 48, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '540073', 'license': 'MIT', 'n_tokens_mistral': 3602, 'n_tokens_neox': 3356, 'n_words': 1055, 'path': 'Applying AI to 2D Medical Imaging Data/Course Project - Pneumonia Detection from Chest X-Rays/EDA.ipynb', 'repo_name': 'mayank1101/AI-for-Healthcare', 'size': '10542'}