<jupyter_start><jupyter_text>Covid-19 Chest X-ray DetectorTry to detect Covid-19 from X-rays, the datasets belongs to [covid-chestxray-dataset](https://github.com/ieee8023/covid-chestxray-dataset) and [chest-xray-pneumonia](https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia)There is 2 sources from X-rays because the first one have only X-rays with pathologies, and the second one contain  normal X-rays (an others pathologies, but we will use only the normal x-rays).**Processing dataset** * Get Posteroanterior x-ray images.  * Keep only the fields 'finding', 'folder' and 'filename' * Change folder columns with the complete local path * Define the value to predict in setting "covid-19" and "covid-19, ARDS" field as true. The other will be set as false * Add new x-ray images to complete dataset * Split data into train, test, predict and copy them to a specific directory structure.<jupyter_code># Import necessary packages
import os
import numpy as np
import pandas as pd

from keras.preprocessing.image import ImageDataGenerator
from sklearn.model_selection import train_test_split

import seaborn as sns
import matplotlib.pyplot as plt
sns.set()
# Load dataset from github and kaggle
## Main dataset (https://github.com/ieee8023/covid-chestxray-dataset)
!git clone https://github.com/ieee8023/covid-chestxray-dataset.git
chestxray_images_folder = './covid-chestxray-dataset'
# Setupo to retreive dataset from kaggle
# Full documentation at https://www.kaggle.com/general/74235
!pip install -q kaggle
files.upload()
!mkdir ~/.kaggle 
!cp kaggle.json ~/.kaggle/
!chmod 600 ~/.kaggle/kaggle.json
# Alternate dataset to get more "normal" images
!kaggle datasets download -d paultimothymooney/chest-xray-pneumonia
!unzip chest-xray-pneumonia.zip
chestxray_normal_folder = './chest_xray/train/NORMAL'
# Load data from covid-chestxray-dataset and display 5 first rows
raw_data = pd.read_csv(os.path.join(chestxray_images_folder,'metadata.csv'))
raw_data.head()<jupyter_output><empty_output><jupyter_text>Prepare datasetTo have a omogenous dataset we will keep only from back to front (Posteroanterior) X-rays. Because we are trying to predict covid from X-ray images, we wil keep only the fields 'finding', 'folder' and 'filename'. We also need to set the correct path for the images.<jupyter_code>#  Get only Posteroanterior X-rays 
chestxrays = raw_data[(raw_data['modality'] == 'X-ray') & (raw_data['view'] == 'PA')]
# keep 'finding', 'folder' and 'filename'
col_to_drop = ['patientid', 'offset', 'sex', 'age', 'survival', 'intubated',
       'intubation_present', 'went_icu', 'in_icu', 'needed_supplemental_O2',
       'extubated', 'temperature', 'pO2_saturation', 'leukocyte_count',
       'neutrophil_count', 'lymphocyte_count', 'view', 'modality', 'date',
       'location', 'doi', 'url', 'license',
       'clinical_notes', 'other_notes','RT_PCR_positive']

chestxrays = chestxrays.drop(col_to_drop, axis=1)
# Set correct path according to local installation
chestxrays['filename'] = chestxrays.apply(lambda x : os.path.join(chestxray_images_folder,x.folder,x.filename),axis=1)
chestxrays.drop('folder', axis=1, inplace=True)
chestxrays.head()<jupyter_output><empty_output><jupyter_text>Define value to predictWhen need to predict covid or not covidchange all 'finding' equal to 'COVID-19' to 1 and the others to 0 in a new col 'covid-19'<jupyter_code>chestxrays["finding"].value_counts()
sns.barplot(x=chestxrays["finding"].value_counts().index, y=chestxrays["finding"].value_counts())
plt.xticks(rotation=70)
plt.show()
# add new column 'covid-19' and drop 'finding'
# set 1 for COVID-19 and COVID-19, ARDS and 0 for others values 
chestxrays['covid-19'] = chestxrays['finding'].map(lambda x:1 if (x == 'COVID-19' or x == 'COVID-19, ARDS') else 0)
chestxrays = chestxrays.drop(['finding'], axis=1)
sns.barplot(x=chestxrays["covid-19"].value_counts().index, y=chestxrays["covid-19"].value_counts())
plt.show()
chestxrays["covid-19"].value_counts()
chestxrays.head()
def display_random_images(df, nb_images):
    """Display x random images from dataset
    """
    random_index = [np.random.choice(df.index) for i in range(nb_images)]        
    plt.figure(figsize=(20,10))
    for i in range(nb_images):        
        image = df.iloc[i]['filename']     
        plt.subplot(3, 3, i + 1)
        img = plt.imread(image)
        plt.imshow(img, cmap='gray')
        plt.axis('off')    
    plt.tight_layout()  
display_random_images(chestxrays, 3)<jupyter_output><empty_output><jupyter_text>Add new datasetAdd more non-covid chest X-rays from https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia in directory train/NORMAL<jupyter_code>nb_images_to_add = chestxrays["covid-19"].value_counts()[1] - chestxrays["covid-19"].value_counts()[0]
print(f"Number of normal images to add {nb_images_to_add}")
images = []
files = os.listdir(chestxray_normal_folder)
for name in files:
    images.append(name)
    
# Select randomly X images and add to new dataset
random_images = [np.random.choice(images) for i in range(nb_images_to_add)]    
column_names = ["filename","covid-19"]
chests = pd.DataFrame(columns = column_names)

for image in random_images:
    new_row = pd.Series(data={'filename': os.path.join(chestxray_normal_folder,image), 'covid-19':0})
    chests = chests.append(new_row, ignore_index=True)

    
display_random_images(chests, 3)<jupyter_output><empty_output><jupyter_text>Organize data<jupyter_code># Merge thge datasets
chestxrays = pd.concat([chestxrays, chests])
chestxrays["covid-19"].value_counts()
# Save as CSV
chestxrays.to_csv('./chestxrays.csv', index=False)
import shutil
import os.path 
def cleanDestination(destination):
    """Delete folder and subdirectories
    """
    try:
        shutil.rmtree(destination)
    except OSError:
        pass
def copyData(source, destination, new_filename=""):
    """
    Copy file to specified directory. Create destination directory if needed
    """
    try :
        os.makedirs(destination)
    except OSError:
        pass
    
    if new_filename:
        destination = os.path.join(destination,new_filename)   
    else:
        destination = os.path.join(destination,os.path.basename(source))   
        
    shutil.copyfile(source, destination)       
        
# Split data into train, test and predict directories
train_dir = 'data/train/'
test_dir  = 'data/test/'
pred_dir  = 'data/predict/'

cleanDestination(train_dir)
cleanDestination(test_dir) 
cleanDestination(pred_dir) 
    
train_test,predict = train_test_split(chestxrays, test_size=0.1, random_state=42)
train, test = train_test_split(train_test, test_size=0.2, random_state=42)

# Predict data
for index, row in predict.iterrows():
    filename = str(row['covid-19']) + '-' + os.path.basename(row['filename'])
    copyData(row['filename'], pred_dir, filename )
    
# Train data
for index, row in train.iterrows():
    destination = train_dir + '0_normal'
    if row['covid-19'] == 1:
        destination = train_dir +  '1_covid'
    copyData(row['filename'], destination)

# Test data
for index, row in test.iterrows():
    destination = test_dir + '0_normal'
    if row['covid-19'] == 1:
        destination = test_dir + '1_covid'
    copyData(row['filename'], destination)

from google.colab import files
!zip -rq data.zip data
files.download( "data.zip" )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12433754586220953, 'fraction_numerical': 0.014675907052588666, 'mean_word_length': 4.777080062794348, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 6, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '49780', 'license': 'MIT', 'n_tokens_mistral': 2613, 'n_tokens_neox': 2484, 'n_words': 617, 'path': 'prepare_data.ipynb', 'repo_name': 'PierreSylvain/covid-chestxray-detector', 'size': '7359'}