<jupyter_start><jupyter_text>**Title**: Upload COVID Chest X-Ray Dataset.   **Date**:  12-Oct-2020     **Description**:With COVID-19 one of the most pressing issues of 2020, the availability of well-curated datasets becomes essential for highly distributed and internationally collaborative research. To this end, <NAME> (Mila, University of Montreal) has collated a dataset of almost a thousand chest x-rays from other public datasets.This notebook downloads the COVID Chest X-Ray Dataset and incorporates it into a Flywheel instance specified by the supplied API-Key.   References:   * https://github.com/ieee8023/covid-chestxray-dataset* https://arxiv.org/abs/2004.12823* https://arxiv.org/abs/2004.05405  Data Use AggreementBefore downloading this data, or any data, make sure you understand the restrictions on the use of data.  Requirements:- **Python** (Preferably >= 3.6):  - Have administrator permissions to create Flywheel Groups and Projects.<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Install and import dependencies<jupyter_code>!pip install pandas pydicom flywheel-sdk
import copy
import csv
import datetime
import logging
import os
import time
from getpass import getpass

import flywheel
import pandas as pd
# Instantiate a logger
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
log = logging.getLogger('root')<jupyter_output><empty_output><jupyter_text>Download covid-chestxray datasetThe following will download the entire github repository to the local drive.<jupyter_code>!git clone <EMAIL>:ieee8023/covid-chestxray-dataset.git<jupyter_output><empty_output><jupyter_text>Initialize Constants<jupyter_code>ROOT_CHESTXRAY_DATA = '/path/to/downloaded/repository/covid-chestxray-dataset'
DEFAULT_SESSION_LABEL = 'offset_NA'
DEFAULT_ACQ_LABEL = 'Chest XR'<jupyter_output><empty_output><jupyter_text>Flywheel API Key and ClientGet an API_KEY. More on this in the Flywheel SDK doc [here](https://flywheel-io.gitlab.io/product/backend/sdk/branches/master/python/getting_started.htmlapi-key).<jupyter_code>API_KEY = getpass('Enter API_KEY here: ')<jupyter_output><empty_output><jupyter_text>Instantiate the Flywheel API client<jupyter_code>fw_client = flywheel.Client(API_KEY if 'API_KEY' in locals() else os.environ.get('FW_KEY'))<jupyter_output><empty_output><jupyter_text>Show Flywheel logging information<jupyter_code>log.info('You are now logged in as %s to %s', fw_client.get_current_user()['email'], fw_client.get_config()['site']['api_url'])<jupyter_output><empty_output><jupyter_text>Container helpersImport container helper functions to find existing or create new containers.<jupyter_code>from container_helpers import (
    find_or_create_group, 
    find_or_create_project, 
    find_or_create_subject, 
    find_or_create_session, 
    find_or_create_acquisition,
    upload_file_to_acquisition
)<jupyter_output><empty_output><jupyter_text>Find or Create Group and Project:Create a group with id "public_data" and label "public_data".Create a project with label 'covid-chestxray-dataset'.Replace with the id and labels of the group and project you want to create.If group and project are already created, the group and project with specified labels will be returned.<jupyter_code># Initialize the group
public_data_group = find_or_create_group(fw_client, 'public_data', 'public_data')

# Initialize the project
project_label = 'covid-chestxray-dataset'
chestxray_project = find_or_create_project(project_label, public_data_group)

# Parse the project description from the README markdown
with open(
    os.path.join(ROOT_CHESTXRAY_DATA, 'README.md'),
    'r',
    encoding='utf-8'
) as fdata:
    readme = fdata.read()
    
for local_file in ['(CONTRIBUTING.md)', '(metadata.csv)', '(images)']:
    readme = readme.replace(
        local_file, 
        '(https://github.com/ieee8023/covid-chestxray-dataset/blob/'
        f'47685d20a1b77f0664ac8b7740ef6d4646be962d/{local_file[1:-1]})'
    )

if chestxray_project:
    chestxray_project.update(description=readme)<jupyter_output><empty_output><jupyter_text>Prepare Dataframe Read the csv, clean the dataframe, produce dictionary list  Formating and Cleaning Helper Functions<jupyter_code>from dataframe_helpers import (
    convert_time_to_seconds, 
    format_sex_string, 
    create_session_label, 
    cleanup_row_dict
)<jupyter_output><empty_output><jupyter_text>Apply Helper Functions to DataframeConvert rows of metadata csv into FW-specific values and a row-dictionary to incorporate into each acquisition file. Columns of metadata csv are as such:```patientid,offset,sex,age,finding,RT_PCR_positive,survival,intubated,intubation_present,went_icu,in_icu,needed_supplemental_O2,extubated,temperature,pO2_saturation,leukocyte_count,neutrophil_count,lymphocyte_count,view,modality,date,location,folder,filename,doi,url,license,clinical_notes,other_notes,```<jupyter_code># Load dataframe from downloaded csv file
df = pd.read_csv(os.path.join(ROOT_CHESTXRAY_DATA, 'metadata.csv'))

# format subject label
df['subject_label'] = df['patientid'].apply(lambda x: f'sub_{str(x).zfill(4)}')
row_dict_list = df.to_dict(orient='records')

# Apply age conversion
df['session_age'] = (
    df['age'].apply(convert_time_to_seconds, scale='Y') + 
    df['offset'].apply(convert_time_to_seconds, scale='D').astype('int64')
)

# Format subject sex
df['subject_sex'] = df['sex'].apply(format_sex_string)

# Apply to session labels with default 
df['session_label'] = df['offset'].apply(
    create_session_label, 
    default_session_label=DEFAULT_SESSION_LABEL
)

# format acquisition label
df['acquisition_label'] = df['filename'].apply(lambda x: x.rsplit('.', maxsplit=1)[0])

# throw out nans
df.fillna('', inplace=True)

# Produced cleaned row dictionary list
row_dict_list = [cleanup_row_dict(row_dict) for row_dict in df.to_dict(orient='records')]<jupyter_output><empty_output><jupyter_text>Create containers Iterate through dataframe and uploadIterate through the metadata csv to create the container hierarchy for this project:1. find or create each subject encountered  a. Encode sex of subject into dictionary and incorporate into subject metadata.2. find or create each session (with `DEFAULT_SESSION_LABEL`) encountered  a. Encode patient age at session date into dictionary and incorporate into session metadata.3. find or create each acquisition (with 'SeriesDescription' or `DEFAULT_ACQ_LABEL`) and add enclosed files.  a. Incorporate `row_dict` of each metadata row into the metadata of the acquisition and acquisition file.<jupyter_code># iterate through rows of dataframe
for row_dict in row_dict_list:
    # (1-1a) Find or create subject with metadata
    subject_label = row_dict.get('subject_label')
    log.info('Processing Subject %s.', subject_label)
    subject_sex = row_dict.get('subject_sex')
    kwargs_dict = {"sex": subject_sex}
    subject = find_or_create_subject(subject_label, chestxray_project, **kwargs_dict)
    if subject:
        # (2-2a) Find or create subject with age metadata
        session_label = row_dict.get('session_label')
        log.info('Processing Session %s.', session_label)
        age_at_session = row_dict.get('session_age')
        kwargs_dict = {"age": age_at_session}
        session = find_or_create_session(session_label, subject, **kwargs_dict)
        if session:
            # (3-3a) Find or create acquisition 
            aqc_label = row_dict.get('acquisition_label')
            log.info('Processing Acquisition %s.', aqc_label)
            # (3a) Metadata for acquisition and acquisition file
            kwargs_dict = {"info": row_dict}
            # (3) Find or create acquisition with (3a) incorporating metadata
            acq = find_or_create_acquisition(aqc_label, session, **kwargs_dict)
            filepath = os.path.join(ROOT_CHESTXRAY_DATA, 'images', row_dict.get('filename'))
            log.info('Uploading file, %s, to acquisition, %s', filepath, acq.label)
            try:
                #(3) Add enclosed file with (3a) prepared metadata from csv row
                upload_file_to_acquisition(acq, filepath, **kwargs_dict)
            except ValueError as e:
                log.warning("File, %s, is not found.", filepath)
# list unique subjects
df.subject_label.unique()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09666626837137053, 'fraction_numerical': 0.011470904528617517, 'mean_word_length': 5.438461538461539, 'pattern_counts': {'":': 3, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '252624', 'license': 'MIT', 'n_tokens_mistral': 2685, 'n_tokens_neox': 2552, 'n_words': 667, 'path': 'import/covid_chestxray_dataset.ipynb', 'repo_name': 'flywheel-apps/flywheel-tutorials', 'size': '8387'}