<jupyter_start><jupyter_text>Data Preparation Example: MIMIC-CXR What is MIMIC-CXR?[MIMIC-CXR](https://mimic-cxr.mit.edu/) contains 227,835 imaging studies for 64,588 patients presenting to the Beth Israel Deaconess Medical Center Emergency Department between 2011 - 2016. A total of 377,110 images are available in the dataset. Each imaging study can contain one or more images, usually a frontal view and a lateral view. Studies are made available with a semi-structured free-text radiology report that describes the radiological findings of the images, written by a practicing radiologist contemporaneously during routine clinical care. All images and reports have been de-identified to protect patient privacy.The current version of the database is v2.0.0. When referencing this version, we recommend using the full title: MIMIC-CXR v2.0.0.MIMIC-CXR v2.0.0 is the first release of the images in their native format, DICOM, and the first release of the free-text radiology reports associated with these images. All data has been de-identified prior to release to protect patient privacy. Requesting access to MIMIC-CXRMIMIC-CXR contains information regarding the clinical care of ICU patients, so it must be treated with appropriate care and respect. In order to access the images and reports, researchers must be credentialed on PhysioNet. The credentialing process involves completion of a training course, described in more detail below. Users who are already credentialed on PhysioNet do not need to repeat the credentialing process.If you are already credentialed, you may immediately access the dataset by logging into PhysioNet signing the data use agreement on the project page: https://physionet.org/content/mimic-cxr/Otherwise, follow the process outlined below to become credentialed. The process takes around 2-3 hours of time from yourself, and usually a week for our team to process. CredentialingA user must be credentialed in order to access the database, as appropriate training is necessary for the handling of human subjects in research. Credentialing requires completion of a training course, as follows:Register on the CITI program website, selecting “Massachusetts Institute of Technology Affiliates” as your organization affiliation: https://www.citiprogram.org/index.cfm?pageID=154&icat=0&ac=0Follow the links to add a Massachusetts Institute of Technology Affiliates course. In the Human Subjects training category, select the “Data or Specimens Only Research” courseComplete the course and save a copy of your certificate.Once completed, register an account on PhysioNet. Navigate to the credentialing page, and apply for credentialed privileges: https://physionet.org/settings/credentialing/  Download dataFirst, download the data from PhysioNet and unzip `cxr-record-list.csv.gz` and `cxr-study-list.csv.gz`. Then we can start the preparation progress.  Extract the dataImport the Modules.<jupyter_code>import os
import pandas as pd
import numpy as np
import pydicom
import matplotlib.pyplot as plt
import cv2<jupyter_output><empty_output><jupyter_text>Set the path as to where you put your data.<jupyter_code>cur_path = "D:/MIMIC-CXR/physionet.org/files/mimic-cxr/2.0.0/"
os.chdir(cur_path)<jupyter_output><empty_output><jupyter_text>Read and join two tables based on the `subject_id` and `study_id`.<jupyter_code>cxr_study_list = pd.read_csv(cur_path+'cxr-study-list.csv')
cxr_record_list = pd.read_csv(cur_path+'cxr-record-list.csv')
cxr_study_list
cxr_record_list
merged = cxr_study_list.merge(cxr_record_list, how='inner', left_on=['subject_id','study_id'], right_on=['subject_id','study_id'])
merged<jupyter_output><empty_output><jupyter_text>Define a function which can extact `FINDING` and `IMPRESSION` part of the input file.<jupyter_code>def extract_finding_impression(origin_file):
    fp = open(origin_file)
    lines = fp.readlines() 
    mark = 0
    find_impression = []
    for line in lines:
        # print(line)
        if 'FINDINGS:' in line:
            mark = 1
            continue
        elif 'IMPRESSION:' in line:
            mark = 1
            continue
        elif ('EXAMINATION:' in line) or ('INDICATION:' in line) or ('TECHNIQUE:' in line) or ('COMPARISON:' in line):
            mark = 0
        else:
            if mark == 1:
                find_impression.append(line)

    fp.close()

    find_impression_str = ''
    for i in find_impression:
        i = i.replace('\n', '')
        find_impression_str += i

    # name = origin_file.split('/')[-1]
    # f = open(target_file, 'w+')
    # f.write(find_impression_str)
    # f.close()

    return find_impression_str<jupyter_output><empty_output><jupyter_text>Extact text content for all of the image.<jupyter_code>paired_data = pd.DataFrame(columns=['image_path', 'text_content'])
sample_count = 0

for index, row in merged.iterrows():
    image_raw = row['path_y']
    text_file = row['path_x']

    if (not os.path.exists(image_raw)) or (not os.path.exists(text_file)):
        print('file not exist.')
        continue

    content = extract_finding_impression(text_file)

    # merged.loc[index,'text_content'] = content

    image_out_path = 'D:/MIMIC-CXR/image/' + image_raw.split('/')[-1].split('.')[0] + '.jpg'
    ds = pydicom.read_file(image_raw) 
    img = ds.pixel_array 
    # plt.imshow(img,  cmap='gray')
    # plt.show()
    newimg = (img - np.min(img)) / (np.max(img) - np.min(img)) 
    newimg = (newimg * 255).astype('uint8')
    cv2.imwrite(image_out_path,newimg)

    paired_data = paired_data.append({'image_path':image_out_path, 'text_content':content},ignore_index=True)

    sample_count += 1
    if sample_count > 10000:
        break
    print(str(sample_count)+' - '+image_out_path)
        
paired_data<jupyter_output><empty_output><jupyter_text>Save the table as a CSV file.<jupyter_code>paired_data.to_csv('D:/MIMIC-CXR/image_path_text.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08929777177582715, 'fraction_numerical': 0.010972316002700877, 'mean_word_length': 4.347472924187725, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '283324', 'license': 'MIT', 'n_tokens_mistral': 1866, 'n_tokens_neox': 1768, 'n_words': 594, 'path': 'data_prepare_MIMIC.ipynb', 'repo_name': 'waittim/ConVIRT-Colab', 'size': '5924'}