<jupyter_start><jupyter_text>Data preparation 2  Imports<jupyter_code>import os
import pandas as pd<jupyter_output><empty_output><jupyter_text>Constants<jupyter_code>DATA_DIR = 'data/'
CSV_FILE = os.path.join(DATA_DIR, 'nih_chest_xray_single_9c.csv')
BB_CSV_FILE = os.path.join(DATA_DIR, 'BBox_List_2017.csv')<jupyter_output><empty_output><jupyter_text>Download the prepared CSV<jupyter_code>os.makedirs(DATA_DIR, exist_ok=True)

if not os.path.exists(CSV_FILE):
    ! gdown --id 1gJJ5ZRfRicuxIoBWLYAUnfUs4yhJ97sX -O $CSV_FILE
else:
    print('File exists: {}'.format(CSV_FILE))

if not os.path.exists(BB_CSV_FILE):
    ! gdown --id 12fA_VUjnt-tyo1d0M2htyQ5w3-nIAVH_ -O $BB_CSV_FILE
else:
    print('File exists: {}'.format(BB_CSV_FILE))<jupyter_output><empty_output><jupyter_text>Explore the datasetTasks:1. Show `df`'s and `df_bb`'s statistics<jupyter_code>df = pd.read_csv(CSV_FILE)
df_bb = pd.read_csv(BB_CSV_FILE)
## Show `df`s statistics

## Show `df_bb`s statistics<jupyter_output><empty_output><jupyter_text>CleanupTasks:1. Delete unused columns ('Unnamed: 0', 'Follow-up ', 'Patient ID', 'View Position', 'OriginalImage[Width', 'Height]', 'OriginalImagePixelSpacing[x', 'y]'). Resulting `df` should have 4 columns left.1. Remove redundant columns from `df_bb`1. Rename `Bbox [x`	`y`	`w`	`h]` columns to `bb_x`, `bb_y`, `bb_w`, `bb_h` in `df_bb`<jupyter_code>## Delete unused columns ('Unnamed: 0', 'Follow-up #', 'Patient ID',
## 'View Position', 'OriginalImage[Width', 'Height]', 'OriginalImagePixelSpacing[x', 'y]').
## Resulting `df` should have 4 columns left.

## Remove redundant columns from `df_bb`

## Rename `Bbox [x`	`y`	`w`	`h]` columns to `bb_x`, `bb_y`, `bb_w`, `bb_h` in `df_bb`<jupyter_output><empty_output><jupyter_text>Merge both data framesTasks:1. Merge both dataframes<jupyter_code>## Merge both dataframes

## Rescale bounding box coordinates

df.describe()<jupyter_output><empty_output><jupyter_text>Encode categoriesTasks:1. Add one-hot encoded labels to `df`<jupyter_code>## Add one-hot encoded labels to `df`

## combine `onehot` with `df` data frame

df.head()<jupyter_output><empty_output><jupyter_text>Add column with file pathsTasks:1. Add column with file path (data/images/IMAGE_INDEX)<jupyter_code>## Add column with file path (data/images/IMAGE_INDEX)

df.head()<jupyter_output><empty_output><jupyter_text>Save results<jupyter_code>df.to_csv(os.path.join(DATA_DIR, 'nih_chest_xray_single_9c_bb_onehot.csv'), index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1444043321299639, 'fraction_numerical': 0.012033694344163659, 'mean_word_length': 7.659722222222222, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '64864', 'license': 'Apache-2.0', 'n_tokens_mistral': 1023, 'n_tokens_neox': 993, 'n_words': 194, 'path': '1_data_preparation_2.ipynb', 'repo_name': 'issdl/from-data-to-solution-2021', 'size': '2493'}