<jupyter_start><jupyter_text>In order to store sequences from the DICOM, we created a JSON. We will load in that JSON now.<jupyter_code># load json
with gzip.open('dicom-metadata.json.gz', 'r') as fp:
    tmp = json.load(fp)

dcm_metadata = dict()
# convert from length list of 1 item dicts to single dict
for d in tmp:
    for k, v in d.items():
        dcm_metadata[k] = v
        
del tmp

# figure out how many unique top level meta-data fields in the json
# also get a list of all the top level tags
json_keys = [list(dcm_metadata[x].keys()) for x in dcm_metadata]
json_keys = set([int(item) for sublist in json_keys for item in sublist])
json_keys = list(json_keys)
json_keys.sort()

n_attrib = len(json_keys)
print(f'There are {n_attrib} top-level attributes in the DICOM json.')

# show an example
dcm_metadata['000046e4-e4d7f796-72c3dba4-8b67a485-0eea211d']<jupyter_output>There are 10 top-level attributes in the DICOM json.<jupyter_text>There are two very useful items in this sequence that we'd like to have in an easier form for all images: the procedure code sequence (`'528434'`), the coded view position (`'5505568'`), and the coded patient orientation (`'5506064'`). For convenience, we will pull the textual description of each (`'524548'`), rather than the ontology code itself.<jupyter_code>cols = ['528434', '5505568', '5506064']
dcm_metadata_simple = {}
for k, v in dcm_metadata.items():
    dcm_metadata_simple[k] = [v[c][0]['524548']
                              for c in cols
                              if c in v and len(v[c])>0]
dcm_metadata_simple = pd.DataFrame.from_dict(dcm_metadata_simple, orient='index')

# convert columns to be human readable
dcm_metadata_simple.columns = [DicomDictionary[int(c)][-1] + '_' + DicomDictionary[int('524548')][-1]  for c in cols]
dcm_metadata_simple.head()
metadata.head()
# load in MIMIC-CXR 2.0.0 record list
records =  pd.read_csv(mimic_cxr_path / 'cxr-record-list.csv.gz')
records.set_index('dicom_id', inplace=True)

# load in a CSV of meta-data derived from MIMIC-CXR
metadata = pd.read_csv('dicom-metadata.csv.gz', index_col=0)
metadata.index.name = 'dicom_id'

# subselect to useful metadata
metadata = metadata[['4194900', '1593601', '2621456', '2621457', '524320', '524336', '1577984']]

# rename columns to be human readable
metadata.columns = [DicomDictionary[int(c)][-1] for c in metadata.columns]

# merge into records
metadata = records[['subject_id', 'study_id']].merge(
    metadata, how='left', left_index=True, right_index=True
)

# add in the metadata from the JSON file
metadata = metadata.merge(
    dcm_metadata_simple, how='left', left_index=True, right_index=True
)
metadata.head()
metadata.sort_values(['subject_id', 'study_id'], inplace=True)
metadata.to_csv('mimic-cxr-2.0.0-metadata.csv.gz', index=True, compression='gzip')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1241669589617678, 'fraction_numerical': 0.051210101718695195, 'mean_word_length': 4.7269076305220885, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '437042', 'license': 'MIT', 'n_tokens_mistral': 1085, 'n_tokens_neox': 994, 'n_words': 290, 'path': 'dcm/create-mimic-cxr-jpg-metadata.ipynb', 'repo_name': 'farrell236/mimic-cxr', 'size': '2851'}