<jupyter_start><jupyter_text>DICOManager Tutorial OverviewDICOManager is designed to sort, reconstruct, process and convert DICOM files to numpy arrays for use in Machine Learning and Deep Learning. SortingDICOManager begins with sorting DICOMs into a file tree with the following heirarchy:1. Cohort2. Patient3. Frame Of Reference4. Study5. Series6. Modality7. DICOM fileFile tree construction is automatic and can be called at any level using `groupings.(files=)`For example:<jupyter_code>from glob import glob
import groupings

files = glob("/list/to/dicoms/*.dcm")
cohort = groupings.Cohort(name="MyCohort", files=files)
print(cohort)<jupyter_output>Cohort: MyCohort<jupyter_text>If a grouping should only contain a subset of the dicoms in a directory, a filter_list can be used, where each parameter is specified as a list or dictionary.If no parameter is specified in the dictionary, dicoms will not be flitered by that field. For the parameter of struct name, either a list or dictionary can be provided. If a list is used, structures matching names within the list will be included. If a dict is provided, structures in each list will be mapped to the corresponding key.For example, below we create a filtering dictionary, titled filter_by, which specifies PatientID, Study date and Structure names which we want to filter the group by. Additionally, in this data set each StudyInstanceUID includes only a single image series, so we specify `include_series=False` to remove an unnecessary additional layer in the file tree.<jupyter_code>filter_by={"PatientID": ['1', '2', 'N'],
           "StudyDate": ["19700101", "20211231"],
           "SturctName": {"Struct1": ["Struct1", "Misspelled1", "AlternateName1"],
                          "Struct2": ["Struct2"]}}

cohort = groupings.Cohort(name="MyCohort", files=files, filter_by=filter_by, include_series=False)<jupyter_output><empty_output><jupyter_text>For data sets of unknown completeness, filtering the group is of limited usefulness. Instead we want to only preserve groups which are "complete". We can check for completenesswith the function `.pull_incompletes`. This function takes a filter_by dictionary and returns a grouping of the excluded data. For example, if we wanted to specify that a complete node is a Frame Of Reference with at least one CT and one RTSTRUCT, we would use the following syntax. The result is that cohort has all incomplete leaves removed and returned as a separate tree.Note: Because reconstruction happens at the Frame Of Reference level, it is most logical to iterate through the tree and remove incomplete frames of reference.<jupyter_code>filter_by = {"Modality": ["CT", "RTSTRUCT"]}

incompletes = cohort.pull_incompletes(group="FrameOfRef", contains=filter_by)<jupyter_output><empty_output><jupyter_text>In the previous example, if a Frame Of Reference had 2 CT and 1 RTSTRUCT, it would be considered complete because it contains at least one of each of the specified modalities. But,take, for example, a cohort where we want the number of modalities to be exactly 2 MR, 1 CT, 1 RTSTRUCT for a given patient. We can instead specify completeness as follows:<jupyter_code>filter_by = {"Modality": ["CT", "RSTRUCT", "MR", "MR"]}
incompletes = cohort.pull_incompletes(group="Patient", contains=filter_by, exact=True)<jupyter_output><empty_output><jupyter_text>Finally, once we have the tree sorted as we desire, we can save the DICOM tree to a specified location.<jupyter_code>cohort.save_tree(path='/path/to/save/directory')<jupyter_output><empty_output><jupyter_text>Upon each grouping level, the following basic functions can be used to alter the organization of the file tree, with `tree` refering to the current tree, `->` refering to the returned object:- `tree.merge(other) -> None`: Merge two trees- `tree.steal(other) -> None`: Moves a child of one tree to another tree- `tree.append(other) -> None`: Appends a node to another tree- `tree.prune(childname) -> None`: Removes a child from the tree- `tree.adopt(child) -> None`: Adopts a child to the current tree- `tree.flatten() -> None`: Restricts each parent to having one child within the tree- `tree.pop() -> `: Removes the first child from the tree and returns it- `save_dicoms(filepath) -> None`: Saves only the dicoms from the tree- `save_volumes(filepath) -> None`: Saves only the reconstructed volumes from the tree- `only_dicoms() -> bool`: Returns true if tree only contains dicoms- `has_dicoms() -> bool`: Returns true if tree contains dicoms- `has_volumes() -> bool`: Returns true if tree contains volumes- `iter_modalities() -> iterator`: Returns iterator of modalities within the tree- `iter_frames() -> iterator`: Returns iterator of frames of reference within the tree- `iter_dicoms() -> iterator`: Returns iterator of lists of dicoms for each series- `iter_volumes() -> iterator`: Returns iterator of each volume file within the tree- `iter_volume_date() -> iterator`: Returns iterator of each Modality.volumes_data dictionary within the tree- `iter_volumes_frames() -> iteratorr`: Returns an iterator of all volumes within each frame of references- `clear_dicoms() -> None`: Removes all dicoms from a tree- `clear_volumes() -> None`: Removes all volumes from a tree- `split_tree() -> tuple`: Splits the tree into a dicom only and volume only tree- `split_dicoms() -> tree`: Returns only a tree of dicoms, removes dicoms from source tree- `split_volumes() -> tree`: Returns only a tree of volumes, removes volumes from source tree- `volumes_to_pointers() -> None`: Converts all volumes to pointers, writing the arrays to disk- `pointers_to_volumes() -> None`: Converts all pointer to volumes, loading the arrays into memory- `recon(in_memory=False, parallelize=True) -> None`: Reconstructs all DICOMs within the tree, loading into memory or writing to disk at ~/.- `apply_tool(toolset, path=None, mod=None, warn=True) -> None`: A single threaded application of a list of tool functions, as specified in toolset- `apply_tools(toolset, path=None, nthreads=None) -> None`: Multithreaded application of a list of tool functions applied to the volumes within the tree- `pull_incompletes(group='Patient', exact=False, contains=None) -> type(self)`: Determines if a group is incomplete as defined by contains, returns the excluded groups from the tree. - `remove_empty_branches(prune_modalities=False) -> None`: Removes any empty branches from a tree. An empty branch is defined as a branch with no children. By default, modalities are excluded from the empty exclusion.Original tree is modified in place.An example for a few is below:<jupyter_code># Checking if the cohort contains dicoms
print(cohort.has_dicoms())
# Flatten all patients
for patient in cohort:
    patient.flatten()
# Adopting the first 3 patients into a new cohort
new_cohort = groupings.Cohort(name='NewCohort', files=None)

for _ in range(3):
    patient = cohort.pop()
    new_cohort.adopt(patient)

print(cohort)
print(new_cohort)<jupyter_output><empty_output><jupyter_text>Reconstruction (DICOM to numpy)Reconstructing DICOMs into numpy arrays suitable for AI can be time consuming and tedious. DICOManager quickly converts DICOMs into volumes using parallelized processes. The simpliest way to reconstruct a patient or cohort is using the `.recon()` function. This function supports reconstruction of CT, MR, PET, NM, RTSTRUCT and RTDOSE files. Default behavior is to write the reconstructed volumes to disk and place a pointer within the tree indicating the volume location. If `.recon(in_memory=True)`, then the volume will be stored in the tree. This process is slower, does not allow for parallelization and can quickly consume an entire systems memory, but it is ideal for single patient inference or usage on systems where read-write access to disk is restricted. In some applications, parallelization across all CPU cores is not desired, in which case `.recon(parallelized=False)` can be used, at the expense of reconstruciton runtime. An example of reconstruction of our cohort is:<jupyter_code>new_cohort.recon(path='/path/to/save/volumes')
print(new_cohort)

new_cohort.pointers_to_volumes()
print(new_cohort)<jupyter_output><empty_output><jupyter_text>Image ManipulationImage manuplation of reconstructed volumes can be conducted on volumes which are stored within memory (for now). These image manuplations are then stored in the header fo the ReconstructedFile or ReconstructedVolume objects. The image manuplation functions, within `dicomanager.processing.tools` are:- `BiasFieldCorrection() -> Reconstructed{Volume, File}` Applies N4 Bias Field Correction to MR image volumes- `Crop(crop_size, centroid=None, centroids=None) -> ReconstructedVolume` Crops the volume to dimensions of crop_size around the center of the volume if no centroid(s) are specified. &emsp;**centroid (list):** A single, common centroid &emsp;**centroids (dict):** A dictionary with a centroid at each frame of reference UID can be given- `Normalize() -> Reconstructed{Volume, File}` Normalizes the reconstructedVolume- `Resample(ratio=None, dims=None, voxel_size=None, dz_limit=None, dz_goal=None, dz_exact=None) -> ReconstructedVolume`  Resamples the image by either a specified ratio, dimensions, voxel size or desired slice thickness (dz). &emsp;**ratio (list):** Resampling ratio, either per-dimension or uniform with r>1 upsampling and r&emsp;**dims (list):** Number of voxels per-dimension to resmaple to, with dimensions of None left unsampled. (e.g. [512, 512, None]). Defaults to None. &emsp;**voxel_size (list):** Voxel size, in mm, to resample image. Defaults to None. &emsp;**dz_limit (float):** Limited slice thickness, with dz > dz_limit being resampled to dz_goal or 1/2 slice thickness otherwise. Defaults to None. &emsp;**dz_goal (float):** Resampling goal if dz_limit is specified. Defaults to None. &emsp;**dz_exact (float):** Resamples all images to this exact dz, if specified. Will override dz_limit and dz_goal. Defaults to None. - `Standardize() -> Reconstructed{Volume, File}`  Standardizes the reconstructedVolume- `WindowLevel(window, level) -> Reconstructed{Volume, File}`  Window and levels the reconstructed volume array. &emsp;**window (float):** Window width &emsp;**level (float):** Level value- `compute_centroids(tree, method, modalities=None, structures=None, struct_filter=None, volume_filter=None, nthreads=None, offset_fn=None) -> dict`  Multithreaded computation of the centroid for each frame of reference. Returns dictionary keyed to FrameOrReferenceUID with centriod voxel location as value &emsp;**tree (NodeMixin):** Tree to iterate through. &emsp;**method (object):** Function to calculate centroid. Takes an array, returns a list of N items corresponding to the centroid in each axis. &emsp;**modalities (list, optional):** Structure name to use for centroid. Defaults to None. &emsp;**structures (list, optional):** List of str to calculate the centroid. Defaults to None. &emsp;**struct_filter (object, optional):** A custom filtering function to pick structures, this is overriden if the paremeter structures is also specified. Function should accept a structure name and return a boolean for calculation of centroid. Defaults to None. &emsp;**volume_filter (object, optional):** A custom filtering function to pick volumes, this is overriden if the parameter structures is also specified. Function should accept a ReconstructedVolume object and return a boolean for calculation of centroid. Defaults to None. &emsp;**nthreads (int, optional):** Number of threads to use for the computation. Higher threads may run faster with higher memroy usage. Defaults to CPU count // 2. &emsp;**offset_fn (object, optional):** A function to offset the centroid. Should accept a the centroid and a ReconstructedVolume object and return a centroid of equivalent dimensions. Defaults to None.  For our working cohort, say we intend to process the images prior to application in a deep learning pipeline. We can process them with the following order of tools:1. We interpolate the image for any missing slices during reconstruction2. Resample the images to 512x512 axial voxels while maintaining the same number of z-axis slices3. Window and level the image to window=500, level=2504. Normalize the image5. Crop the image around its center to 100x100x100 voxels<jupyter_code>from DICOManager import tools

toolset = [tools.Interpolate(),
           tools.Resample(dims=[512, 512, None]),
           tools.WindowLevel(window=500, level=250),
           tools.Normalize(),
           tools.Crop(crop_size=[100, 100, 100])]

cohort.apply_tools(toolset)<jupyter_output><empty_output><jupyter_text>We can also make these operations more powerful if needed.1. Interpolate the image, but allow for extrapolation if missing slices are along the exterior of the image2. Resample the image to 512x512 in the axial dimensions. If the slice thickness (dz) exceeds 2.39mm, we resample it by 0.5x (doubling number of axial slices)3. Normalize the imageThen we can apply these opperations, same as before.But, say we want to now compute a cropping centroid not at the center of the volume but instead defined by a structure. We should compute these values followingany resampling and interpolation to ensure our voxel location cropping centroid is not shifted by resampling.1. We can calculate the centroid for each frame of reference based on the specified structure name (or list of names, using whichever is found first), with thecentroid computed using a specified method. If a custom method is specified, the method should take a numpy array as input and return a list of voxel locations.2. We can then apply these computed centroids during the cropping operation<jupyter_code>from scipy.ndimage import center_of_mass

# Compute the centroids for cropping

toolset = [tools.Interpolate(extrapolate=True),
           tools.Resample(dims=[512, 512, None], dz_limit=2.39),
           tools.Normalize()]

cohort.apply_tools(toolset)

# Then compute the cropping size on the newly interpolated / resampled images
centroids = tools.calculate_centroid(tree=cohort, modalities='RTSTRUCT', structure='body', method=center_of_mass)
cropping = [tools.Crop(crop_size=[100, 100, 100], centroids=centroids)]

cohort.apply_tools(cropping)<jupyter_output><empty_output><jupyter_text>Deconstruction (Numpy to RTSTRUCT)In some instances, particularly inference, conversion from boolean numpy array to an RTSTRUCT is desired. For this to be possible, the associated image DICOM files must be contained within the tree. Deconstruction is only possible at the Frame Of Reference level, or lower, as the RTSTRUCT must contain the equivalent dimensions to a CT group. If a Frame of Reference contains more than one CT, deconstruction is not currently supported. Deconstruction can occur as:- `from_ct()`: Creates a new RTSTRUCT from a CT files based upon the CT header- `from_rt()`: Creates a new RTSTRUCT from an existin RTSTRUCT file- `to_rt()`: Appends a segmentation to an existing RTSTRUCT fileAn example of deconstruction is provided below:<jupyter_code>import numpy as np

frame = new_cohort.iter_frames()[0]  # One frame of reference from the cohort
vol = frame.iter_volumes()[0]  # One volume from the frame of reference
rtstruct = np.zeros((1, vol.shape))  # Create an example array for demonstration

print(f'before: {frame}')
frame.decon.from_ct(rtstruct)
print(f'after: {frame}')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07997411840828211, 'fraction_numerical': 0.008605629246198641, 'mean_word_length': 5.437317784256559, 'pattern_counts': {'":': 7, '<': 40, '<?xml version=': 0, '>': 80, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '256109', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4385, 'n_tokens_neox': 4136, 'n_words': 1933, 'path': 'tutorial.ipynb', 'repo_name': 'evan-porter/DICOManager', 'size': '15455'}