<jupyter_start><jupyter_text>DICOM to Nifti Data Conversion > This code replies upon Linux with a GPU accelerator. It is intended to be utilized in a Google Colab Enviroment. The purpose of this code is to provide a walkthrough of the steps needed to convert DICOM (.dcm) collections to a Nifti format (niix.gz) for analysis using a modified version of the Sarcopenia-AI tool for identifying L3 vertabrae.  Set Up  Establish connection to driveMount your Google Drive - Your drive must be mounted prior to accessing the data. While mounting your drive, you will be required to follow a link and confirm that this notebook may access your drive.<jupyter_code>from google.colab import drive
drive.mount('/content/gdrive')
%cd '/content/gdrive/MyDrive/L3-Clean/'<jupyter_output>/content/gdrive/.shortcut-targets-by-id/1YdeMHAwQy3TZggu1P3vt94i0Ff14D0TR/L3-Clean<jupyter_text>Complete Set Up If users have followed the model structure instructions, the remainder of the setup may be run as one piece.<jupyter_code>#Installing dcm2niix 
!sudo apt-get install pigz
!sudo apt-get install dcm2niix

!dcm2niix -h
!pigz -h
#Importing Libraries
import os 
import shutil<jupyter_output>Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  pigz
0 upgraded, 1 newly installed, 0 to remove and 31 not upgraded.
Need to get 57.4 kB of archives.
After this operation, 259 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Fetched 57.4 kB in 0s (144 kB/s)
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin: 
Selecting previously unselected package pigz.
(Reading database ... 160983 files and directories currently installed[...]<jupyter_text>Data Processing The following code will analyze data in the folder you point to.  *Note: This requires that the only thing in the folder you point to is sub folders at the patient level. Anything else such as a metadata.csv file (part of TCIA files) will cause errors. Relocate such files prior to processing* Step 1: Create set input parameters This step copies the file structure of DICOM directory you direct it to. It is .... to finish later KLJ<jupyter_code>#file name in the data folder
collection = 'Anti-PD-1_Lung_Subset'

import shutil
import os 

#Getting the working directory name 
main = os.getcwd()

#Setting Path Names for 
directory = os.path.join(main+"/Data/"+collection) 
prep = os.path.join(main+"/NII_Data/"+collection)
print(prep)
print(directory)


#Copying Folder Structer 
def ig_f(dir, files):
	return [f for f in files if os.path.isfile(os.path.join(dir, f))]

print('copying struct')

if os.path.exists(prep):
	shutil.rmtree(prep)
	shutil.copytree(directory, prep, ignore=ig_f)
else: 
	shutil.copytree(directory, prep, ignore=ig_f)

print('moving files')
#Copying over meta data file 
for dir, path, files in os.walk(directory):
	for file in files: 
		if file.endswith("csv"):
			print(file)
			print(dir+file)
			shutil.move(os.path.join(dir,file), prep+"metadata.csv")
		if file.endswith("html"):
		 shutil.move(os.path.join(dir,file), os.path.join(prep,file))
   
print('procedure')
count = 0
for folder in os.listdir(directory):
    print("IN FOLDER: "+folder)
    if(folder=='.DS_Store'):
        continue
    for sub_folder in os.listdir(directory+"/"+folder):
        if(sub_folder=='.DS_Store'):
            continue
        #print("IN SUB-FOLDER: "+sub_folder)
        for sub_sub_folder in os.listdir(directory+"/"+folder+"/"+sub_folder):
            if(sub_sub_folder=='.DS_Store'):
                continue
            print("IN SUB-SUB-FOLDER: "+sub_sub_folder)
            command = 'dcm2niix -0 -z y -f "%i_%k" -p y -b n -m y  -o "'+prep+"/"+folder+"/"+sub_folder+"/"+sub_sub_folder+'" "'+directory+"/"+folder+"/"+sub_folder+"/"+sub_sub_folder+'"'
            print("#########COMMAND:"+command)
            count = count+1
            os.system(command)  # returns the exit code in unix
            print(count)
print("Count: "+str(count))


print('------------ Fully Processed------------')<jupyter_output>/content/gdrive/.shortcut-targets-by-id/1YdeMHAwQy3TZggu1P3vt94i0Ff14D0TR/L3-Clean/NII_Data/Anti-PD-1_Lung_Subset
/content/gdrive/.shortcut-targets-by-id/1YdeMHAwQy3TZggu1P3vt94i0Ff14D0TR/L3-Clean/Data/Anti-PD-1_Lung_Subset
copying struct
moving files
procedure
IN FOLDER: PD-1-Lung-00006
IN SUB-SUB-FOLDER: 303.000000-CAP AX 2.5-83135
#########COMMAND:dcm2niix -0 -z y -f "%i_%k" -p y -b n -m y  -o "/content/gdrive/.shortcut-targets-by-id/1YdeMHAwQy3TZggu1P3vt94i0Ff14D0TR/L3-Clean/NII_Data/Anti-PD-1_Lung_Subset/PD-1-Lung-00006/10-22-2009-CT CHEST ABDOMEN PELVIS W CONTRAST-92268/303.000000-CAP AX 2.5-83135" "/content/gdrive/.shortcut-targets-by-id/1YdeMHAwQy3TZggu1P3vt94i0Ff14D0TR/L3-Clean/Data/Anti-PD-1_Lung_Subset/PD-1-Lung-00006/10-22-2009-CT CHEST ABDOMEN PELVIS W CONTRAST-92268/303.000000-CAP AX 2.5-83135"
1
IN SUB-SUB-FOLDER: 602.000000-CH SAG PV-36840
#########COMMAND:dcm2niix -0 -z y -f "%i_%k" -p y -b n -m y  -o "/content/gdrive/.shortcut-targets-by-id/1YdeMHAwQy3TZggu1P3vt94i0Ff[...]
{'fraction_non_alphanumeric': 0.11754101316026681, 'fraction_numerical': 0.04092302145303768, 'mean_word_length': 4.089908256880734, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '333673', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2126, 'n_tokens_neox': 1935, 'n_words': 570, 'path': 'Data Prep.ipynb', 'repo_name': 'kaylajanos1/TeamSpark-L3Detection', 'size': '5547'}