<jupyter_start><jupyter_text>Simplifying brain-twisting endless commandsand minimize the chance of typos between two tries Let's say I need to type these commands :ANTS 3 -m CC[ /home/grg/data/ALFA_DWI/10013/T1/10013_mabonlm_nobias_spm_c2_d2.nii.gz, /home/grg/data/ALFA_DWI/10013/DWI/10013_B0_brain_seg_1.nii.gz, 1, 4] -r Gauss[0,3] -t Elast[1.5] -i 30x20x10 -o /home/grg/data/ALFA_DWI/10013/DWI/ants_elast_dwi-to-t1_.nii.gzWarpImageMultiTransform 3 /home/grg/data/ALFA_DWI/10013/DWI/10013_MD.nii.gz /home/grg/data/ALFA_DWI/10013/DWI/10013_MD_t1space.nii.gz -R /home/grg/data/ALFA_DWI/10013/T1/10013_mabonlm_nobias.nii /home/grg/data/ALFA_DWI/10013/DWI/ants_elast_dwi-to-t1_Warp.nii.gz /home/grg/data/ALFA_DWI/10013/DWI/ants_elast_dwi-to-t1_Affine.txt  I'll first create a JSON giving a template of the commands.<jupyter_code>j = {'ants_dwi_to_t1': u'ANTS 3 -m CC[ %s, %s, 1, 4] -r Gauss[0,3] -t Elast[1.5] -i 30x20x10 -o %s',
     'warp_md_to_t1': u'WarpImageMultiTransform 3 %s %s -R %s %s %s'}
json.dump(j, open('/tmp/templates.json','w'))<jupyter_output><empty_output><jupyter_text>The $\%s$ represent parameters (such as filenames) that will be provided later.This is like separating bones (the command itself) from the meat (the data). $thesaurus$ will then fill in the ($\%s$) blanks. Just name the desired command and give the parameters in the same order as they appear in the command.<jupyter_code>cmd = './thesaurus.py -n --name ants_dwi_to_t1 /tmp/templates.json target.nii moving.nii trans_'
print '$', cmd
print shell_execute(cmd)

cmd = './thesaurus.py -n --name warp_md_to_t1 /tmp/templates.json target.nii moving.nii output.nii trans_Warp.nii trans_Affine.txt'
print '\n$', cmd
print shell_execute(cmd)<jupyter_output>$ ./thesaurus.py -n --name ants_dwi_to_t1 /tmp/templates.json target.nii moving.nii trans_
ANTS 3 -m CC[ target.nii, moving.nii, 1, 4] -r Gauss[0,3] -t Elast[1.5] -i 30x20x10 -o trans_


$ ./thesaurus.py -n --name warp_md_to_t1 /tmp/templates.json target.nii moving.nii output.nii trans_Warp.nii trans_Affine.txt
WarpImageMultiTransform 3 target.nii moving.nii -R output.nii trans_Warp.nii trans_Affine.txt<jupyter_text>The $-n$ option is just a safety lock to display the command only and not to run it.  Pushing it further.. $thesaurus$ can work with structured databases and preexisting filetypes.Here let's define the various IO of the previous commands.$@$ indicate input items that must exist prior to running the command.$>$ indicate output items that will be generated by the command.Naturally all the different filetypes have to be predefined and recognized by the database system.<jupyter_code>j = {'ants_dwi_to_t1': [u'@ALFA Denoised Nobias SPM Dilated White matter',
         u'@ALFA DWI B0 Brain FSL FAST White matter',
         u'ALFA ANTS Elast DWI to T1 Transformation Template Filename'],
     'warp_md_to_t1': [u'@ALFA Mean Diffusivity Image',
         u'>ALFA DWI MD Map Warped to T1 space',
         u'@ALFA Denoised Nobias T1 Image',
         u'@ALFA ANTS Elast DWI to T1 Transformation',
         u'@ALFA ANTS Elast DWI to T1 Affine Transformation']}
json.dump(j, open('/tmp/templates_io.json', 'w'))<jupyter_output><empty_output><jupyter_text>Then call the desired command by its keyname, with the desired subject and feel light...<jupyter_code>import thesaurus as t
a = t.ALFAHelper() #jsonfile='/tmp/templates_io.json')
a.parse_command(subject='10013', name='warp_md_to_t1')<jupyter_output><empty_output><jupyter_text>Based on this file it can also recreate the sequence of a pipeline and that provide you with the progression state of a subject along this pipeline.<jupyter_code>step1 = a.current_stage('10013')
print 'step:', step1, '\n'

step2 = a.current_stage('55855')
print 'step:', step2<jupyter_output>10013 is complete
step: 0 

55855 is stuck at step before warp_aal_to_dwi (missing: ALFA ANTS Elast T1 to DWI Transformation)
step: warp_aal_to_dwi-1<jupyter_text>OK now enough talking. Can you please run that d@£( step over my entire dataset ?<jupyter_code>d = '/home/grg/data/ALFA_DWI/'
from glob import glob
subjects = [e.split('/')[-1] for e in glob(osp.join(d, '*')) if osp.isdir(e)]
a.generate_batch(subjects, step='ants_dwi_to_t1', batchfile='/tmp/batch.sh', create_workflow=True)
b = open('/tmp/batch.sh').readlines()
for each in b[:20]:
    print each<jupyter_output>ANTS 3 -m CC[ /home/grg/data/ALFA_DWI/21056/T1/21056_mabonlm_nobias_spm_c2_d2.nii.gz, /home/grg/data/ALFA_DWI/21056/DWI/21056_B0_brain_seg_1.nii.gz, 1, 4] -r Gauss[0,3] -t Elast[1.5] -i 30x20x10 -o /home/grg/data/ALFA_DWI/21056/DWI/ants_elast_dwi-to-t1_.nii.gz

ANTS 3 -m CC[ /home/grg/data/ALFA_DWI/11136/T1/11136_mabonlm_nobias_spm_c2_d2.nii.gz, /home/grg/data/ALFA_DWI/11136/DWI/11136_B0_brain_seg_1.nii.gz, 1, 4] -r Gauss[0,3] -t Elast[1.5] -i 30x20x10 -o /home/grg/data/ALFA_DWI/11136/DWI/ants_elast_dwi-to-t1_.nii.gz

ANTS 3 -m CC[ /home/grg/data/ALFA_DWI/12636/T1/12636_mabonlm_nobias_spm_c2_d2.nii.gz, /home/grg/data/ALFA_DWI/12636/DWI/12636_B0_brain_seg_1.nii.gz, 1, 4] -r Gauss[0,3] -t Elast[1.5] -i 30x20x10 -o /home/grg/data/ALFA_DWI/12636/DWI/ants_elast_dwi-to-t1_.nii.gz

ANTS 3 -m CC[ /home/grg/data/ALFA_DWI/10472/T1/10472_mabonlm_nobias_spm_c2_d2.nii.gz, /home/grg/data/ALFA_DWI/10472/DWI/10472_B0_brain_seg_1.nii.gz, 1, 4] -r Gauss[0,3] -t Elast[1.5] -i 30x20x10 -o /home/grg/data/A[...]
{'fraction_non_alphanumeric': 0.12932162212670528, 'fraction_numerical': 0.06316576340870865, 'mean_word_length': 4.586638830897703, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '122236', 'license': 'MIT', 'n_tokens_mistral': 2448, 'n_tokens_neox': 2247, 'n_words': 518, 'path': 'doc/Thesaurus.ipynb', 'repo_name': 'xgrg/thesaurus', 'size': '5351'}