<jupyter_start><jupyter_text>Analysis 20190419 * Written by: <NAME> * Running under conda env "lipidraft" * Source Images: testdata<jupyter_code># import resource
import subprocess
import os, sys
from core.fileop import DirCheck, ListFiles, GetPendingList, GetGrpFLs

#Functions ---------------------------------------------------------------
def dict2arg_fiji(arg):
    arg_str = str()
    for key, val in arg.items():
        str_tmp = str(key + '="' + val + '", ')
        arg_str = arg_str + str_tmp
    return arg_str

def list2arg_python(arg):
    arg_str = str()
    for var in arg:
        arg_str = arg_str +  "'" + str(var) + "' " 
    return arg_str
# -------------------------------------------------------------------------<jupyter_output><empty_output><jupyter_text>Specify Parameters * `fiji`: path to the fiji application * `codepath`: the workspace for code/script * `datapath`: the workspace for STORM images and data * `analysis_dir`: the folder hosting data and images generated from this pipeline<jupyter_code>fiji = '/Applications/Fiji.app/Contents/MacOS/ImageJ-macosx'
codepath = '/Users/michaelshih/Documents/code/wucci/storm_image_processing'
datapath = '/Volumes/LaCie_DataStorage/xiaochao_wei_STORM imaging/STORM_imaging'
analysis_dir = 'analysis_20190419'<jupyter_output><empty_output><jupyter_text>Script name: `imgproc.py` * env: ImageJ/Fiji * rewritten: 04/26/2019 * verified * output folder: `preprocessing`  Output   1. `preproimg` (`.tif`): seperate the channels into individual folders   2. `imgintensity` (`.csv`): return mean intensity for each time frame from the center of the image (128 pixel * 128 pixel)   3. `imginfo`:       1. `imgmetadata` (`.txt`): return the metadata       2. `imgstat.csv`: statistics from images           * image_name: the name of the image without extension           * ip_file_name: the name of the image with extension           * xSize: frame size on the x-axis           * ySize: frame size on the y-axis           * nSlices: frame size on the z-axis           * nFrames: number of time frames           * nChannels: number of channles           * size_MB: size of the file<jupyter_code>script_name = 'imgproc.py'
ippath = '/Volumes/LaCie_DataStorage/xiaochao_wei_STORM imaging/STORM_imaging/resource/testdata'
opdir = 'preprocessing'
oppath = os.path.join(datapath, analysis_dir, opdir)

# create dict to store parameters
arg_dict = {
    'path': datapath,
    'dir_output': analysis_dir,
    'ippath': ippath,
    'outputdir': oppath,
    'batchmodeop': 'false',
}

arg = dict2arg_fiji(arg_dict)
print(arg)
# Run the script
subdir = 'Fiji'
scriptpath = os.path.join(codepath, subdir, script_name)
print("Start running: {}".format(script_name))
subprocess.check_output([fiji, '--ij2', '--run', arg])
print("End: {}".format(script_name))<jupyter_output><empty_output><jupyter_text>Script name: `tstormanalysis.py` * env: ImageJ/Fiji * plugin: [ThunderSTORM](https://github.com/zitmen/thunderstorm) * rewritten: 04/26/2019 * verified * output folder: `tstorm`  Output generated by ThunderSTORM   1. `csvdata`:       * (`.csv`): STORM dataset       * (`.txt`): protocol   2. `driftcorr` (`.json`): drift correction profile   3. `histogram_plot` (`.tif`): histogram image with manification = 5.0<jupyter_code>script_name = 'tstormanalysis.py'
ippath = '/Volumes/LaCie_DataStorage/xiaochao_wei_STORM imaging/STORM_imaging/analysis_20190419/preprocessing/preproimg'
opdir = 'preprocessing'
opsubdir = 'preproimg'

arg_dict = {
    'path': datapath,
    'dir_output': analysis_dir,
    'ippath': ippath,
    'batchmodeop': 'false', 
}
arg = dict2arg_fiji(arg_dict)
print(arg)

subdir = 'Fiji'
scriptpath = os.path.join(codepath, subdir, script_name)
print("Start running: {}".format(script_name))
subprocess.check_output([fiji, '--ij2', '--run', scriptpath, arg])
print("End: {}".format(script_name))<jupyter_output><empty_output><jupyter_text>Script name: `csv_slicer_T.py` * env: ImageJ/Fiji * rewritten: 04/26/2019 * verified * output folder: `csvdata_sliced`  Output generated by ThunderSTORM   * `csvdata_sliced_T` (`.csv`): STORM data from T0 to T1<jupyter_code>script_name = 'csv_slicer_T.py'
nchannel = 2
T0 = 5000
T1 = 10001
arg_list = [datapath, analysis_dir,'tstorm','csvdata', nchannel, T0, T1, 'csvdata_sliced_T']
arg = list2arg_python(arg_list)
print(arg)

print("Start running: {}".format(script_name))
shellcmd = str('python '+ os.path.join(codepath, script_name)+ ' '+ arg)
print(shellcmd)
process = subprocess.run(shellcmd, shell = True, check=True)
print("End: {}".format(script_name))<jupyter_output><empty_output><jupyter_text>Script name: `csv_slicer_ROI.py` * env: ImageJ/Fiji * rewritten: 04/29/2019 * verified * output folder: `csvdata_sliced`  Output generated by ThunderSTORM   * `csvdata_sliced_T_ROI` (`.csv`): STORM data cropped by given region of interest.<jupyter_code># csv_slicer_crop.py
script_name = 'csv_slicer_ROI.py'
arg_list = [datapath, analysis_dir,'tstorm','csvdata_sliced_T', 2, 3, 'csvdata_sliced_T_ROI']
arg = list2arg_python(arg_list)
print(arg)

print("Start running: {}".format(script_name))
shellcmd = str('python '+ os.path.join(codepath, script_name)+ ' '+ arg)
print(shellcmd)
process = subprocess.run(shellcmd, shell = True, check=True)
print("End: {}".format(script_name))<jupyter_output><empty_output><jupyter_text>Script name: `spatialanaylsis.R` * env: ImageJ/Fiji * rewritten: 04/29/2019 * verified * output folder: `spacial_test`  Output generated by ThunderSTORM   * `spacialdata` (`.csv`): Results of Ripley's K       * K_r: $\hat{K}(d)= \hat{\lambda}^{-1}\sum_{i}\sum_{j\neq1}\frac{\omega(l_i, l_j)I(d_{ij<t})}{N}$       * L_r:       * H_r:   * `spacialdata_bi` (`.rda`):<jupyter_code># spatialanaylsis.R
script_name = 'spatialanaylsis.R'
arg_list = [datapath, analysis_dir,'tstorm','csvdata_sliced_T_ROI', 2, 3, 'spacial_test', 'spacialdata', 'spacialdata_bi']
arg = list2arg_python(arg_list)
print(arg)

print("Start running: {}".format(script_name))
shellcmd = str('Rscript --vanilla '+ os.path.join(codepath, script_name)+ ' '+ arg)
print(shellcmd)
process = subprocess.run(shellcmd, shell = True, check=True)
print("End: {}".format(script_name))

# plot_spatialdata.py
script_name = 'plot_spatialdata.py'
arg_list = [datapath, analysis_dir,'spacial_test','spacialdata', 2, 'plot']
arg = list2arg_python(arg_list)
print(arg)

print("Start running: {}".format(script_name))
shellcmd = str('python '+ os.path.join(codepath, script_name)+ ' '+ arg)
print(shellcmd)
process = subprocess.run(shellcmd, shell = True, check=True)
print("End: {}".format(script_name))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15992147387496225, 'fraction_numerical': 0.01706433101781939, 'mean_word_length': 4.665526090675791, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '498039', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2434, 'n_tokens_neox': 2315, 'n_words': 512, 'path': '.ipynb_checkpoints/pipeline_controller_04192019_test-checkpoint.ipynb', 'repo_name': 'eufmike/storm_image_processing', 'size': '6632'}