<jupyter_start><jupyter_text>Ilastik for 10/31/16 WeekFrom last week, I was able to generate 3D TIFF slices and image classifiers on Fear199 downsampled data.  However, my problems were that:  1)  The TIFF slices were odd, cigar-shaped tubes.  2)  I was unable to generate a significant classifier using the existing data because of the weird image layout.  3)  I had trouble loading in the TIFF stack despite having generated one via ImageJ  What I did this week was:  1)  Figure out why my original data was the odd cigar-shaped data.    2)  Correctly generate a subset of TIFF slices for Fear199.   3)  Generate a pixel-based object classifier.  What I need help with/still need to learn:  1)  How to interpret/better validate my classifier results (currently have hdf5/TIFF output, how can I validate this?)  2)  How to apply this to density mapping  Task 1: Why was my original data cigar-shaped?  When downloading the image from ndreg, there were two different approaches to generating the numpy array.  I've shown both below:<jupyter_code>## Script used to download nii run on Docker
from ndreg import *
import matplotlib
import ndio.remote.neurodata as neurodata
import nibabel as nb
inToken = "<PASSWORD>"
nd = neurodata()
print(nd.get_metadata(inToken)['dataset']['voxelres'].keys())
inImg = imgDownload(inToken, resolution=5)
imgWrite(inImg, "./Fear199.nii")
## Method 1:
import os
import numpy as np
from PIL import Image
import nibabel as nib
import scipy.misc
TokenName = '<PASSWORD>'
img = nib.load(TokenName)

## Convert into np array (or memmap in this case)
data = img.get_data()
print data.shape
print type(data)
## Method 2:
rawData = sitk.GetArrayFromImage(inImg)  ## convert to simpleITK image to normal numpy ndarray
print type(rawData)<jupyter_output><empty_output><jupyter_text>In a nutshell, method 1 generates an array with shape (x, y, z) -- specifically, (540, 717, 1358).  The method 2 generates a numpy array with shape (z, y, x) -- specifically, (1358, 717, 540).  Since we want the first column to be z slices, the original method was granting me x-slices (hence the cigar-tube dimensions).In order to interconvert, we can either just use the rawData approach after directly calling from ndstore, or we can take our numpy array after loading from nibabel and use numpy's swapaxes method to just swap two of the dimensions (shown below).<jupyter_code>## if we have (i, j, k), we want (k, j, i)  (converts nibabel format to sitk format)
new_im = newer_img.swapaxes(0,2) # just swap i and k<jupyter_output><empty_output><jupyter_text>Task 2: Generating raw TIFF slices.Now that I have appropiate coordinates, I generated a subset of TIFF slices to run the training module for the image classifier.  Using the script here:<jupyter_code>plane = 0;
for plane in (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 100, 101, 102, 103, 104):
    output = np.asarray(rawData[plane])
    ## Save as TIFF for Ilastik
    scipy.misc.toimage(output).save('RAWoutfile' + TokenName + 'ITK' + str(plane) + '.tiff')<jupyter_output><empty_output><jupyter_text>As shown above, I generated data for the first 13 planes, and then some subset of planes from z = 100 to z = 104.  I then trained the classifier on the 0 through 12 data, and used the 100 to 104 slices to validate my results.  Below I've included images of the raw and histogram equalized images at one specific slice to just show the raw images I was working with.**Raw Fear 199 at Z = 100:**![rawFear199Slice100](images/oct31/RAWoutfileFear199.niiITK100.png)**Histogram Equalized Fear 199 at Z = 100:**![histeqFear199Slice100](images/oct31/HISTEQoutfileFear199ITK100.png)In order to generate the histogram equalized TIFF slices, I wrote a Jupyter notebook here:https://github.com/NeuroDataDesign/seelviz/blob/gh-pages/Tony/ipynb/generate%2BhistEQ.ipynb  Task 3:  Generate a pixel-based object-classifier for both the raw and the histogram equalized data.From there, I generated a pixel-based object-classifier for both the raw and the histogram equalized data.  Shown first are images demonstrating me generating the classifier for the raw data.  Then I show the images of me generating the classifier for the histogram equalized data. First, the raw data classifier.  I load in planes 0-12 TIFF slices and generate a classifier by training it to select for borders, background, and individual bright points:**Loading Raw Training Data**![loadingrawFear199Training](images/oct31/loadingrawfear199.png)**Selecting Borders in Raw Training Data**![trainingbordersFear199](images/oct31/trainingonbordersrawfear199.png)**Selecting Bright Points in Raw Training Data**![trainingbrightpointsFear199](images/oct31/trainingonbrightpointsrawfear199.png)**Closeup of Feature Selection in Raw Training Data**![closeupfeatureselectionrawfear199](images/oct31/closeupfeatureselectionrawfear199.png)**Object Output for Raw Training Data**![exampleobjectsrawfear199](images/oct31/exampleobjectsrawfear199.png) In order to run Ilastik headlessly on 3D classified data, use globstring syntax to tell ilastik which images to combine for each volume.  EG:<jupyter_code>./run_ilastik.sh --headless --project=MyProject.ilp "my_next_stack_*.png" "my_other_stack_*.png"<jupyter_output><empty_output><jupyter_text>For me to get it to work, I made a folder called "runtime" where each file was named WITHOUT the underscores.  Make sure to add the * to get it to work, also.<jupyter_code>/Applications/ilastik-1.2.0rc10-OSX.app/Contents/ilastik-release/run_ilastik.sh --headless --project=ilastikClassifierFear199.ilp --output_format=tiff "../rawFear199ITK/runtime/RAWoutfileFear199.niiITK100*.tiff" "../rawFear199ITK/runtime/RAWoutfileFear199.niiITK101*.tiff"<jupyter_output><empty_output><jupyter_text>Repeating the process for the histogram equalized brains, we get:**Loading HistEq Training Data**![loadinghisteqFear199Training](images/oct31/loadinghisteqfear199.png)**Selecting Borders and Background in HistEq Training Data**![trainingbordersandbackgroundhisteqfear199](images/oct31/trainingbordersandbackgroundhisteqfear199.png)**Selecting Bright Points in HistEq Training Data**![trainingonbrightpointshisteqfear199](images/oct31/trainingonbrightpointshisteqfear199.png)**Object Output for HistEq Training Data**![exampleobjectclassificationhisteqfear199](images/oct31/exampleobjectclassificationhisteqfear199.png) Again running our classifier using the headless display (command below), we can generate a TIFF probability mapping.<jupyter_code>/Applications/ilastik-1.2.0rc10-OSX.app/Contents/ilastik-release/run_ilastik.sh --headless --project=ilastikClassifierFear199.ilp --output_format=tiff "../rawFear199ITK/runtime/RAWoutfileFear199.niiITK100*.tiff" "../rawFear199ITK/runtime/RAWoutfileFear199.niiITK101*.tiff"<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08387286639199529, 'fraction_numerical': 0.04017068864037669, 'mean_word_length': 5.703155818540434, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '310661', 'license': 'Apache-2.0', 'n_tokens_mistral': 2285, 'n_tokens_neox': 2006, 'n_words': 749, 'path': 'Tony/Ilastik on Raw and HistEq Fear199 Data.ipynb', 'repo_name': 'alee156/cldev', 'size': '6794'}