<jupyter_start><jupyter_text>Bloby Usage Example Bloby is an end-to-end package* get data from BOSS* detect blobs from a 3D TIF stack* quantitatively evaluate detection results and * upload the results to BOSS for qualitative visualizationThis Jupyter Notebook will explain how to use each section of this package's pipeline.  0. Get data from BOSS<jupyter_code>%reload_ext autoreload
%autoreload 2

# from src.BossDataFetcher import fetch_data_from_boss

exp_name = 'cell_detection_9'
input_tif_path = 'data/s3617/tifs/{}.tiff'.format(exp_name)

# boss_params = {
#     'config': 'intern.cfg',
#     'collection': 'ailey-dev',
#     'experiment': 's3617',
#     'channel': 'channel0',
#     'dtype': 'uint16',
#     'z_range': [620, 720],
#     'y_range': [6300, 6800],
#     'x_range': [6900, 7400],
#     'resolution': 0,
#     'filename': input_tif_path
# }

# fetch_data_from_boss(boss_params)<jupyter_output><empty_output><jupyter_text>1. Blob Detection The blob detection part of the pipeline uses Gaussian Mixture Model (GMM) clustering for thresholding, followed by morphological erosion and then finally finds centroids using connected components. The below snippet shows how to instantiate the _Bloby_ class and find blobs from a TIF stack.<jupyter_code>from src.BlobDetector import BlobDetector
from src import util

output_csv_path = 'data/s3617/prediction_csv/{}.csv'.format(exp_name)

detector = BlobDetector(input_tif_path)
centroids = detector.get_blob_centroids()
util.write_list_to_csv(centroids, output_csv_path) #writing the detection output to CSV in (z,y,x) format<jupyter_output><empty_output><jupyter_text>Viewing the output CSV<jupyter_code>centroid_list = util.get_list_from_csv(output_csv_path)
for centroid in centroid_list[:5]: #sample rows
    print(centroid)
print('...')<jupyter_output>[7.0, 87.0, 248.0]
[4.0, 126.0, 245.0]
[6.0, 129.0, 284.0]
[7.0, 141.0, 247.0]
[23.0, 145.0, 259.0]
...<jupyter_text>2. Quantitative Evaluation - _BlobMetrics_ class The _BlobMetrics_ class which is a part of this package, can be used to evaluate Quantitatively the blob detection results, given a manually annotated ground truth CSV.<jupyter_code>from src.BlobMetrics import BlobMetrics

ground_truth_path = 'data/s3617/annotations/{}.csv'.format(exp_name)
ground_truth = util.get_list_from_csv(ground_truth_path)

metrics = BlobMetrics(ground_truth, centroids)
print('Precision: {}\nRecall: {}'.format(metrics.precision(), metrics.recall()))<jupyter_output>Precision: 0.861
Recall: 0.288<jupyter_text>Apart from numerical metrics, we can evaluate the prediction based on the following two plots  Number of predictions per ground truth label<jupyter_code>metrics.plot_predictions_per_ground_truth()<jupyter_output><empty_output><jupyter_text>Number of ground truth labels per prediction<jupyter_code>metrics.plot_ground_truths_per_prediction()<jupyter_output><empty_output><jupyter_text>3. Qualitative evaluation - BOSS Upload In order to visualize the prediction results, the package supports BOSS upload **Step 1**: First we need to plot the prediction results in an empty TIF stack<jupyter_code>output_tif_path = 'data/s3617/prediction_tifs/{}.tiff'.format(exp_name)
util.plot_csv_on_tif(centroids, input_tif_path, output_tif_path)<jupyter_output><empty_output><jupyter_text>**Step 2**: Create config params - The BOSS upload needs certain parameters to be configured. It can be done as follows. _intern.cfg_ should be in the following format```protocol = httpshost = api.boss.neurodata.iotoken = ```<jupyter_code>from src.IngestTifStack import ConfigParams
util.create_intern_config()

ingest_conf = {
    'collection': 'cell_detection',
    'experiment': exp_name,
    'channel': 'sprint3_final',
    'tif_stack': 'data/s3617/prediction_tifs/{}.tiff'.format(exp_name),
    'type': 'annotation',
    'new_channel': True,
    'source_channel': 'raw_data',
    'config': 'intern.cfg'
}
params = ConfigParams(ingest_conf)<jupyter_output><empty_output><jupyter_text>**Step 3**: Upload it to BOSS. The following code uploads the specified TIF task to the specified collection/experiment/channel as a new _annotation_ channel which is sourced to the **raw_data** channel.<jupyter_code>from src.IngestTifStack import IngestTifStack

group_name = 'ndd17_claritrons'
#optional parameter if you want to grant permissions to the new channel that you have created

ingest = IngestTifStack(params)
upload_link = ingest.start_upload(group_name=group_name)
print('Results uploaded to {}'.format(upload_link))<jupyter_output>Results uploaded to https://ndwebtools.neurodata.io/ndviz_url/cell_detection/cell_detection_9/
{'fraction_non_alphanumeric': 0.10287443267776097, 'fraction_numerical': 0.026583099200345798, 'mean_word_length': 6.26530612244898, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '223674', 'license': 'Apache-2.0', 'n_tokens_mistral': 1636, 'n_tokens_neox': 1511, 'n_words': 406, 'path': 'Package Usage.ipynb', 'repo_name': 'NeuroDataDesign/lids-bloby', 'size': '4627'}