<jupyter_start><jupyter_text>Graphene-U-Net Usage Example Step 0: Install Dependencies**NERSC Note:**  * Use pytorch-1.71-gpu kernel  * Uncomment the following cell when running the notebook for the first time to install the required OpenCV libraray<jupyter_code>#%pip install --user opencv-python<jupyter_output><empty_output><jupyter_text>Step 1: Import Library<jupyter_code>from Microscopy_Unet import Segmentation_Network<jupyter_output><empty_output><jupyter_text>Step 2: Instantiate NetworkD is the 'Depth' of the network/number of down convolutional blocks (which will be followed with D-1 up convolutional blocks)W is the network 'Width' which is the power of 2 number of features in the first block (which will be doubled for every subsequent block in the contraction path, and then divided by two in the contraction path)model_name is the name of the model used to save the network which will also be used to save training progress datasave_freq is the frequency in terms of epochs how often the network will be saved to disk<jupyter_code>network= Segmentation_Network(D = 4,       #network depth
                W = 5,                     #netork width
                LR = .0001,                #learning rate
                model_name = "UNET_GRPH",  #name model for checkpoints
                save_freq=10               #frequency of network checkpoint saves
               )<jupyter_output>initializing model
>>>from blank slate<jupyter_text>Step 3: Load Training DataNext, we load the training data from the `training_images/` folder using GetDataSet, where data_path is the path to the folder where the labeled training data is. The term 'handle' is the marker name for the labels of the training data. The naming convention of the training data is as such: For every input image "`filename.png`" in the data_path folder there should be a corresponding label image with the name "`filename_.png`". For example, if the handle is "`_LABEL`" then for a folder with input images named "`training_img_0.png`", "`training_img_1.png`" there should be corresponging files named "`training_img_0_LABEL.png`", "`training_img_1_LABEL.png`" to get proper behaivior.    Once data is loaded, it is important to shuffle it.<jupyter_code>network.GetDataSet(data_path = "training_images/", handle="_LABEL") #load in dataset
network.DataShuffle(seed = 4)<jupyter_output><empty_output><jupyter_text>Step 4: Train NetworkNow we train the network using K-Fold cross Validation where K = 4<jupyter_code>network.Kfold(k=4, epochs=15)<jupyter_output>==========RUNNING 4-FOLD CROSS VALIDATION==========

 ###############STARTING FOLD: 0 of 4###############

initializing model
>>>from blank slate

Evaluating...<jupyter_text>Step 5: Visualize Training Progress Over EpochsData about the training progress is automatically saved in CSV Files associated with the network name for convenience of observing the progress over time. These can be loaded into the notebook using pandas and visualized using matplotlib.<jupyter_code>import pandas as pd
import os
import matplotlib.pyplot as plt
import numpy as np
from os import path
name = "UNET_GRPH"
plt.figure()

for fold in range(4):
    file = name+"_fold_"+str(fold)+".csv"
    if path.exists(file):
        data = pd.read_csv(file)['F1_1'].to_numpy()
        plt.plot(data, label = 'fold '+str(fold))

plt.legend()
plt.xlabel('Epoch')
plt.ylabel('Dice Score')
plt.title('Training Progress Visualized')
plt.show()<jupyter_output><empty_output><jupyter_text>Step 6: Load and Use ModelNow that we have trained the model, we can load it and then run it on an example image. For simplicity we a re-use a training image.First we load an image.<jupyter_code>import cv2
import matplotlib.pyplot as plt

img = cv2.imread("./training_images/2013_1114_Gr_FS_011_layer_0007.png", 0)
plt.imshow(img, cmap = 'gray')<jupyter_output><empty_output><jupyter_text>Then, we load the trained model into a new network instance.<jupyter_code>trained_network= Segmentation_Network(D = 4,   #network depth
                W = 5,                         #netork width
                LR = .0001,                    #learning rate
                model_name = "UNET_GRPH_demo", #name model for checkpoints
                save_freq=10,                  #frequency of network checkpoint saves
                load = "UNET_GRPH_fold_0.pth"  #path to trained network                              
               )<jupyter_output>initializing model
>>>from saved model
loading model from UNET_GRPH_fold_0.pth
reseting optimizer<jupyter_text>Finally, we use this model to predict the segmentation for the loaded image.<jupyter_code>output = trained_network.infer(img)
plt.figure()
plt.subplot(1,2,1)
plt.imshow(img)
plt.subplot(1,2,2)
plt.imshow(output)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08574401664932361, 'fraction_numerical': 0.014360041623309052, 'mean_word_length': 4.0010405827263265, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '564427', 'license': 'BSD-3-Clause-LBNL', 'n_tokens_mistral': 1487, 'n_tokens_neox': 1377, 'n_words': 512, 'path': 'usage.ipynb', 'repo_name': 'lbnlcomputerarch/graphene-u-net', 'size': '4805'}