<jupyter_start><jupyter_text>Seal Detection Pipeline---This jupyter notebook will go through assembling the main components of a complete pipeline for counting seals in high-resolution satellite imagery (figure 1, steps 3 and 4) and show some experimental results with different pipeline designs. The ultimate goal of this pipeline is to perform a pan-Antarctic pack-ice seal census. ** Running this code will require input satellite imagery and at least one GPU with >8GB of memory **  Table of contents---* [Getting started](intro)    * [Setup](setup)    * [Visualize training set](vis_imgs)* [Pipeline 1 - Seal haulout detector](1)    * [Training](1T)    * [Validation](1V)    * [Ablation experiment](1A)* [Pipeline 1.1 - Seal haulout detector + count](1.1)    * [Training](1.1T)    * [Validation](1.1V)    * [Ablation experiment / testing](1.1A)* [Pipeline 1.2 - Seal haulout detector + single seal detector](1.2)    * [Training](1.2T)    * [Validation](1.2V)    * [Testing](1.2A)  Getting started---If you followed the *training_set_generation* jupyter notebook (also present in this repo), you should have training sets generated and hyperparameter sets to try out, and be ready to search for a best performing seal detection pipeline.  Output files in this repository are organized as follows: *'./{dest_folder}/{pipeline}/{model_settings}/{model_settings}_{file}'*  Setup environmentBefore training and validating model/hyperparameter combinations inside the pipelines, we need to load the required python modules and a few global variables. Running this script will also display a list of training classes.<jupyter_code># import required packages
import os
import rasterio
import pandas as pd
import numpy as np
import operator
from PIL import Image 
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import matplotlib as mpl
from functools import reduce
from utils.model_library import * 

%matplotlib inline
mpl.rcParams['figure.dpi']= 400

# destination folder for saved models and model stats
dest_folder = 'saved_models'

# save class names
class_names = sorted([subdir for subdir in os.listdir('./training_sets/training_set_vanilla/training')])<jupyter_output><empty_output><jupyter_text>Visualizing training images (Optional)To get a better sense for what the training set is like, the next cell will display a few random images from the training classes. Displayed images are extracted from a pool of ~70000 training images.<jupyter_code># store images
images = []

# loop over labels
for label in class_names:
    for path, _, files in os.walk('./training_sets/training_set_vanilla/training/{}'.format(label)):
        files = np.random.choice(files, 5)
        for filename in files:
            images.append(np.asarray(Image.open(os.path.join(path, filename))))

images = np.array(images)

# display images 
ncols=len(class_names)
nindex, height, width, intensity = images.shape
nrows = nindex // ncols
# check if rows and columns can fit the number of images
assert nindex == nrows * ncols
result = (images.reshape(nrows, ncols, height, width, intensity)
          .swapaxes(1,2)
          .reshape(height*nrows, width*ncols, intensity))

plt.imshow(result)
cur_axes = plt.gca()
cur_axes.axes.get_xaxis().set_visible(False)
cur_axes.axes.get_yaxis().set_visible(False)
plt.show()<jupyter_output><empty_output><jupyter_text>Pipeline 1 - Heatmap models ---  TrainingThe first step to find a best performing model is to train different model setups using our training set. To keep track of which combinations we have tried, how well they performed and the specifics of each model setup, we will store results in folders (under './{dest_folder}') named after each specific model combination.<jupyter_code># switch pipeline
pipeline = 'Heatmap'

# generate model combinations
combinations_1 = {'model_architecture': ['Unet'],
                  'training_dir': ['training_set_vanilla'],
                  'hyperparameter_set': ['E']}

# read as a DataFrame
combinations_1 = pd.DataFrame(combinations_1)
                    
# create folders for resulting files
for row in combinations_1.iterrows():
    mdl = row[1]['model_architecture'] + '_ts-' + row[1]['training_dir'].split('_')[-1]              
    if not os.path.exists("./{}/{}/{}".format(dest_folder, pipeline, mdl)):
        os.makedirs("./{}/{}/{}".format(dest_folder, pipeline, mdl))<jupyter_output><empty_output><jupyter_text>We can then provide model combinations created above as arguments to the training script, *train_sealnet.py*. A list of required arguments can be displayed by running the cell below.<jupyter_code>%run train_sealnet.py -h
# iterate over combinations
for row in combinations_1.iterrows():
    
    # read hyperparameters
    t_dir, arch, hyp_st = row[1]['training_dir'], row[1]['model_architecture'], \
                          row[1]['hyperparameter_set']
    out = arch + '_ts-' + t_dir.split('_')[-1]
    
    # check if model is already trained
    if "{}.tar".format(out) in os.listdir('./{}/{}/{}/'.format(dest_folder, pipeline, out)): 
        print('{} was already trained'.format(out))
        continue
    
    print()
    !echo training $out
    print()
    
    # run training
    !python train_sealnet.py --training_dir=$t_dir --model_architecture=$arch \
                             --hyperparameter_set=$hyp_st \
                             --output_name=$out --dest_folder=$dest_folder<jupyter_output>training Unet_ts-vanilla

Epoch 1/75
----------

training 

heatmap loss: 0.17458381634927667
count loss: 6657.09390962841
occupancy loss: 0.0

validation 

heatmap loss: 0.027488740813612706
count loss: 555.4578534004345
occupancy loss: 0.0
training time: 0.0h 37m 21s

Epoch 2/75
----------

training 

heatmap loss: 0.018631256396823302
count loss: 227.32557576500676
occupancy loss: 0.0

validation 

heatmap loss: 0.010977884428580937
count loss: 63.05856414103473
occupancy loss: 0.0
training time: 1.0h 14m 22s

Epoch 3/75
----------

training 

heatmap loss: 0.012972626867151116
count loss: 74.9885837019747
occupancy loss: 0.0

validation 

heatmap loss: 0.014404363794392033
count loss: 45.37391026394929
occupancy loss: 0.0
training time: 1.0h 51m 51s

Epoch 4/75
----------

training 

heatmap loss: 0.011121060806777421
count loss: 57.260663389702856
occupancy loss: 0.0

validation 

heatmap loss: 0.004026610648244039
count loss: 28.759098939440673
occupancy loss: 0.0
training time:[...]<jupyter_text>Validation<jupyter_code><jupyter_output><empty_output><jupyter_text>Pipeline 2.1 - Heatmap + count---Here we will generate seal counting CNNs, train them and validate them. Seal counting CNNs will be trained to minimize the mean squared error (MSE) between predicted counts and ground-truth counts. Though they will be trained and validated separately from the haul out detector (Pipeline 1), these approaches will be tested on top of the haul out detector and as standalones.  TrainingSimilar to the previous pipeline, we will store results in folders (under './saved_models') named after each specific model combination for bookkeeping.<jupyter_code># switch pipeline
pipeline = 'Heatmap-cnt'

# generate model combinations
combinations_21 = {'model_architecture': ['UnetCntWRN'],
                   'training_dir': ['training_set_vanilla'],
                   'hyperparameter_set': ['E']}       

# read as a DataFrame
combinations_21 = pd.DataFrame(combinations_21)
                    

# create folders for resulting files
for row in combinations_21.iterrows():
    mdl = row[1]['model_architecture'] + '_ts-' + row[1]['training_dir'].split('_')[-1]               
    if not os.path.exists("./{}/{}/{}".format(dest_folder, pipeline, mdl)):
        os.makedirs("./{}/{}/{}".format(dest_folder, pipeline, mdl))<jupyter_output><empty_output><jupyter_text>To train a counting model, model combinations created above are used as argument to to a new training script, *train_sealnet_count.py*, which uses MSE loss. It accepts the same arguments as the previous.<jupyter_code># iterate over combinations
for row in combinations_21.iterrows():
    
    # read hyperparameters
    t_dir, arch, hyp_st = row[1]['training_dir'], row[1]['model_architecture'], \
                          row[1]['hyperparameter_set']
    out = arch + '_ts-' + t_dir.split('_')[-1]
    
    # check if model is already trained
    if "{}.tar".format(out) in os.listdir('./{}/{}/{}/'.format(dest_folder, pipeline, out)): 
        print('{} was already trained'.format(out))
        continue
    
    print()
    !echo training $out
    print()
    
    # run training
    !python train_sealnet.py --training_dir=$t_dir --model_architecture=$arch \
                             --hyperparameter_set=$hyp_st --output_name=$out  \
                             --dest_folder=$dest_folder<jupyter_output>training WideResnetCount_ts-vanilla

Epoch 1/5
----------
Traceback (most recent call last):
  File "train_sealnet_count.py", line 236, in <module>
    main()
  File "train_sealnet_count.py", line 232, in main
    num_epochs=hyperparameters[args.hyperparameter_set]['epochs'])
  File "train_sealnet_count.py", line 146, in train_model
    for data in dataloaders[phase]:
  File "/home/bento/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 286, in __next__
    return self._process_next_batch(batch)
  File "/home/bento/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 307, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
OSError: Traceback (most recent call last):
  File "/home/bento/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 57, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/bento/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.[...]<jupyter_text>Validation  Pipeline 2.2 - Heatmap + occupancy ---  Training<jupyter_code># switch pipeline
pipeline = 'Heatmap-occ'

# generate model combinations
combinations_22 = {'model_architecture': ['UnetOccDense'],
                   'training_dir': ['training_set_vanilla'],
                   'hyperparameter_set': ['E'] }       

# read as a DataFrame
combinations_22 = pd.DataFrame(combinations_22)
                    

# create folders for resulting files
for row in combinations_22.iterrows():
    mdl = row[1]['model_architecture'] + '_ts-' + row[1]['training_dir'].split('_')[-1]                  
    if not os.path.exists("./{}/{}/{}".format(dest_folder, pipeline, mdl)):
        os.makedirs("./{}/{}/{}".format(dest_folder, pipeline, mdl)) 
# iterate over combinations
for row in combinations_22.iterrows():
    
    # read hyperparameters
    t_dir, arch, hyp_st = row[1]['training_dir'], row[1]['model_architecture'], \
                          row[1]['hyperparameter_set']
    out = arch + '_ts-' + t_dir.split('_')[-1]
    
    # check if model is already trained
    if "{}.tar".format(out) in os.listdir('./{}/{}/{}/'.format(dest_folder, pipeline, out)): 
        print('{} was already trained'.format(out))
        #continue
    
    print()
    !echo training $out
    print()
    
    # run training
    !python train_sealnet.py --training_dir=$t_dir --model_architecture=$arch \
                             --hyperparameter_set=$hyp_st --output_name=$out  \
                             --dest_folder=$dest_folder<jupyter_output>training UnetDet3Dense_ts-vanilla

Epoch 1/75
----------

training 

train_sealnet_det3_L1.py:277: UserWarning: torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.
  nn.utils.clip_grad_norm(model.parameters(), 0.5)

 0 training iterations
   Hubber loss count: 0.004639265537261963
   BCE loss: 0.007423256039619446
   Occupancy loss: 0.012063369750976563
   Total loss: 0.024125891327857973

 200 training iterations
   Hubber loss count: 0.46058855775372054
   BCE loss: 0.4915643134346409
   Occupancy loss: 1.0132792065089702
   Total loss: 1.9654320776973315

 400 training iterations
   Hubber loss count: 0.5155673709519228
   BCE loss: 0.46340175502996145
   Occupancy loss: 1.1233477759469919
   Total loss: 2.102316901928876

 600 training iterations
   Hubber loss count: 0.49676646453236
   BCE loss: 0.40183933784053594
   Occupancy loss: 1.091930503409359
   Total loss: 1.990536305782255

 800 training iterations
   Hubber loss count: 0.533261[...]<jupyter_text>Validation  Pipeline 3 - Heatmap + Count + Occupancy  Training<jupyter_code># switch pipeline
pipeline = 'Heatmap-Cnt-Occ'

# generate model combinations
combinations_3 = {'model_architecture': ['UnetCntWRNOccDense'],
                   'training_dir': ['training_set_vanilla'],
                   'hyperparameter_set': ['E'] }       

# read as a DataFrame
combinations_3 = pd.DataFrame(combinations_3)
                    

# create folders for resulting files
for row in combinations_3.iterrows():
    mdl = row[1]['model_architecture'] + '_ts-' + row[1]['training_dir'].split('_')[-1]                  
    if not os.path.exists("./{}/{}/{}".format(dest_folder, pipeline, mdl)):
        os.makedirs("./{}/{}/{}".format(dest_folder, pipeline, mdl)) 
# iterate over combinations
for row in combinations_22.iterrows():
    
    # read hyperparameters
    t_dir, arch, hyp_st = row[1]['training_dir'], row[1]['model_architecture'], \
                          row[1]['hyperparameter_set']
    out = arch + '_ts-' + t_dir.split('_')[-1]
    
    # check if model is already trained
    if "{}.tar".format(out) in os.listdir('./{}/{}/{}/'.format(dest_folder, pipeline, out)): 
        print('{} was already trained'.format(out))
        #continue
    
    print()
    !echo training $out
    print()
    
    # run training
    !python train_sealnet.py --training_dir=$t_dir --model_architecture=$arch \
                             --hyperparameter_set=$hyp_st --output_name=$out  \
                             --dest_folder=$dest_folder<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11717576364546821, 'fraction_numerical': 0.055225695686386723, 'mean_word_length': 3.4966227082663237, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '117831', 'license': 'MIT', 'n_tokens_mistral': 4855, 'n_tokens_neox': 4298, 'n_words': 1195, 'path': 'src/iceberg_seals/search/training/model_search_stable.ipynb', 'repo_name': 'iceberg-project/Seals', 'size': '13979'}