<jupyter_start><jupyter_text>The original notebook is made by Ratthachat [Jung] (https://github.com/ratthachat) 1. Initial Setup- Mount google drive (optional if you want to save weights to Gdrive)- Clone Nitin's forked resnest repo, - Re-Install kaggle (optional if you want to upload dataset) / albumentations -- Since Colab contains already the old versions, we need to uninstall them first- Download and extract wheat dataset<jupyter_code>!nvidia-smi
!pip uninstall kaggle 

!pip3 install -U kaggle 

import os
os.environ["KAGGLE_USERNAME"] = "" # Specify your Kaggle Username and Key (see kaggle.json)
os.environ["KAGGLE_KEY"] = "" # input Key here

from google.colab import drive
drive.mount('/content/drive')
!pip uninstall albumentations
!pip install albumentations
!git clone -b develop --single-branch https://github.com/ratthachat/wheatdetection
#git://sub.domain.com/repo.git

!kaggle competitions download -c global-wheat-detection
DATA_PATH = './global-wheat-detection/'
!mkdir {DATA_PATH}
!unzip -q global*.zip -d {DATA_PATH}
!ls -sh {DATA_PATH}
%cd wheatdetection
!ls -sh<jupyter_output>/content/wheatdetection
total 56K
4.0K config   4.0K engine    4.0K layers     4.0K solver  4.0K utils
4.0K configs  4.0K evaluate  4.0K modeling   4.0K tests
4.0K data     4.0K external  8.0K README.md  4.0K tools<jupyter_text>The below code is commented out and can be used to change the LR scheduler from `CosineAnnealingWarmRestarts` to `ReduceLROnPlateau`Steps to change the LRScheduler are: 1.   Go to training cell2.   Add `SOLVER.SCHEDULER_NAME "ReduceLROnPlateau"` at the end<jupyter_code># %%writefile ./solver/lr_scheduler.py 
# # encoding: utf-8
# """
# @author:  nitindatta
# @contact: <EMAIL>
# """
# import torch

# def make_scheduler(cfg, optimizer, train_loader):
#     scheduler = getattr(torch.optim.lr_scheduler, cfg.SOLVER.SCHEDULER_NAME)(optimizer,mode='min',factor=0.5,patience=2,threshold_mode='abs',cooldown=1,min_lr=1e-8,eps=1e-08)
#     return scheduler<jupyter_output><empty_output><jupyter_text>2. Training and/or Finetuning<jupyter_code>FOLD, EPOCHS = 1,78 # Change the epochs and fold values here
OUTPUT_PATH = "/content/drive/My Drive/experiments"<jupyter_output><empty_output><jupyter_text>2.1 Training - If you want to save output into your Gdrive, you can modify `OUTPUT_DIR` argument- If you want to finetune you can skip this one and go to 2.2 . - I adopt HSV parameters from YoloV5- CutOut can remove the small wheats in an image, so I disable it (Note that the original ResNest author mis-spell CutOut option to '`COTOUT`')<jupyter_code>!python ./tools/train_net.py OUTPUT_DIR '{OUTPUT_PATH}' DATASETS.VALID_FOLD '{FOLD}' \
SOLVER.MAX_EPOCHS '{EPOCHS}' INPUT.HSV_H '0.014' INPUT.HSV_S '0.68' INPUT.HSV_V '0.36' \
INPUT.COTOUT_NUM_HOLES '16' SOLVER.MOMENTUM '0.937'<jupyter_output>2020-08-03 15:23:38,829 wheatdetection INFO: Using 1 GPUS
2020-08-03 15:23:38,830 wheatdetection INFO: Namespace(config_file='', opts=['OUTPUT_DIR', '/content/drive/My Drive/experiments', 'DATASETS.VALID_FOLD', '1', 'SOLVER.MAX_EPOCHS', '78', 'INPUT.HSV_H', '0.014', 'INPUT.HSV_S', '0.68', 'INPUT.HSV_V', '0.36', 'INPUT.COTOUT_NUM_HOLES', '16', 'SOLVER.MOMENTUM', '0.937'])
2020-08-03 15:23:38,831 wheatdetection INFO: Running with config:
DATALOADER:
  NUM_WORKERS: 2
DATASETS:
  ROOT_DIR: /content/global-wheat-detection
  VALID_FOLD: 1
DEBUG: False
INPUT:
  BC_B: 0.2
  BC_C: 0.2
  BC_PROB: 0.9
  COLOR_PROB: 0.9
  COTOUT_FILL_VALUE: 0
  COTOUT_MAX_H_SIZE: 64
  COTOUT_MAX_W_SIZE: 64
  COTOUT_NUM_HOLES: 16
  COTOUT_PROB: 0.5
  HFLIP_PROB: 0.5
  HSV_H: 0.014
  HSV_PROB: 0.9
  HSV_S: 0.68
  HSV_V: 0.36
  RSC_HEIGHT: 1024
  RSC_MIN_MAX_HEIGHT: (800, 800)
  RSC_PROB: 0.5
  RSC_WIDTH: 1024
  TOFGRAY_PROB: 0.01
  VFLIP_PROB: 0.5
MODEL:
  DEVICE: cuda
  NUM_CLASSES: 2
OUTPUT_DIR: /content/drive/My [...]<jupyter_text>```function ClickConnect(){console.log("Working"); document  .querySelector('top-toolbar > colab-connect-button')  .shadowRoot.querySelector('connect')  .click() }setInterval(ClickConnect,60000)```<jupyter_code># %cd ..
# !pwd
!ls -lh {OUTPUT_PATH}<jupyter_output>total 1.2G
-rw-r--r-- 1 root root 2.0M Jul  9 13:05 all_predictions.csv
-rw-r--r-- 1 root root 489M Jul  9 13:05 best-checkpoint.bin
-rw-r--r-- 1 root root 246M Jul  9 13:06 best-model.bin
-rw-r--r-- 1 root root 489M Jul 11 09:27 last-checkpoint.bin
-rw-r--r-- 1 root root 4.3K Jul 11 09:29 log.txt<jupyter_text>2.1.1 Save fileOptional step, if you want to upload the saved weights to Kaggle dataset<jupyter_code>print(FOLD,EPOCHS)
cmd = """
cat <<EOF > %sdataset-metadata.json
{
  "title": "resnest-f%d-%de",
  "id": "models/f%d-%de",
  "licenses": [
    {
      "name": "CC0-1.0"
    }
  ]
}
EOF
""" % (OUTPUT_PATH, FOLD,EPOCHS,FOLD,EPOCHS)
!{cmd}
!kaggle datasets create --dir-mode zip -p {OUTPUT_PATH} # ADD -u if public<jupyter_output>all_predictions.csv  best-model.bin	    last-checkpoint.bin
best-checkpoint.bin  dataset-metadata.json  log.txt
Starting upload for file last-checkpoint.bin
100% 488M/488M [00:04<00:00, 108MB/s]
Upload successful: last-checkpoint.bin (488MB)
Starting upload for file best-model.bin
100% 245M/245M [00:02<00:00, 106MB/s]
Upload successful: best-model.bin (245MB)
Starting upload for file best-checkpoint.bin
100% 488M/488M [00:04<00:00, 111MB/s]
Upload successful: best-checkpoint.bin (488MB)
Starting upload for file all_predictions.csv
100% 1.97M/1.97M [00:00<00:00, 7.27MB/s]
Upload successful: all_predictions.csv (2MB)
Starting upload for file log.txt
100% 15.6k/15.6k [00:00<00:00, 81.0kB/s]
Upload successful: log.txt (16KB)
Your public Dataset is being created. Please check progress at https://www.kaggle.com/ratthachat/f0-35e<jupyter_text>2.2. FinetuningTo finetune from the last checkpoint, we must copy `last-checkpoint.bin` into the `OUTPUT_DIR`. In this example, I download from my Kaggle's dataset (you can download from your GDrive)<jupyter_code>%cd wheatdetection
!kaggle datasets download ratthachat/f0-35e
!mkdir {OUTPUT_PATH}
!unzip -q f0-35e.zip -d {OUTPUT_PATH}
!ls -sh {OUTPUT_PATH}<jupyter_output>total 1.2G
2.0M all_predictions.csv  246M best-model.bin	     16K log.txt
489M best-checkpoint.bin  489M last-checkpoint.bin<jupyter_text>2.2.1 Start Finetuning hereYou can see all options here : https://github.com/ratthachat/wheatdetection/blob/develop/config/defaults.pyNote that - load previous checkpoint using `SOLVER.TRAIN_CHECKPOINT True`- Remember `last-checkpoint.bin` must already inside the `OUTPUT_DIR` - I guess we should reduce `base_lr` / `momentum` to be small for finetuning, so I reduce momentum from 0.9 to 0.5 ... `base_lr` is automatically decayed for each epoch, so we set it to the default value of 0.005  - Default implementation use CosineScheduler. If you don't need CosineScehduler, just simply set `WARMUP_EPOCHS` to be higher than `MAX_EPOCHS` ... (In the code, if `epoch < WARMUP_EPOCHS` , no scheduler will be used )<jupyter_code>FOLD, EPOCHS = 1,90 # EPOCHS continue from the last-checkpoint.bin
# !python ./tools/train_net.py OUTPUT_DIR '{OUTPUT_PATH}' DATASETS.VALID_FOLD 0 \
# SOLVER.MAX_EPOCHS 72 INPUT.HSV_H '0.03' INPUT.HSV_S '0.68' INPUT.HSV_V '0.36' INPUT.BC_B '0.1' \
# INPUT.BC_C '0.1' INPUT.COTOUT_NUM_HOLES '0' SOLVER.BASE_LR '0.003' SOLVER.MOMENTUM '0.5' \
# SOLVER.WARMUP_EPOCHS '200' SOLVER.TRAIN_CHECKPOINT True

!python ./tools/train_net.py OUTPUT_DIR '{OUTPUT_PATH}' DATASETS.VALID_FOLD '{FOLD}' \
SOLVER.MAX_EPOCHS '{EPOCHS}' INPUT.HSV_H '0.014' INPUT.HSV_S '0.68' INPUT.HSV_V '0.36' \
INPUT.COTOUT_NUM_HOLES '16' SOLVER.MOMENTUM '0.937'   SOLVER.TRAIN_CHECKPOINT True<jupyter_output>2020-07-15 12:03:54,438 wheatdetection INFO: Using 1 GPUS
2020-07-15 12:03:54,438 wheatdetection INFO: Namespace(config_file='', opts=['OUTPUT_DIR', '/content/drive/My Drive/experiments', 'DATASETS.VALID_FOLD', '0', 'SOLVER.MAX_EPOCHS', '90', 'INPUT.HSV_H', '0.014', 'INPUT.HSV_S', '0.68', 'INPUT.HSV_V', '0.36', 'SOLVER.BASE_LR', '0.003', 'INPUT.COTOUT_NUM_HOLES', '4', 'SOLVER.MOMENTUM', '0.5', 'INPUT.TOFGRAY_PROB', '0.', 'SOLVER.TRAIN_CHECKPOINT', 'True'])
2020-07-15 12:03:54,438 wheatdetection INFO: Running with config:
DATALOADER:
  NUM_WORKERS: 2
DATASETS:
  ROOT_DIR: /content/global-wheat-detection
  VALID_FOLD: 0
DEBUG: False
INPUT:
  BC_B: 0.2
  BC_C: 0.2
  BC_PROB: 0.9
  COLOR_PROB: 0.9
  COTOUT_FILL_VALUE: 0
  COTOUT_MAX_H_SIZE: 64
  COTOUT_MAX_W_SIZE: 64
  COTOUT_NUM_HOLES: 4
  COTOUT_PROB: 0.5
  HFLIP_PROB: 0.5
  HSV_H: 0.014
  HSV_PROB: 0.9
  HSV_S: 0.68
  HSV_V: 0.36
  RSC_HEIGHT: 1024
  RSC_MIN_MAX_HEIGHT: (800, 800)
  RSC_PROB: 0.5
  RSC_WIDTH: 1024
  TOFGRAY_PROB: 0.0
  [...]<jupyter_text>2.2.2 Optional - Save weights to Kaggle dataset<jupyter_code>EPOCHS=75 # if you manually interrupted the epoch, you can change name here
OUTPUT_PATH 
cmd = """
cat <<EOF > "%s"/dataset-metadata.json
{
  "title": "resnest-f%d-%de",
  "id": "nitindatta/f%d-%de",
  "licenses": [
    {
      "name": "CC0-1.0"
    }
  ]
}
EOF
""" % (OUTPUT_PATH, FOLD,EPOCHS,FOLD,EPOCHS)
!{cmd}
!cat "/content/drive/My Drive/experiments/dataset-metadata.json"
!kaggle datasets create --dir-mode zip -p {"/content/drive/My Drive/experiments/"} # ADD -u if public<jupyter_output>{
  "title": "resnest-f0-75e",
  "id": "nitindatta/f0-75e",
  "licenses": [
    {
      "name": "CC0-1.0"
    }
  ]
}
usage: kaggle [-h] [-v] {competitions,c,datasets,d,kernels,k,config} ...
kaggle: error: unrecognized arguments: Drive/experiments/
{'fraction_non_alphanumeric': 0.13423174872665536, 'fraction_numerical': 0.062075551782682516, 'mean_word_length': 4.57033096926714, 'pattern_counts': {'":': 12, '<': 41, '<?xml version=': 0, '>': 34, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '83706', 'license': 'MIT', 'n_tokens_mistral': 4177, 'n_tokens_neox': 3780, 'n_words': 920, 'path': 'Faster R-CNN Resnest-101.ipynb', 'repo_name': 'NitinDatta8/GWD-Faster-R-CNN', 'size': '9434'}