<jupyter_start><jupyter_text>rareplanes-tltTrain satellite imagery objection detection models with AI.Reverie's synthetic data and NVIDIA's Transfer Learning Toolkit[RarePlanes Dataset Homepage](https://www.cosmiqworks.org/rareplanes/)  Learning Objectives* Take a pretrained resnet18 model and train a ResNet-18 DetectNet_v2 model on AI.Reverie's RarePlanes synthetic dataset* Take the best performing model weights and finetune them with 10% of the real world data included in the RarePlanes dataset* Prune the trained detectnet_v2 model* Retrain the pruned model to recover lost accuracy* Export the pruned model* Quantize the pruned model using QAT* Run Inference on the trained model  Prerequisites* We are testing our code with Python 3.8.8, using Anaconda 4.9.2 to manage our dependencies and virtual environment. Our code may work with different versions of Python and other virtual environment solutions, but we have not tested those configurations.* We are using Ubuntu 18.04.5 LTS* We are using NVIDIA driver 460.32.03 and CUDA Version 11.2. TLT requires driver 455.xx or above.* You'll need to set up the NVIDIA Container Toolkit / nvidia-docker2. For more information see the [install guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html).* Set up NGC to be able to download NVIDIA docker containers. Follow steps 4 and 5 in the [TLT User Guide](https://docs.nvidia.com/metropolis/TLT/tlt-user-guide/text/requirements_and_installation.htmlinstallation-prerequisites). The download link for the NGC CLI tool is located [here](https://ngc.nvidia.com/setup/installers/cli).* At least 250 GB hard disk space to store dataset and model weights  Setting up a virtual environmentClone this repo:```git clone <EMAIL>:aireveries/rareplanes-tlt.git ~/Code/rareplanes-tlt```Create a conda environment:```conda create -f env.yaml```Activate:```source activate rareplanes-tlt```<jupyter_code>import os
from pathlib import Path<jupyter_output><empty_output><jupyter_text>Downloading the datasetsThere is detailed information about the contents of the RarePlanes dataset available in the [user guide](https://www.cosmiqworks.org/rareplanes-public-user-guide).For our tutorial, we only need to download a subset of the data.First, create the folders:<jupyter_code>!mkdir -p data/real/tarballs/{train,test}
!mkdir -p data/synthetic<jupyter_output><empty_output><jupyter_text>Now download:<jupyter_code>def download(s3_path, out_folder, out_file_count):
    rel_file_path = Path('data') / Path(s3_path.replace('s3://rareplanes-public/', ''))
    rel_folder = rel_file_path.parent / out_folder
    num_files = !ls $rel_folder | wc -l
    try:
        if int(num_files[0]) == out_file_count:
            print(f'{s3_path} already downloaded and extracted')
        else:
            raise Exception
    except:
        if not rel_file_path.exists():
            print('Starting download')
            !aws s3 cp $s3_path $rel_file_path;   
        else:
            print(f'{s3_path} already downloaded')
        print('Extracting...')
        !cd {rel_folder.parent}; pv {rel_file_path.name} | tar xz;
        print('Removing compressed file.')
        !rm $rel_file_path
download('s3://rareplanes-public/real/tarballs/metadata_annotations.tar.gz', 
         'metadata_annotations', 9)
download('s3://rareplanes-public/real/tarballs/train/RarePlanes_train_PS-RGB_tiled.tar.gz', 
         'PS-RGB_tiled', 11630)
download('s3://rareplanes-public/real/tarballs/test/RarePlanes_test_PS-RGB_tiled.tar.gz', 
          'PS-RGB_tiled', 5420)
!aws s3 cp --recursive s3://rareplanes-public/synthetic/ data/synthetic
!du -h data<jupyter_output>131G	data/synthetic/train/images
59G	data/synthetic/train/masks
1.1G	data/synthetic/train/xmls
190G	data/synthetic/train
335M	data/synthetic/metadata_annotations
15G	data/synthetic/test/images
6.6G	data/synthetic/test/masks
116M	data/synthetic/test/xmls
22G	data/synthetic/test
212G	data/synthetic
11M	data/kitti/real_test/images
11M	data/kitti/real_test/labels
22M	data/kitti/real_test
20M	data/kitti/synthetic_test/images
20M	data/kitti/synthetic_test/labels
40M	data/kitti/synthetic_test
24M	data/kitti/real_train/images
24M	data/kitti/real_train/labels
47M	data/kitti/real_train
179M	data/kitti/synthetic_train/images
179M	data/kitti/synthetic_train/labels
357M	data/kitti/synthetic_train
465M	data/kitti
2.0G	data/real/tarballs/train/PS-RGB_tiled
3.9G	data/real/tarballs/train
76M	data/real/tarballs/metadata_annotations
903M	data/real/tarballs/test/PS-RGB_tiled
1.8G	data/real/tarballs/test
5.7G	data/real/tarballs
5.7G	data/real
218G	data<jupyter_text>Converting from COCO to KITTI format TLT uses the KITTI format for object detection model training. RarePlanes is in the COCO format, so we need to run a conversion script.<jupyter_code>%run convert_coco_to_kitti.py<jupyter_output>4%|▍         | 115/2710 [00:00<00:02, 1146.11it/s]<jupyter_text>There should now be a folder for each dataset split inside of `data/kitti` that contains the KITTI formatted annotation text files and symlinks to the original images.<jupyter_code>!ls data
!ls data/kitti
!ls data/kitti/real_train
!ls data/kitti/real_train/images | head -n5
!ls data/kitti/real_train/labels | head -n5
!file -L data/kitti/real_train/images/100_1040010029990A00_tile_319.png
!cat data/kitti/real_train/labels/100_1040010029990A00_tile_319.txt<jupyter_output>aircraft 0 0 0 271.2198036387563 126.75882516428828 383.1428900398314 216.42792563699186 0 0 0 0 0 0 0
aircraft 0 0 0 429.42371606081724 130.2578795682639 512.0 229.21113768219948 0 0 0 0 0 0 0<jupyter_text>Setting up TLT mounts<jupyter_code># Mapping up the local directories to the TLT docker.
import json
mounts_file = os.path.expanduser("~/.tlt_mounts.json")
source = !pwd

drive_map = {
    "Mounts": [
        {
            "source": source[0],
            "destination": "/workspace/tlt-experiments"
        }
    ],
    # Update if you prefer to use a different GPU in multi-GPU systems
    "Envs": [
        {
            "variable": "CUDA_VISIBLE_DEVICES",
            "value": "0"
        }
    ],
    # Update based on your system's available resources
    "DockerOptions": {
        "shm_size": "16G",
        "ulimits": {
            "memlock": -1,
            "stack": 67108864
        },
        "user": "1001:1001"
    }
}

# Writing the mounts file.
with open(mounts_file, "w") as mfile:
    json.dump(drive_map, mfile, indent=4)
!cat ~/.tlt_mounts.json<jupyter_output>{
    "Mounts": [
        {
            "source": "/home/patrick.rodriguez/Code/rareplanes-tlt",
            "destination": "/workspace/tlt-experiments"
        }
    ],
    "Envs": [
        {
            "variable": "CUDA_VISIBLE_DEVICES",
            "value": "0"
        }
    ],
    "DockerOptions": {
        "shm_size": "16G",
        "ulimits": {
            "memlock": -1,
            "stack": 67108864
        },
        "user": "1001:1001"
    }
}<jupyter_text>Processing datasets into TFRecords We need to turn the KITTI labels into the TFRecord format used by TLT.<jupyter_code>!cat specs/detectnet_v2_tfrecords_kitti_real_train.txt
a = 'hello'
!echo {a}
def convert_split(name):
    !tlt detectnet_v2 dataset_convert --gpu_index 0 \
        -d /workspace/tlt-experiments/specs/detectnet_v2_tfrecords_{name}.txt \
        -o /workspace/tlt-experiments/data/tfrecords/{name}/{name}

# tlt detectnet_v2 dataset_convert --gpu_index 0 \
#     -d /workspace/tlt-experiments/specs/detectnet_v2_tfrecords_kitti_real_train.txt \
#     -o /workspace/tlt-experiments/data/tfrecords/kitti_real_train/kitti_real_train
convert_split('kitti_real_train')
convert_split('kitti_real_test')
convert_split('kitti_synthetic_train')
convert_split('kitti_synthetic_test')<jupyter_output>Matplotlib created a temporary config/cache directory at /tmp/matplotlib-l6ltk9a9 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2021-04-06 05:15:52,498 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2021-04-06 05:15:52,498 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Creating output directory /workspace/tlt-experiments/data/tfrecords/kitti_synthetic_test
2021-04-06 05:15:52,513 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 0	Val: 5000
2021-04-06 05:15:52,513 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in pa[...]<jupyter_text>Download the ResNet18 convolutional backbone We can download this model through NGC.<jupyter_code>!ngc registry model download-version nvidia/tlt_pretrained_detectnet_v2:resnet18<jupyter_output>Downloaded 82.28 MB in 18s, Download speed: 4.56 MB/s               
----------------------------------------------------
Transfer id: tlt_pretrained_detectnet_v2_vresnet18 Download status: Completed.
Downloaded local path: /home/patrick.rodriguez/Code/rareplanes-tlt/tlt_pretrained_detectnet_v2_vresnet18
Total files downloaded: 1 
Total downloaded size: 82.28 MB
Started at: 2021-04-06 05:21:04.080455
Completed at: 2021-04-06 05:21:22.107625
Duration taken: 18s
----------------------------------------------------<jupyter_text>The model should now be located at: `./tlt_pretrained_detectnet_v2_vresnet18/resnet18.hdf5`<jupyter_code>!ls -lh ./tlt_pretrained_detectnet_v2_vresnet18<jupyter_output>total 90M
-rw------- 1 patrick.rodriguez patrick.rodriguez 90M Apr  6 05:21 resnet18.hdf5<jupyter_text>Run a benchmark experiment using real data We want to see how well a model trained on the full set of real world data performs. If you have a GPU that supported mixed precision training, we recommend using the `--use_amp` option:<jupyter_code>!tlt detectnet_v2 train --key tlt --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_train_resnet18_kitti_real.txt \
    -r /workspace/tlt-experiments/detectnet_v2_outputs/resnet18_real_amp16 \
    -n resnet18_real_amp16 \
    --use_amp > out_resnet18_real_amp16.log<jupyter_output><empty_output><jupyter_text>We send the output to a logfile so we can easily reference it later. To follow along while the training is in progress, execute this command in a different shell:```tail -f ./out_resnet18_real_amp16.log```<jupyter_code>def get_model_param_counts(log_file):
    !cat {log_file} | grep -i params

def get_best_epoch(log_file):  
    accs = !cat {log_file} | grep -i aircraft | cut -d' ' -f 2-
    accs = zip(range(1, len(accs)+1), [float(acc.strip()) for acc in accs])

    best_epoch = max(accs, key=lambda x: x[1])
    return best_epoch

def get_best_checkpoint(output_folder, best_epoch):
    p = Path(output_folder) / 'model*.tlt'
    checkpoints = !ls -v {p}
    best_checkpoint = checkpoints[best_epoch[0]]
    return best_checkpoint
get_model_param_counts('./out_resnet18_real_amp16.log')
best_epoch = get_best_epoch('./out_resnet18_real_amp16.log')
best_epoch
best_checkpoint = get_best_checkpoint('./detectnet_v2_outputs/resnet18_real_amp16/', best_epoch)
best_checkpoint
!tlt detectnet_v2 evaluate --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_evaluate_real.txt \
    -m /workspace/tlt-experiments/{best_checkpoint} \
    -k tlt<jupyter_output>Matplotlib created a temporary config/cache directory at /tmp/matplotlib-i7nme6tx because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

2021-04-06 05:46:33,294 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tlt-experiments/specs/de[...]<jupyter_text>Running an experiment with synthetic data Run the next cell and tail its logs in another shell. To narrow down on model performance, you can run `cat out_resnet18_synth_amp16.log | grep -i aircraft` and see the validation score per epoch. Feel free to interupt the cell execution once you are satisfied with the results. You should be getting 70-73 mAP50 before continuing.<jupyter_code>!tlt detectnet_v2 train --key tlt --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_train_resnet18_kitti_synth.txt \
    -r /workspace/tlt-experiments/detectnet_v2_outputs/resnet18_synth_amp16 \
    -n resnet18_synth_amp16 \
    --use_amp > out_resnet18_synth_amp16.log
!cat out_resnet18_synth_amp16.log | grep -i aircraft
get_model_param_counts('./out_resnet18_synth_amp16.log')
best_epoch = get_best_epoch('./out_resnet18_synth_amp16.log')
best_epoch
best_checkpoint = get_best_checkpoint('./detectnet_v2_outputs/resnet18_synth_amp16', best_epoch)
best_checkpoint
!tlt detectnet_v2 evaluate --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_evaluate_synth_on_real.txt \
    -m /workspace/tlt-experiments/{best_checkpoint} \
    -k tlt<jupyter_output>Matplotlib created a temporary config/cache directory at /tmp/matplotlib-49zu0jym because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

2021-04-06 16:31:40,401 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tlt-experiments/specs/de[...]<jupyter_text>Finetuning the synthetic trained model with real data We will fine tune our best performing synthetic-data trained model with 10% of the real data. To do so, we must first create the 10% split.<jupyter_code>%run ./create_train_split.py<jupyter_output>100%|██████████| 581/581 [00:00<00:00, 858.59it/s]<jupyter_text>Then, we create the tfrecords:<jupyter_code>convert_split('kitti_real_train_10')
with open('./specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10.txt', 'r') as f_in:
    with open('./specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10_replaced.txt', 'w') as f_out:
        out = f_in.read().replace('REPLACE', best_checkpoint)
        f_out.write(out)
!tlt detectnet_v2 train --key tlt --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10_replaced.txt \
    -r /workspace/tlt-experiments/detectnet_v2_outputs/resnet18_synth_finetune_10_amp16 \
    -n resnet18_synth_finetune_10_amp16 \
    --use_amp > out_resnet18_synth_finetune_10_amp16.log
get_model_param_counts('./out_resnet18_synth_finetune_10_amp16.log')
best_epoch = get_best_epoch('./out_resnet18_synth_finetune_10_amp16.log')
best_epoch
best_checkpoint = get_best_checkpoint('./detectnet_v2_outputs/resnet18_synth_finetune_10_amp16', best_epoch)
best_checkpoint
!tlt detectnet_v2 evaluate --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_evaluate_real.txt \
    -m /workspace/tlt-experiments/{best_checkpoint} \
    -k tlt<jupyter_output>Matplotlib created a temporary config/cache directory at /tmp/matplotlib-yxzq9tsr because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

2021-04-06 18:04:59,652 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tlt-experiments/specs/de[...]<jupyter_text>Pruning the model<jupyter_code>!mkdir -p detectnet_v2_outputs/pruned

!tlt detectnet_v2 prune \
    -m /workspace/tlt-experiments/{best_checkpoint} \
    -o /workspace/tlt-experiments/detectnet_v2_outputs/pruned/pruned-model.tlt \
    -eq union \
    -pth 0.5 \
    -k tlt<jupyter_output>Matplotlib created a temporary config/cache directory at /tmp/matplotlib-mlhhw0sq because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2021-04-06 18:14:37,641 [INFO] modulus.pruning.pruning: Exploring graph for retainable indices
2021-04-06 18:14:38,335 [INFO] modulus.pruning.pruning: Pruning model and appending pruned nodes to new graph
2021-04-06 18:14:55,235 [INFO] iva.common.magnet_prune: Pruning ratio (pruned model / original model): 0.30121496963759165
2021-04-06 18:14:56,863 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.<jupyter_text>Evaluate the pruned model:<jupyter_code>!tlt detectnet_v2 evaluate --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_evaluate_real.txt \
    -m /workspace/tlt-experiments/detectnet_v2_outputs/pruned/pruned-model.tlt \
    -k tlt > out_pruned.txt
get_model_param_counts('./out_pruned.txt')

!cat out_pruned.txt | grep -i aircraft<jupyter_output>aircraft                         68.8865<jupyter_text>Retraining the models<jupyter_code>with open('./specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10_pruned_retrain.txt', 'r') as f_in:
    with open('./specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10_pruned_retrain_replaced.txt', 'w') as f_out:
        out = f_in.read().replace('REPLACE', 'detectnet_v2_outputs/pruned/pruned-model.tlt')
        f_out.write(out)
!tlt detectnet_v2 train --key tlt --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10_pruned_retrain_replaced.txt \
    -r /workspace/tlt-experiments/detectnet_v2_outputs/resnet18_synth_finetune_10_pruned_retrain_amp16 \
    -n resnet18_synth_finetune_10_pruned_retrain_amp16 \
    --use_amp > out_resnet18_synth_finetune_10_pruned_retrain_amp16.log
get_model_param_counts('./out_resnet18_synth_finetune_10_pruned_retrain_amp16.log')
best_epoch = get_best_epoch('./out_resnet18_synth_finetune_10_pruned_retrain_amp16.log')
best_epoch
best_checkpoint = get_best_checkpoint('./detectnet_v2_outputs/resnet18_synth_finetune_10_pruned_retrain_amp16', best_epoch)
best_checkpoint
!tlt detectnet_v2 evaluate --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_evaluate_real.txt \
    -m /workspace/tlt-experiments/{best_checkpoint} \
    -k tlt<jupyter_output>Matplotlib created a temporary config/cache directory at /tmp/matplotlib-tzn6gu_8 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

2021-04-06 19:33:09,611 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tlt-experiments/specs/de[...]<jupyter_text>Quantizing the models<jupyter_code>with open('./specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10_pruned_retrain_qat.txt', 'r') as f_in:
    with open('./specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10_pruned_retrain_qat_replaced.txt', 'w') as f_out:
        out = f_in.read().replace('REPLACE', 'detectnet_v2_outputs/pruned/pruned-model.tlt')
        f_out.write(out)
!tlt detectnet_v2 train --key tlt --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10_pruned_retrain_qat_replaced.txt \
    -r /workspace/tlt-experiments/detectnet_v2_outputs/resnet18_synth_finetune_10_pruned_retrain_qat_amp16 \
    -n resnet18_synth_finetune_10_pruned_retrain_qat_amp16 \
    --use_amp > out_resnet18_synth_finetune_10_pruned_retrain_qat_amp16.log
get_model_param_counts('./out_resnet18_synth_finetune_10_pruned_retrain_qat_amp16.log')
best_epoch = get_best_epoch('./out_resnet18_synth_finetune_10_pruned_retrain_qat_amp16.log')
best_epoch
best_checkpoint = get_best_checkpoint('./detectnet_v2_outputs/resnet18_synth_finetune_10_pruned_retrain_qat_amp16', best_epoch)
best_checkpoint
!tlt detectnet_v2 evaluate --gpu_index 0 \
    -e /workspace/tlt-experiments/specs/detectnet_v2_evaluate_real.txt \
    -m /workspace/tlt-experiments/{best_checkpoint} \
    -k tlt
!tlt detectnet_v2 export \
  -m /workspace/tlt-experiments/{best_checkpoint} \
  -o /workspace/tlt-experiments/detectnet_v2_outputs/qat/resnet18_detector_qat.etlt \
  -k tlt  \
  --data_type int8 \
  --batch_size 64 \
  --max_batch_size 64\
  --engine_file /workspace/tlt-experiments/detectnet_v2_outputs/qat/resnet18_detector_qat.trt.int8 \
  --cal_cache_file /workspace/tlt-experiments/detectnet_v2_outputs/qat/calibration_qat.bin \
  --verbose
!tlt detectnet_v2 evaluate -e /workspace/tlt-experiments/specs/detectnet_v2_train_resnet18_kitti_synth_finetune_10_pruned_retrain_qat_replaced.txt \
                           -m /workspace/tlt-experiments/detectnet_v2_outputs/qat/resnet18_detector_qat.trt.int8 \
                           -f tensorrt<jupyter_output>Matplotlib created a temporary config/cache directory at /tmp/matplotlib-4nrn9agi because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.

2021-04-06 23:06:19,732 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tlt-experiments/specs/de[...]
{'fraction_non_alphanumeric': 0.09519116258059217, 'fraction_numerical': 0.05034700833641329, 'mean_word_length': 4.729882352941177, 'pattern_counts': {'":': 24, '<': 61, '<?xml version=': 0, '>': 65, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '337028', 'license': 'MIT', 'n_tokens_mistral': 9432, 'n_tokens_neox': 8547, 'n_words': 1818, 'path': 'Training an Airplane Detector with NVIDIA TLT and AI.Reverie Synthetic Data.ipynb', 'repo_name': 'aireveries/rareplanes-tlt', 'size': '24358'}