<jupyter_start><jupyter_text>Real-time object detection for disaster response using Transfer Learning Toolkit<NAME>, <EMAIL>  SetupSign up for a free NVIDIA GPU Cloud (NGC) account at [ngc.nvidia.com](https://ngc.nvidia.com), then pull the Transfer Learning Toolkit (TLT) container from https://ngc.nvidia.com/catalog/containers/nvidia:tlt-streamanalyticsPull and enter the container    DATA_DIR=/path/to/your/data    WORKING_DIR=/path/to/workingdir  include the "specs" and "deepstream" directories    docker pull nvcr.io/nvidia/tlt-streamanalytics:v1.0_py2    docker run --runtime=nvidia -it -v $DATA_DIR:/data \         -v $WORKING_DIR:/src -p 8888:8888 \             nvcr.io/nvidia/tlt-streamanalytics:v1.0_py2 /bin/bashConfigure TLT to use your NGC API key    ngc config set Set you NGC API key and some local directories<jupyter_code>KEY='<YOUR API KEY>'

#Where we will save our data 
TLT_DIR='/data/tlt_working_dir'

#Where out data is stored
DATA_DIR='/data/stanford/kitti2'<jupyter_output><empty_output><jupyter_text>Download pre-trained modelView models available on NGC.NB: If this doesn't work, did you run `ngc config set`?<jupyter_code>!ngc registry model list *detectnet*<jupyter_output><empty_output><jupyter_text>Download your chosen model<jupyter_code>!ngc registry model download-version nvidia/iva/tlt_resnet50_detectnet_v2:1 -d $TLT_DIR<jupyter_output><empty_output><jupyter_text>Let's examine the contents of that folder, to check that a `.hdf5` file has been downloaded.<jupyter_code>!ls $TLT_DIR/tlt_resnet50_detectnet_v2_v1<jupyter_output><empty_output><jupyter_text>Review dataWe are using the [Stanford Drones Dataset](http://cvgl.stanford.edu/projects/uav_data/). An extract from the raw data annotations.<jupyter_code>!head /data/stanford/annotations/coupa/video0/annotations.txt<jupyter_output><empty_output><jupyter_text>Each video is of a different resolution and duration.<jupyter_code>%%bash
ffmpeg -i /data/stanford/videos/coupa/video0/video.mov 2>&1 | grep Video: | grep -Po '\d{3,5}x\d{3,5}'
ffprobe -i /data/stanford/videos/coupa/video0/video.mov 2>&1 -show_format | grep duration<jupyter_output><empty_output><jupyter_text>I have pre-processed this by :1. Saving out every frame from each video, for example:        ffmpeg -i videos/bookstore/video0/video.mov -qscale:v 2 raw_jpgs/bookstore/bookstore_video0_%06d.jpg2. Randomly selecting some frames from each video        selected_frames = random.sample(frameslist, n_frames_per_vid)3. Randomly cropping each frame to 768 x 768.        im = Image.open(framepath)        width, height = im.size         Select random crop        crop_xmin = random.randint(0, width - crop_w)        crop_ymin = random.randint(0, height - crop_h)        crop_xmax = crop_xmin + crop_w        crop_ymax = crop_ymin + crop_h4. Saving the annotations out in [KITTI format](https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.htmlkitti_file)        kikki_output_list = [label, truncated, occluded, alpha, xmin, ymin, xmax, ymax, height_metres,                             width_metres, length_metres, cam_x, cam_y, cam_z, rot] An example of a KITTI annotation file:<jupyter_code>!head $DATA_DIR/labels/coupa_video0_000047.txt<jupyter_output><empty_output><jupyter_text>An example image<jupyter_code>from IPython.display import display
from PIL import Image
import os

def display_image(img_path):
    pil_im = Image.open(img_path) #Take jpg + png
    im_array = np.asarray(pil_im)
    plt.imshow(im_array)
    plt.show()

path = os.path.join(DATA_DIR, 'images/coupa_video0_000047.jpg')
display(Image.open(path))<jupyter_output><empty_output><jupyter_text>Convert dataset to TFRecordsWe convert our dataset into TFRecords using the `tlt-dataset-convert` command.We use a spec file to describe the dataset: [convert.txt](specs/convert.txt)**Change needed**: Be sure to update `root_directory_path` and `image_directory_path` to the location of your data.<jupyter_code>!more specs/convert.txt<jupyter_output><empty_output><jupyter_text>Now we make out TFRecords, creating the 15% validation split, as specified in convert.txt.<jupyter_code>!tlt-dataset-convert -d specs/convert.txt -o $TLT_DIR/tfrecords<jupyter_output><empty_output><jupyter_text>TrainWe use a spec file to control the training process: [train.txt](specs/train.txt)<jupyter_code># If necessary, first make a directory in which to save your trained model.
!mkdir $TLT_DIR/trained
!tlt-train detectnet_v2 -e specs/train.txt \
        -r $TLT_DIR/trained --gpus 8 -k KEY<jupyter_output><empty_output><jupyter_text>You should now find that your `trained` directory contains a number of models, named `model.step-xxx.tlt`, where for me `xxx` = 133080<jupyter_code>ls $TLT_DIR/trained<jupyter_output><empty_output><jupyter_text>As we are going to use this model in several commands, let's save it as a variable.<jupyter_code>MODEL=os.path.join(TLT_DIR, 'trained', 'model.step-133080.tlt')<jupyter_output><empty_output><jupyter_text>Evaluate<jupyter_code>!tlt-evaluate detectnet_v2 -e specs/train.txt -m $MODEL -k $KEY<jupyter_output><empty_output><jupyter_text>I achieved the following accuracy:    class name      average precision (in %)    ------------  --------------------------    person                           43.9481    vehicle                          72.5628  InferWe use a spec file to control the inference process: [infer.txt](specs/infer.txt)<jupyter_code>INFER_DIR='/data/stanford/kitti2/infer_imgs'
OUTPUT_DIR='~/inferred_images'
!tlt-infer detectnet_v2 -m $MODEL -i $INFER_DIR -o $OUTPUT_DIR -k $KEY -bs 16 -cp specs/infer.txt<jupyter_output><empty_output><jupyter_text>Prune & retrainIn the webinar, I didn't prune or re-train my model due to time constraints. The [Getting Started Guide](https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.htmlpruning_models) contains information about the various pruning options. Prune<jupyter_code>!mkdir $TLT_DIR/pruned
!tlt-prune -pm $MODEL -o $TLT_DIR/pruned -pth 0.30 -nf 16 -k $KEY<jupyter_output><empty_output><jupyter_text>Now we trained the pruned model Re-trainWe use a new spec file to control the training process: [retrain.txt](specs/retrain.txt)Make sure you change the `pretrained_model_file` to the model produced by the pruning process:<jupyter_code>!ls $TLT_DIR/pruned/*<jupyter_output><empty_output><jupyter_text>Now we can re-train<jupyter_code>!mkdir $TLT_DIR/retrained
!tlt-train detectnet_v2 -e specs/retrain.txt \
        -r $TLT_DIR/retrained --gpus 8 -k $KEY<jupyter_output><empty_output><jupyter_text>As before, we need to establish the best trained model and save that to a convenient variable<jupyter_code>!ls $TLT_DIR/retrained
MODEL=os.path.join(TLT_DIR, 'retrained', 'model.step-133080.tlt')<jupyter_output><empty_output><jupyter_text>ExportYou can export in any of FP32, FP16 or INT8 precision. FP32 FP16    tlt-export $TLT_DIR/trained/model.step-133080.tlt -k $API_KEY \        --export_module detectnet_v2 --outputs output_bbox/BiasAdd,output_cov/Sigmoid \        --data_type fp16 --output_file $TLT_DIR/exports/FP16_model.etlt INT8First generate the INT8 calibration file<jupyter_code>#If necessary, make an export directory
!mkdir $TLT_DIR/exports<jupyter_output><empty_output><jupyter_text>Export with FP32 precision<jupyter_code>!tlt-export $MODEL -k $KEY \
    --export_module detectnet_v2 --outputs output_bbox/BiasAdd,output_cov/Sigmoid \
    --data_type fp32 --output_file $TLT_DIR/exports/FP32_model.etlt<jupyter_output><empty_output><jupyter_text>Export with FP16 precision<jupyter_code>!tlt-export $MODEL -k $KEY \
    --export_module detectnet_v2 --outputs output_bbox/BiasAdd,output_cov/Sigmoid \
    --data_type fp16 --output_file $TLT_DIR/exports/FP16_model.etlt<jupyter_output><empty_output><jupyter_text>Export with INT8 precisionFirst we generate an INT8 calibration file<jupyter_code>!tlt-int8-tensorfile detectnet_v2 -e specs/train.txt \
        -o $TLT_DIR/exports/calibration.tensor -m 20<jupyter_output><empty_output><jupyter_text>Now we export the model<jupyter_code>!tlt-export $MODEL -k $KEY --export_module detectnet_v2 --outputs output_bbox/BiasAdd,output_cov/Sigmoid --data_type int8  --output_file $TLT_DIR/exports/INT8_model.etlt  --cal_data_file $TLT_DIR/exports/calibration.tensor --cal_cache_file $TLT_DIR/exports/calibration.bin --input_dims 3,768,768<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10724949241609937, 'fraction_numerical': 0.019586766989131733, 'mean_word_length': 5.046209386281588, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 102, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '383210', 'license': 'MIT', 'n_tokens_mistral': 3079, 'n_tokens_neox': 2923, 'n_words': 672, 'path': 'TLT_notebook.ipynb', 'repo_name': 'NVIDIA-AI-IOT/tlt-iva-examples', 'size': '8392'}