<jupyter_start><jupyter_text>Integrating TAO Models in DeepStream In the first of two notebooks, we will be building a 4-class object detection pipeline as shown in the illustration below using Nvidia's TrafficCamNet pretrained model, directly downloaded from NGC. Note: This notebook has code inspired from a sample application provided by NVIDIA in a GitHub repository. You can find this repository [here](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps). The Pipeline![Test1](../images/test1.png) We notice there are multiple DeepStream plugins used in the pipeline , Let us have a look at them and try to understand them.  NVIDIA DeepStream Plugins NvinferThe nvinfer plugin provides [TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html)-based inference for detection and tracking. The lowlevel library (libnvds_infer) operates either on float RGB or BGR planar data with dimensions of Network Height and Network Width. The plugin accepts NV12/RGBA data from upstream components like the decoder, muxer, and dewarper.The Gst-nvinfer plugin also performs preprocessing operations like format conversion, scaling, mean subtraction, and produces final float RGB/BGR planar data which is passed to the low-level library. The low-level library uses the TensorRT engine for inferencing. It outputs each classified object’s class and each detected object’s bounding boxes (Bboxes) after clustering.![NVINFER](../images/nvinfer.png) Nvvidconv We create the nvvidconv plugin that performs color format conversions, which is required to make data ready for the nvosd plugin.![NVVIDCONV](../images/nvvidconv.png) NvosdThe nvosd plugin draws bounding boxes, text, and RoI (Regions of Interest) polygons (Polygons are presented as a set of lines). The plugin accepts an RGBA buffer with attached metadata from the upstream component. Itdraws bounding boxes, which may be shaded depending on the configuration (e.g. width, color, and opacity) of a given bounding box. It also draws text and RoI polygons at specified locations in the frame. Text and polygon parameters are configurable through metadata.![NVOSD](../images/nvosd.png)Now with this idea , let us get started into building the pipeline.  Building the pipeline ![Test1](../images/test1.png)<jupyter_code># Import Required Libraries 
import sys
sys.path.append('../source_code')
import gi
import time
gi.require_version('Gst', '1.0')
from gi.repository import GObject, Gst, GLib
from common.bus_call import bus_call
import pyds

# Defining the Class Labels
PGIE_CLASS_ID_VEHICLE = 0
PGIE_CLASS_ID_BICYCLE = 1
PGIE_CLASS_ID_PERSON = 2
PGIE_CLASS_ID_ROADSIGN = 3

# Defining the input output video file 
INPUT_VIDEO_NAME  = '../videos/sample_720p.h264'
OUTPUT_VIDEO_NAME = "../videos/out.mp4"<jupyter_output><empty_output><jupyter_text>We define a function `make_elm_or_print_err()` to create our elements and report any errors if the creation fails.Elements are created using the `Gst.ElementFactory.make()` function as part of Gstreamer library.<jupyter_code>## Make Element or Print Error and any other detail
def make_elm_or_print_err(factoryname, name, printedname, detail=""):
  print("Creating", printedname)
  elm = Gst.ElementFactory.make(factoryname, name)
  if not elm:
     sys.stderr.write("Unable to create " + printedname + " \n")
  if detail:
     sys.stderr.write(detail)
  return elm<jupyter_output><empty_output><jupyter_text>Initialise GStreamer and Create an Empty Pipeline<jupyter_code># Standard GStreamer initialization
Gst.init(None)


# Create Gstreamer elements
# Create Pipeline element that will form a connection of other elements
print("Creating Pipeline \n ")
pipeline = Gst.Pipeline()

if not pipeline:
    sys.stderr.write(" Unable to create Pipeline \n")<jupyter_output>Creating Pipeline<jupyter_text>Create Elements that are required for our pipeline<jupyter_code># Creating elements required for the pipeline
# Source element for reading from file
source = make_elm_or_print_err("filesrc", "file-source","Source")
# Parse the data since the input is an elementary .h264 stream
h264parser = make_elm_or_print_err("h264parse", "h264-parser","h264 parse")
# For hardware accelerated decoding of the stream
decoder = make_elm_or_print_err("nvv4l2decoder", "nvv4l2-decoder","Nvv4l2 Decoder")
# Form batches from one or more sources
streammux = make_elm_or_print_err("nvstreammux", "Stream-muxer",'NvStreamMux')
# Run inference on the decoded stream, this property is set through a configuration file later
pgie = make_elm_or_print_err("nvinfer", "primary-inference" ,"pgie")
# Convert output stream to formatted buffer accepted by Nvosd
nvvidconv = make_elm_or_print_err("nvvideoconvert", "convertor","nvvidconv")
# Draw on the buffer
nvosd = make_elm_or_print_err("nvdsosd", "onscreendisplay","nvosd")
# Encode and save the OSD output
queue = make_elm_or_print_err("queue", "queue", "Queue")
# Convert output for saving
nvvidconv2 = make_elm_or_print_err("nvvideoconvert", "convertor2","nvvidconv2")
# Save as video file
encoder = make_elm_or_print_err("avenc_mpeg4", "encoder", "Encoder")
# Parse output from encoder
codeparser = make_elm_or_print_err("mpeg4videoparse", "mpeg4-parser", 'Code Parser')
# Create a container
container = make_elm_or_print_err("qtmux", "qtmux", "Container")
# Create sink for string the output
sink = make_elm_or_print_err("filesink", "filesink", "Sink")<jupyter_output>Creating Source
Creating h264 parse
Creating Nvv4l2 Decoder
Creating NvStreamMux
Creating pgie
Creating nvvidconv
Creating nvosd
Creating Queue
Creating nvvidconv2
Creating Encoder
Creating Code Parser
Creating Container
Creating Sink<jupyter_text>Now that we have created the elements ,we can now set various properties for out pipeline at this point.  Understanding the configuration file We set an `config-file-path` for our nvinfer ( Interference plugin ) and it points to the file `config_infer_primary_trafficcamnet.txt`You can have a have a look at the [file](../configs/config_infer_primary_trafficcamnet.txt)Here are some parts of the configuration file : ``` Copyright (c) 2020 NVIDIA Corporation.  All rights reserved. NVIDIA Corporation and its licensors retain all intellectual property and proprietary rights in and to this software, related documentation and any modifications thereto.  Any use, reproduction, disclosure or distribution of this software and related documentation without an express license agreement from NVIDIA Corporation is strictly prohibited.[property]gpu-id=0net-scale-factor=0.0039215697906911373tlt-model-key=tlt_encodetlt-encoded-model=../models/trafficcamnet/resnet18_trafficcamnet_pruned.etltlabelfile-path=labels_trafficnet.txtint8-calib-file=../models/trafficcamnet/trafficnet_int8.binmodel-engine-file=../models/trafficcamnet/resnet18_trafficcamnet_pruned.etlt_b1_gpu0_int8.engineinput-dims=3;544;960;0uff-input-blob-name=input_1batch-size=1process-mode=1model-color-format=0 0=FP32, 1=INT8, 2=FP16 modenetwork-mode=2num-detected-classes=4interval=0gie-unique-id=1output-blob-names=output_bbox/BiasAdd;output_cov/Sigmoid[class-attrs-all]pre-cluster-threshold=0.2group-threshold=1 Set eps=0.7 and minBoxes for cluster-mode=1(DBSCAN)eps=0.2minBoxes=3```Here we define all the parameters of our model. In this example we use model-file `resnet18_trafficcamnet_pruned`. `Nvinfer` creates an TensorRT Engine specific to the Host GPU to accelerate it's inference performance.<jupyter_code># Set properties for elements
print("Playing file %s" %INPUT_VIDEO_NAME)
# Set input file
source.set_property('location', INPUT_VIDEO_NAME)
# Set input height, width, and batch size
streammux.set_property('width', 1920)
streammux.set_property('height', 1080)
streammux.set_property('batch-size', 1)
# Set timer (in microseconds) to wait after the first buffer is available
# to push the batch even if batch is never completely formed
streammux.set_property('batched-push-timeout', 4000000)
# Set configuration files for Nvinfer
pgie.set_property('config-file-path', "../configs/config_infer_primary_trafficcamnet.txt")
# Set encoder bitrate for output video
encoder.set_property("bitrate", 2000000)
# Set output file location, disable sync and async
sink.set_property("location", OUTPUT_VIDEO_NAME)
sink.set_property("sync", 0)
sink.set_property("async", 0)<jupyter_output>Playing file  ../videos/sample_720p.h264<jupyter_text>We now link all the elements in the order we prefer and create Gstreamer bus to feed all messages through it.<jupyter_code># Add and link all elements to the pipeline
# Adding elements
print("Adding elements to Pipeline \n")

pipeline.add(source)
pipeline.add(h264parser)
pipeline.add(decoder)
pipeline.add(streammux)
pipeline.add(pgie)
pipeline.add(nvvidconv)
pipeline.add(nvosd)
pipeline.add(queue)
pipeline.add(nvvidconv2)
pipeline.add(encoder)
pipeline.add(codeparser)
pipeline.add(container)
pipeline.add(sink)

# Linking elements
# Order: source -> h264parser -> decoder -> streammux -> pgie ->
# -> vidconv -> osd -> queue -> vidconv2 -> encoder -> parser ->
# -> container -> sink

print("Linking elements in the Pipeline \n")
source.link(h264parser)
h264parser.link(decoder)


sinkpad = streammux.get_request_pad("sink_0")
if not sinkpad:
    sys.stderr.write(" Unable to get the sink pad of streammux \n")
# Create source pad from Decoder   
srcpad = decoder.get_static_pad("src")
if not srcpad:
    sys.stderr.write(" Unable to get source pad of decoder \n")
    
srcpad.link(sinkpad)
streammux.link(pgie)
pgie.link(nvvidconv)
nvvidconv.link(nvosd)
nvosd.link(queue)
queue.link(nvvidconv2)
nvvidconv2.link(encoder)
encoder.link(codeparser)
codeparser.link(container)
container.link(sink)
# Create an event loop and feed GStreamer bus messages to it
loop = GLib.MainLoop()
bus = pipeline.get_bus()
bus.add_signal_watch()
bus.connect ("message", bus_call, loop)<jupyter_output><empty_output><jupyter_text>Working with the Metadata Our pipeline now carries the metadata forward but we have not done anything with it until now, but as mentoioned in the above pipeline diagram , we will now create a callback function to write relevant data on the frame once called and create a sink pad in the `nvosd` element to call the function.<jupyter_code># Working with metadata
def osd_sink_pad_buffer_probe(pad,info,u_data):
    
    obj_counter = {
        PGIE_CLASS_ID_VEHICLE:0,
        PGIE_CLASS_ID_PERSON:0,
        PGIE_CLASS_ID_BICYCLE:0,
        PGIE_CLASS_ID_ROADSIGN:0
    }
    # Reset frame number and number of rectanges to zero
    frame_number=0
    num_rects=0
    
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        print("Unable to get GstBuffer ")
        return

    # Retrieve metadata from gst_buffer
    # Note: since we use the pyds shared object library,
    # the input is the C address of gst_buffer
    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        try:
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
        except StopIteration:
            break
        
        # Get frame number, number of rectangles to draw and object metadata
        frame_number=frame_meta.frame_num
        num_rects = frame_meta.num_obj_meta
        l_obj=frame_meta.obj_meta_list
        
        while l_obj is not None:
            try:
                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
            except StopIteration:
                break
            # Increment object class by 1 and set box border color to red
            obj_counter[obj_meta.class_id] += 1
            obj_meta.rect_params.border_color.set(0.0, 0.0, 1.0, 0.0)
            try: 
                l_obj=l_obj.next
            except StopIteration:
                break
        
        # Setting metadata display configuration
        # Acquire display meta object
        display_meta=pyds.nvds_acquire_display_meta_from_pool(batch_meta)
        display_meta.num_labels = 1
        py_nvosd_text_params = display_meta.text_params[0]
        # Set display text to be shown on screen
        py_nvosd_text_params.display_text = "Frame Number={} Number of Objects={} Vehicle_count={} Person_count={}".format(frame_number, num_rects, obj_counter[PGIE_CLASS_ID_VEHICLE], obj_counter[PGIE_CLASS_ID_PERSON])
        # Set where the string will appear
        py_nvosd_text_params.x_offset = 10
        py_nvosd_text_params.y_offset = 12
        # Font, font colour and font size
        py_nvosd_text_params.font_params.font_name = "Serif"
        py_nvosd_text_params.font_params.font_size = 10
        # Set color (We used white)
        py_nvosd_text_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0)
        # Set text background colour (We used black)
        py_nvosd_text_params.set_bg_clr = 1
        py_nvosd_text_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0)
        # Print the display text in the console as well
        print(pyds.get_string(py_nvosd_text_params.display_text))
        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)
        
        
        try:
            l_frame=l_frame.next
        except StopIteration:
            break
    return Gst.PadProbeReturn.OK
# Adding probe to sinkpad of the OSD element
osdsinkpad = nvosd.get_static_pad("sink")
if not osdsinkpad:
    sys.stderr.write(" Unable to get sink pad of nvosd \n")
    
osdsinkpad.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe, 0)<jupyter_output><empty_output><jupyter_text>Now with everything defined , we can start the playback and listen the events.<jupyter_code># Start the pipeline
print("Starting pipeline \n")
start_time = time.time()
pipeline.set_state(Gst.State.PLAYING)
try:
    loop.run()
except:
    pass
# Cleanup
pipeline.set_state(Gst.State.NULL)
print("--- %s seconds ---" % (time.time() - start_time))<jupyter_output>Starting pipeline 

Frame Number=0 Number of Objects=6 Vehicle_count=3 Person_count=3
Frame Number=1 Number of Objects=7 Vehicle_count=2 Person_count=5
Frame Number=2 Number of Objects=5 Vehicle_count=3 Person_count=2
Frame Number=3 Number of Objects=6 Vehicle_count=4 Person_count=2
Frame Number=4 Number of Objects=6 Vehicle_count=3 Person_count=3
Frame Number=5 Number of Objects=7 Vehicle_count=5 Person_count=2
Frame Number=6 Number of Objects=7 Vehicle_count=5 Person_count=2
Frame Number=7 Number of Objects=6 Vehicle_count=4 Person_count=2
Frame Number=8 Number of Objects=8 Vehicle_count=6 Person_count=2
Frame Number=9 Number of Objects=7 Vehicle_count=5 Person_count=2
Frame Number=10 Number of Objects=6 Vehicle_count=4 Person_count=2
Frame Number=11 Number of Objects=8 Vehicle_count=6 Person_count=2
Frame Number=12 Number of Objects=7 Vehicle_count=5 Person_count=2
Frame Number=13 Number of Objects=3 Vehicle_count=1 Person_count=2
Frame Number=14 Number of Objects=5 Vehicle_count=3 [...]<jupyter_text>This video output is not compatible to be shown in this Notebook. To circumvent this, we convert the output in a Jupyter Notebook-readable format. For this we use the shell command `ffmpeg`.<jupyter_code># Convert video profile to be compatible with the Notebook
!ffmpeg -loglevel panic -y -an -i ../videos/out.mp4 -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -level 3 ../videos/output.mp4<jupyter_output><empty_output><jupyter_text>Finally, we display the output in the notbook by creating an HTML video element.<jupyter_code># Display the Output
from IPython.display import HTML
HTML("""
 <video width="640" height="480" controls>
 <source src="../videos/output.mp4"
 </video>
""".format())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08253235934896834, 'fraction_numerical': 0.018710752274766117, 'mean_word_length': 4.845318352059925, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 53, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '11623', 'license': 'CC0-1.0', 'n_tokens_mistral': 5073, 'n_tokens_neox': 4812, 'n_words': 1492, 'path': 'notebooks/TAO_Integration_pgie.ipynb', 'repo_name': 'MesaTech/deepstream_tao_integration', 'size': '15606'}