<jupyter_start><jupyter_text>![Xilinx Logo](images/xilinx_logo.png "Xilinx Logo")  1. Introduction:This notebook demonstrates how to use up to 4 channels **1080p H264/H265** of RTSP video streams or local video file source, and perform pedestrian detection with Refinedet models and ReID and tracking on each channel, finally display the images with pedestrain bounding boxes and ReID labels on the 4K monitors with a 2x2 layout.**Note** For security reasons, the Jupyter Lab Server should not be started as root user, but because of Linux kernel constraints, this restriction causes this Jupyter notebook can only display one channel video, this restriction may be removed later with Linux kernel update.**Note** If you want to run with multiple channels, please switch to the command line aibox-reid application.The application is based on the VVAS (Vitis Video Analytics SDK) framework, also utilizing the open source GStreamer plugins.Vitis Video Analytics SDK (VVAS) is developed by Xilinx to provide many useful GStreamer plugins as the middleware between the application and underlying FPGA acclerators, including DPU AI inference engine, and other PL accelerators such as the one for AI input preprocessing.Please refer to the [Kria™ KV260 Vision AI Starter Kit Applications GitHub Pages](https://xilinx.github.io/kria-apps-docs/index.html) for detailed HW/SW architecture and [Vitis Video Analytics SDK GitHub Pages](https://xilinx.github.io/VVAS/) for the VVAS related info.  2. Imports and Initialization  Preapare Data to Visualize the Pipeline Create a directory for saving the pipeline graph as dot file.Set the GStreamer debug dot directory environment variable to point to that directory<jupyter_code>nb="aibox-reid"
dotdir = "/tmp/gst-dot/" + nb + "/"
!mkdir -p $dotdir
%env GST_DEBUG_DUMP_DOT_DIR = $dotdir<jupyter_output><empty_output><jupyter_text>Import all python modules required for this notebook. * Import system, util modules<jupyter_code>import sys
import glob
import subprocess
import re
import os<jupyter_output><empty_output><jupyter_text>* Add some util path<jupyter_code>pathv="{}:/usr/sbin:/sbin".format(os.getenv("PATH"))
%env PATH = $pathv<jupyter_output><empty_output><jupyter_text>* GStreamer related library import.<jupyter_code>import gi
gi.require_version('Gst', '1.0')
gi.require_version("GstApp", "1.0")
gi.require_version('GstVideo', '1.0')
gi.require_version('GIRepository', '2.0')
from gi.repository import GObject, GLib, Gst, GstVideo, GLib, GIRepository
Gst.init(None)
Gst.debug_set_threshold_from_string('*:1', True)

confdir="/opt/xilinx/share/ivas/aibox-reid"
pip=""<jupyter_output><empty_output><jupyter_text>Mixer setting<jupyter_code>! echo | modetest -M xlnx -D 80000000.v_mix -s 52@40:3840x2160@NV16<jupyter_output><empty_output><jupyter_text>3. Construct the Pipeline and Run Application ***Notice*** After run the application, please "Shut Down" or "Restart" the kernel of this notebook, otherwise the process will keep occupying the DPU device.  User Options:  1. Option to set the RTSP stream URLs or video file location * RTSP input  You can normally get or configure in the Admin page of your IP camera.  Such as: rtsp://ip-address:port/name     When you get it, you may first verify if it's playable  * File input       To demonstrate the application in the case where no IP camera is available, a video source may be played from a file on the SD card instead.      You can download video files from the following links, which is of MP4 format.    Demo video:    * https://pixabay.com/videos/liverpool-people-couple-pier-head-46090/    * https://pixabay.com/videos/liverpool-pier-head-england-uk-46098/    * https://pixabay.com/videos/spring-walk-park-trees-flowers-15252/    * https://pixabay.com/videos/walking-people-city-bucharest-6099/    Then you need to transcode it to H264 file which is the supported input format.    > ffmpeg -i input-video.mp4 -c:v libx264 -pix_fmt nv12 -r 30 output.nv12.h264    Finally, please upload or copy these transcoded H264 files to the board, place it to somewhere under /home/petalinux, which is the home directory of the user you login as. **Note** For security reasons, Jupyter Lab server should not be started as root user, but because of Linux kernel constraints, this restriction causes this Jupyter notebook can only display one channel video.So though the below variable *aitask* is to configure multiple input channels, only one channel could be displayed. Please  use the command line aibox-reid application to run multiple channels. The var ***aitask*** is an array of tuple, of which each corresponds to one channel.In each tuple in the array:* The 1st value: The position of the channel    0: TopLeft, 1: TopRight, 2: Bottom Left, 3: Bottom Right    * The 2nd value: The string of either the RTSP URL, or path to local video file* The 3rd value: The encoding type of the video file, only support h264/h265.* The 4th value: "rtsp" or "file" indicating the source type.<jupyter_code>file="/home/petalinux/walking-people.nv12.30fps.1080p.h264"
aitask=[ \
#        (0, url, "h264", "rtsp"), \
        (1,file, "h264", "file"), \
#        (2, url, "h264", "rtsp"), \
#        (3,file, "h264", "file") \
       ]<jupyter_output><empty_output><jupyter_text>Construct Pipeline for the First Channel<jupyter_code>i=0
pos=aitask[0][0]
url=aitask[0][1]
fmt=aitask[0][2]
src=aitask[0][3]
if src == "file":
    if not os.path.isfile(url):
        print("The video file {} doesn't exist.".format(url))
        assert(False)
    pip += "multifilesrc location=\"{}\" ".format(url)
    que = ""
elif src == "rtsp":
    pip += "rtspsrc location=\"{}\" ! queue ! rtp{}depay ! queue ".format(url, fmt)
    que = " ! queue max-size-buffers=2 leaky=2 "
pip += "! {fmt}parse ! queue ! omx{fmt}dec ! video/x-raw, format=NV12 {que} ".format(fmt=fmt, que=que)<jupyter_output><empty_output><jupyter_text>* Do the color conversion and preprocess to the pedestrian detection.<jupyter_code>pip += " ! tee name=t{ind} t{ind}.src_0 ! queue \
! ivas_xmultisrc kconfig=\"{conf}/ped_pp.json\" ".format(ind=i, conf=confdir)<jupyter_output><empty_output><jupyter_text>* Perform pedestrain detection with refinedet models.<jupyter_code>pip += " ! queue ! ivas_xfilter name=refinedet_{ind} kernels-config=\"{conf}/refinedet.json\" ".format(ind=i, conf=confdir)<jupyter_output><empty_output><jupyter_text>* Crop the detected pedestrain into multiple buffer, and pass it down with inference meta.<jupyter_code>pip += " ! queue ! ivas_xfilter name=crop_{ind} kernels-config=\"{conf}/crop.json\" ".format(ind=i, conf=confdir)<jupyter_output><empty_output><jupyter_text>* Perform ReID on the previously cropped object to get character, and track it with dedicated ReID Tracker algorithm.Here ReID is done on DPU, and tracker on ARM, both of them is done in this plugin.<jupyter_code>pip += " ! queue ! ivas_xfilter name=reid_{ind} kernels-config=\"{conf}/reid.json\" ".format(ind=i, conf=confdir)<jupyter_output><empty_output><jupyter_text>* Pass inference meta of the scaled image to ivas_xmetaaffixer, to be scaled back to the original input buffer.<jupyter_code>pip += " ! queue ! scalem{ind}.sink_master ivas_xmetaaffixer name=scalem{ind} scalem{ind}.src_master ! fakesink ".format(ind=i)<jupyter_output><empty_output><jupyter_text>* Connect the original input buffer to ivas_xmetaaffixer to get the inference meta by scaling the meta from previous one, then draw the bbox and identity on it.<jupyter_code>pip += " t{ind}.src_1 ! queue ! scalem{ind}.sink_slave_0 scalem{ind}.src_slave_0 \
! queue ! ivas_xfilter kernels-config=\"{conf}/draw_reid.json\" ".format(ind=i, conf=confdir)<jupyter_output><empty_output><jupyter_text>* Pass to mixer to be rendered on 1/4 of 4K screen.<jupyter_code>pip += " ! kmssink bus-id=b0000000.v_mix plane-id={pid} render-rectangle=\"<{w},{h},1920,1080>\" show-preroll-frame=false sync=false can-scale=false  ". \
format(pid=34+i, w=pos%2*1920, h=pos//2*1080)<jupyter_output><empty_output><jupyter_text>Wrap above one channel pipeline creation to one function.<jupyter_code>def One_channel_pipe(i):
    global pip
    pos=aitask[i][0]
    url=aitask[i][1]
    fmt=aitask[i][2]
    src=aitask[i][3]

    if src == "file":
        if not os.path.isfile(url):
            print("The video file {} doesn't exist.", url)
            assert(False)
        pip += "multifilesrc location=\"{}\" ".format(url)
        que = ""
    elif src == "rtsp":
        pip += "rtspsrc location=\"{}\" ! queue ! rtp{}depay ! queue ".format(url, fmt)    
        que = " ! queue max-size-buffers=2 leaky=2 "        
    pip += "! {fmt}parse ! queue ! omx{fmt}dec ! video/x-raw, format=NV12 {que} ".format(fmt=fmt, que=que)
    pip += " ! tee name=t{ind} t{ind}.src_0 ! queue ! ivas_xmultisrc kconfig=\"{conf}/ped_pp.json\" ".format(ind=i, conf=confdir)
    pip += " ! queue ! ivas_xfilter name=refinedet_{ind} kernels-config=\"{conf}/refinedet.json\" ".format(ind=i, conf=confdir)
    pip += " ! queue ! ivas_xfilter name=crop_{ind} kernels-config=\"{conf}/crop.json\" ".format(ind=i, conf=confdir)
    pip += " ! queue ! ivas_xfilter name=reid_{ind} kernels-config=\"{conf}/reid.json\" ".format(ind=i, conf=confdir)
    pip += " ! queue ! scalem{ind}.sink_master ivas_xmetaaffixer name=scalem{ind} scalem{ind}.src_master ! fakesink ".format(ind=i)
    pip += " t{ind}.src_1 ! queue ! scalem{ind}.sink_slave_0 scalem{ind}.src_slave_0 \
    ! queue ! ivas_xfilter kernels-config=\"{conf}/draw_reid.json\" ".format(ind=i, conf=confdir)
    pip += " ! kmssink bus-id=b0000000.v_mix plane-id={pid} render-rectangle=\"<{w},{h},1920,1080>\" show-preroll-frame=false sync=false ". \
    format(pid=34+i, w=pos%2*1920, h=pos//2*1080)
for i in range(1, min(4,len(aitask))):
    One_channel_pipe(i)<jupyter_output><empty_output><jupyter_text>3. Run the Application<jupyter_code>pipe = Gst.parse_launch(pip)
pipe.set_state(Gst.State.PLAYING)<jupyter_output><empty_output><jupyter_text>4. View the GStreamer Pipeline Graph * Dump dot file of pipeline<jupyter_code>Gst.debug_bin_to_dot_file(pipe, Gst.DebugGraphDetails.ALL, nb)<jupyter_output><empty_output><jupyter_text>* Convert the dot file to png and display the pipeline graph.    The image will be displayed bellow the following code cell.    **Note**: This step may take a few seconds.<jupyter_code>import pydot
from IPython.display import Image, display, clear_output
dotfile = dotdir + "/" + nb + ".dot"

print("Converting dot to graph...")
graph = pydot.graph_from_dot_file(dotfile, 'utf-8')
print("Conversion done.")
display(Image(graph[0].create(None,'png', 'utf-8')))
print("Pipeline graph is shown, double click it to zoom in and out.")<jupyter_output><empty_output><jupyter_text>* After the pipeline dot graph is shown in the previous step, the process can be interrupted by clicking the stop square button on the Jupyter toolbar.<jupyter_code>loop = GLib.MainLoop()
try:
    loop.run()
except:
    sys.stdout.write("Interrupt caught\n")
    pipe.set_state(Gst.State.NULL)
    loop.quit()
    pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1219379915032089, 'fraction_numerical': 0.020338063816324685, 'mean_word_length': 4.932439678284182, 'pattern_counts': {'":': 4, '<': 79, '<?xml version=': 0, '>': 80, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '389477', 'license': 'Apache-2.0', 'n_tokens_mistral': 3841, 'n_tokens_neox': 3620, 'n_words': 1098, 'path': 'notebook/aibox-reid.ipynb', 'repo_name': 'balancezhai/Pose-detection', 'size': '11063'}