<jupyter_start><jupyter_text>OpenCV Overlay: Filter2D and Dilate This notebook illustrates the kinds of things you can do with accelerated openCV cores built as PYNQ overlay. The overlay consists of a 2D filter and a dilate function and this example notebook does the following.1. Sets up HDMI drivers2. Run software only filter2D + dilate pipeline on HDMI input and output results to HDMI output3. Run hardware accelerated dilate function4. Sets up widget for controlling different filter kernels5. Run hardware accelerated filter2D + dilate functionNOTE: Rough FPS values are computed for each stage  Program overlayHere we program the overlay and load the pynq python libraries for a memory manager and the accelerator drivers.NOTE: All overlay and python libraries should be loaded prior to assigning the HDMI input/outputs. This is necessary right now to ensure correct functionality but will be enhanced in future releases. For now, please copy this block as is when using it in your own designs.<jupyter_code># Load filter2D + dilate overlay
from pynq import Overlay
bareHDMI = Overlay("/usr/local/lib/python3.6/dist-packages/"
               "pynq_cv/overlays/xv2Filter2DDilate.bit")
import pynq_cv.overlays.xv2Filter2DDilate as xv2

# Load xlnk memory mangager
from pynq import Xlnk
Xlnk.set_allocator_library("/usr/local/lib/python3.6/dist-packages/"
                           "pynq_cv/overlays/xv2Filter2DDilate.so")
mem_manager = Xlnk()

hdmi_in = bareHDMI.video.hdmi_in
hdmi_out = bareHDMI.video.hdmi_out<jupyter_output><empty_output><jupyter_text>Setup and configure HDMI drivers ~15 seconds to initialize HDMI input/output<jupyter_code>from pynq.lib.video import *
hdmi_in.configure(PIXEL_GRAY)
hdmi_out.configure(hdmi_in.mode)

hdmi_in.cacheable_frames = False
hdmi_out.cacheable_frames = False

hdmi_in.start()
hdmi_out.start()<jupyter_output><empty_output><jupyter_text>Setup up HDMI input/output parameters These parameters are referenced in later function calls<jupyter_code>mymode = hdmi_in.mode
print("My mode: "+str(mymode))

height = hdmi_in.mode.height
width = hdmi_in.mode.width
bpp = hdmi_in.mode.bits_per_pixel<jupyter_output>My mode: VideoMode: width=1920 height=1080 bpp=8<jupyter_text>Run SW Filter2D + Dilate ~10 seconds<jupyter_code>import numpy as np
import time
import cv2

#laplacian filter, high-pass  
kernelF = np.array([[0.0, 1.0, 0],[1.0, -4, 1.0],[0, 1.0, 0.0]],np.float32)
kernelD = np.ones((3,3),np.uint8)
buf     = np.ones((height,width),np.uint8)

numframes = 20

start = time.time()
for _ in range(numframes):
    inframe = hdmi_in.readframe()
    outframe = hdmi_out.newframe()
    cv2.filter2D(inframe, -1, kernelF, dst=buf)
    cv2.dilate(buf, kernelD, dst=outframe, iterations=1)
    inframe.freebuffer()
    hdmi_out.writeframe(outframe)
end = time.time()
print("Frames per second:  " + str(numframes / (end - start)))<jupyter_output>Frames per second:  2.422169479400087<jupyter_text>Show input frame in notebook.<jupyter_code>import PIL.Image

image = PIL.Image.fromarray(inframe)
image<jupyter_output><empty_output><jupyter_text>Show output frame after filter2D in notebook.<jupyter_code>import PIL.Image

image = PIL.Image.fromarray(outframe)
image<jupyter_output><empty_output><jupyter_text>Run HW dilate only ~3 secondsBased on a kernel frequency of 100 MHz, this block should run at ~40 fps. This is also true for the filter2D block as well.<jupyter_code>import numpy as np

kernelVoid = np.zeros(0)

numframes = 200

start=time.time()
for _ in range(numframes):
    inframe = hdmi_in.readframe()
    outframe = hdmi_out.newframe()
    xv2.dilate(inframe, kernelVoid, dst=outframe, iterations=1, borderType=cv2.BORDER_CONSTANT)
    inframe.freebuffer()
    hdmi_out.writeframe(outframe)
end=time.time()

print("Frames per second:  " + str(numframes / (end - start)))<jupyter_output>Frames per second:  59.86234941121783<jupyter_text>Show output dilated frame in notebook.<jupyter_code>import PIL.Image

image = PIL.Image.fromarray(outframe)
image<jupyter_output><empty_output><jupyter_text>Setup control widgetsHere, we define some kernel configurations that will be used to change the functionality of the 2D filter on the fly. A pulldown menu will appear below this cell to be used to change the filter2D kernel used subsequent cells.<jupyter_code>from ipywidgets import interact, interactive, fixed, interact_manual, IntSlider, FloatSlider
import ipywidgets as widgets

kernel_g = np.array([[0.0, 1.0, 0],[1.0, -4, 1.0],[0, 1.0, 0.0]],np.float32) #laplacian filter, high-pass  

def setKernelAndFilter3x3(kernelName):
    global kernel_g

    kernel_g = {
        'laplacian high-pass':  np.array([[0.0, 1.0, 0],[1.0, -4, 1.0],[0, 1.0, 0.0]],np.float32),
        'gaussian high-pass': np.array([[-0.0625,-0.125,-0.0625],[-0.125,0.75,-0.125],[-0.0625,-0.125,-0.0625]],np.float32),
        'average blur':  np.ones((3,3),np.float32)/9.0,
        'gaussian blur': np.array([[0.0625,0.125,0.0625],[0.125,0.25,0.125],[0.0625,0.125,0.0625]],np.float32),
        'Sobel Ver': np.array([[1.0,0.0,-1.0],[2.0,0.0,-2.0],[1.0,0.0,-1.0]],np.float32),
        'Sobel Hor': np.array([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]],np.float32)
    }.get(kernelName, np.ones((3,3),np.float32)/9.0)

interact(setKernelAndFilter3x3, kernelName=['laplacian high-pass','gaussian high-pass','average blur','gaussian blur','Sobel Hor','Sobel Ver']);<jupyter_output><empty_output><jupyter_text>Run HW filter2D + dilate ~20 secondsRunning both blocks in series means the effective performance is approximately halved or ~20 fps.NOTE: In order to allow kernel redefinition on the fly, subsequent function call are run as threads. This means you will not know if the cell is finished based on the cell status on the left. Be sure to wait until FPS information is reported before running other cells. Also note that if you use the widget to change the kernel, the FPS info will show up underneath the widget cell rather than the function block cell.<jupyter_code>import numpy as np

def loop_hw2_app():
    global kernel_g
    kernelVoid = np.zeros(0)
    
    xFbuf = mem_manager.cma_array((height,width),np.uint8)
    
    numframes = 600

    start=time.time()
    for _ in range(numframes):
        inframe = hdmi_in.readframe()
        outframe = hdmi_out.newframe()
        xv2.filter2D(inframe, -1, kernel_g, dst=xFbuf,
                     borderType=cv2.BORDER_CONSTANT)
        xv2.dilate(xFbuf, kernelVoid, dst=outframe, 
                   borderType=cv2.BORDER_CONSTANT)
        inframe.freebuffer()
        hdmi_out.writeframe(outframe)
    end=time.time()
    print("Frames per second:  " + str(numframes / (end - start)))

from threading import Thread

t = Thread(target=loop_hw2_app, )
t.start()
import PIL.Image

image = PIL.Image.fromarray(outframe)
image<jupyter_output><empty_output><jupyter_text>Clean up hdmi driversNOTE: This is needed to reset the HDMI drivers in a clean state. If this is not run, subsequent executions of this notebook may show visual artifacts on the HDMI out (usually a shifted output image)<jupyter_code>hdmi_out.close()
hdmi_in.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10927661956065482, 'fraction_numerical': 0.042395410661816145, 'mean_word_length': 4.623918174665618, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '561282', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2631, 'n_tokens_neox': 2458, 'n_words': 665, 'path': 'boards/Pynq-Z2/notebooks/filter2d_and_dilate.ipynb', 'repo_name': 'yutiansut/PYNQ-ComputerVision', 'size': '7147'}