<jupyter_start><jupyter_text>OpenCV Overlay: Filter2D and Dilate This notebook takes the design from notebook [1\_\_Intro_to_OpenCV_on_Jupyter_notebooks.ipynb](http://192.168.3.1:9090/notebooks/computer_vision/1__Intro_to_OpenCV_on_Jupyter_notebooks.ipynb) and shows the power of FPGA acceleration through the use of PYNQ overlays. If you are unfamiliar with the OpenCV function filter2D and dilate, please go through [1\_\_Intro_to_OpenCV_on_Jupyter_notebooks.ipynb](http://192.168.3.1:9090/notebooks/computer_vision/1__Intro_to_OpenCV_on_Jupyter_notebooks.ipynb) first. This example consists of a 2D filter and a dilate function and does the following.1. Program overlay2. Sets up USB camera3. Run software only filter2D 4. Run SW filter2D + dilate and measure performance5. Run SW filter2D + dilate and measure performance6. Plot performance7. Setup widgets8. Run HW filter2D + dilate on input frames from USB camera in real-time  Program overlayHere we program the overlay on the FPGA, load the associated overlay library and load the PYNQ xlnk memory manager library. This process takes a few seconds to do. Overlays are generally composed of an FPGA bitstream and a shared library to access the accelerators via function calls. We use the pynq.Bitstream class to load the overlay and rename the associated shared library to xv2. It is also important to set the xlnk allocator library to point the same shared library so allocated memory maps are consistent. The pynq.Xlnk class contains hooks to the SDx memory allocation functions which is important when allocating certain kinds of memory for our accelerators (e.g. continguous non-cacheable buffers)<jupyter_code>import cv2 #NOTE: This needs to be loaded first

# Load filter2D + dilate overlay
from pynq import Bitstream
bs = Bitstream("/usr/local/lib/python3.6/dist-packages/pynq_cv/overlays/xv2Filter2DDilate.bit")
bs.download()
import pynq_cv.overlays.xv2Filter2DDilate as xv2

# Load xlnk memory mangager
from pynq import Xlnk
Xlnk.set_allocator_library('/usr/local/lib/python3.6/dist-packages/pynq_cv/overlays/xv2Filter2DDilate.so')
mem_manager = Xlnk()<jupyter_output><empty_output><jupyter_text>Setup and configure USB camera We use OpenCV (cv2) for capturing frames from a USB camera and processing those image frames. Here, we start by setting up the interface to the USB camera and configuring its resolution (1080p). A successful camera setup returns a 'True'. If something is is outputted, shutdown the notebook and restart it again.<jupyter_code>import cv2

camera = cv2.VideoCapture(0)

width = 1920
height = 1080
camera.set(cv2.CAP_PROP_FRAME_WIDTH,width)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT,height)<jupyter_output><empty_output><jupyter_text>We add another helper function which sets up an IPython-based imshow call which encodes OpenCV image data to jpeg format before displaying it within the notebook itself. Other methods of displaying image data would perform similar conversions as well.<jupyter_code>import IPython

def imshow(img):
    returnValue, buffer = cv2.imencode('.jpg', img)
    IPython.display.display(IPython.display.Image(data=buffer.tobytes()))<jupyter_output><empty_output><jupyter_text>Read input frame from USB camera Read input frame and convert image to gray scale with OpenCV function cvtColor.**NOTE**: We do a few extra reads up front to flush out the frame buffers in case camera was previously used<jupyter_code># Flush webcam buffers (needed when rerunning notebook)
for _ in range(5):
    ret, frame_in = camera.read()

# Read in a frame
ret, frame_in = camera.read()
if ret:
    frame_in_gray = cv2.cvtColor(frame_in,cv2.COLOR_RGB2GRAY)
else:
    print("Error reading frame from camera.")<jupyter_output><empty_output><jupyter_text>Show input frame in notebook.<jupyter_code>imshow(frame_in_gray)<jupyter_output><empty_output><jupyter_text>Run SW Filter2D Here, we call the OpenCV 2D filter function on the input frame using kernel coefficients for a Laplacian high-pass filter (which gives a kind edge detection). Note that many of the vision processing functions used will operate on gray scale only images. The matrix returned by filter2D is an 2D array (1920x1080) of 8-bit values denoting the brightness of each pixel.* [OpenCV Filter2D](https://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.htmlga27c049795ce870216ddfb366086b5a04)<jupyter_code>import numpy as np

#Sobel Hor filter  
kernelF    = np.array([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]],np.float32)
frameF = np.ones((height,width),np.uint8)

cv2.filter2D(frame_in_gray, -1, kernelF, frameF, borderType=cv2.BORDER_CONSTANT) # software 2D filter
imshow(frameF)<jupyter_output><empty_output><jupyter_text>Run SW Filter2D + SW Dilate and measure performanceWe first run our vision processing pipeline in software only mode. By doing a 2D filter followed by a dilate function, we effectively brighten all the white edge lines of our image. In addition, we will iterate our small image processing pipeline (filter2d, dilate) over a few frames and measure the performance of each function as well as the overall performance. We also enable profiling with the %%prun command.<jupyter_code>%%prun -s cumulative -q -l 10 -T prunSW
import numpy as np
import time

kernelF  = np.array([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]],np.float32) #Sobel Hor filter
kernelD  = np.ones((3,3),np.uint8) 
frameF   = np.ones((height,width),np.uint8)
frameD   = np.ones((height,width),np.uint8)

num_frames    = 20

start = time.time()
for _ in range(num_frames):
    cv2.filter2D(frame_in_gray, -1, kernelF, frameF)
    cv2.dilate(frameF, kernelD, frameD, iterations=1)
time_sw_total = time.time() - start
print("Frames per second:  " + str(num_frames / time_sw_total))
imshow(frameD)<jupyter_output><empty_output><jupyter_text>Process SW profile results<jupyter_code>print(open('prunSW','r').read())
res = !cat prunSW | grep filter2D | awk '{{print $$2}}'
tottime_sw_filter2d = float(res[0])
res = !cat prunSW | grep dilate   | awk '{{print $$2}}'
tottime_sw_dilate   = float(res[0])<jupyter_output><empty_output><jupyter_text>Run HW Filter2D + HW Dilate and measure performanceNow we take advantage of the library of accelerators in our overlay and accelerate the same two OpenCV functions in hardware. From a function signature point of view, it's about as simple as replacing the OpenCV library call (cv2) with a python-extended Xilinx OpenCV function call (xv2) provided to us by the overlay. The other concept necessary for hardware acceleration is making sure we use continguous memory for our frames. This is done through the use of cma_array calls as opposed to the numpy calls. Some data copying will be necessary when moving data between numpy arrays and cma_arrays if the data.* [xFOpenCV Filter2D](https://github.com/Xilinx/xfopencv/blob/master/include/imgproc/xf_custom_convolution.hpp) ([ug1233](https://www.xilinx.com/support/documentation/sw_manuals/xilinx2018_2/ug1233-xilinx-opencv-user-guide.pdf))<jupyter_code>%%prun -s cumulative -q -l 10 -T prunHW
import numpy as np
import time

#laplacian filter, high-pass  
kernelF  = np.array([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]],np.float32) #Sobel Hor filter
kernelVoid = np.zeros(0)
xFin       = mem_manager.cma_array((height,width),np.uint8)
xFbuf      = mem_manager.cma_array((height,width),np.uint8)
xFout      = mem_manager.cma_array((height,width),np.uint8)

num_frames       = 20

xFin[:]    = frame_in_gray[:]

start = time.time()
for _ in range(num_frames):
    xv2.filter2D(xFin, -1, kernelF, xFbuf, borderType=cv2.BORDER_CONSTANT)
    xv2.dilate(xFbuf, kernelVoid, xFout, borderType=cv2.BORDER_CONSTANT)    
time_hw_total = time.time() - start
print("Frames per second:  " + str(num_frames / time_hw_total))

frame_out = np.ones((height,width),np.uint8)
frame_out[:] = xFout[:]
imshow(frame_out)<jupyter_output><empty_output><jupyter_text>Process HW profile results<jupyter_code>print(open('prunHW','r').read())
res = !cat prunHW | grep filter2D | awk '{{print $$2}}'
tottime_hw_filter2d = float(res[0])
res = !cat prunHW | grep dilate   | awk '{{print $$2}}'
tottime_hw_dilate   = float(res[0])<jupyter_output><empty_output><jupyter_text>Plot performanceIn addition to having easy access to OpenCV functions, we can access functions from pyPlot for plotting results in graphs and charts. Here, we take the recorded time data and plot out the processing times in a bar chart along with computed FPS of each function. Pay particular attention to the actual performance of each function and note the effect when placing two functions back-to-back in this example.<jupyter_code>%matplotlib inline
from matplotlib import pyplot as plt

TIME_SW   = [t*1000/num_frames for (t) in (time_sw_total, tottime_sw_dilate, tottime_sw_filter2d)]
FPS_SW    = [1000/t for (t) in (TIME_SW)]
TIME_HW   = [t*1000/num_frames for (t) in (time_hw_total, tottime_hw_dilate, tottime_hw_filter2d)]
FPS_HW    = [1000/t for (t) in (TIME_HW)]
LABELS    = ['Total','Dilate','Filter2D']

f, ((ax1, ax2),(ax3, ax4)) = plt.subplots(2, 2, sharex='col', sharey='row', figsize=(15,4))
x_pos = np.arange(len(LABELS))
plt.yticks(x_pos, LABELS)

ax1.barh(x_pos, TIME_SW, height=0.6, color='g', zorder=3)
ax1.invert_yaxis()
ax1.set_xlabel("Execution Time per frame [ms]")
ax1.set_ylabel("Kernel (SW)")
ax1.grid(zorder=0)

ax2.barh(x_pos, FPS_SW, height=0.6, color='b', zorder=3)
ax2.invert_yaxis()
ax2.set_xlabel("Frames per second")
ax2.grid(zorder=0)

ax3.barh(x_pos, TIME_HW, height=0.6, color='g', zorder=3)
ax3.invert_yaxis()
ax3.set_xlabel("Execution Time per frame [ms]")
ax3.set_ylabel("Kernel (HW)")
ax3.grid(zorder=0)

ax4.barh(x_pos, FPS_HW, height=0.6, color='b', zorder=3)
ax4.invert_yaxis()
ax4.set_xlabel("Frames per second")
ax4.grid(zorder=0)

plt.show()<jupyter_output><empty_output><jupyter_text>Setup control widgetsHere, we define some kernel configurations that will be used to change the functionality of the 2D filter on the fly. A pulldown menu will appear below this cell to be used to change the filter2D kernel used subsequent cells.<jupyter_code>from ipywidgets import interact, interactive, fixed, interact_manual, IntSlider, FloatSlider
import ipywidgets as widgets

kernel_g = np.array([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]],np.float32) #Sobel Hor  

def setKernelAndFilter3x3(kernelName):
    global kernel_g

    kernel_g = {
        'laplacian high-pass':  np.array([[0.0, 1.0, 0],[1.0, -4, 1.0],[0, 1.0, 0.0]],np.float32),
        'gaussian high-pass': np.array([[-0.0625,-0.125,-0.0625],[-0.125,0.75,-0.125],[-0.0625,-0.125,-0.0625]],np.float32),
        'gaussian blur': np.array([[0.0625,0.125,0.0625],[0.125,0.25,0.125],[0.0625,0.125,0.0625]],np.float32),
        'Sobel Ver': np.array([[1.0,0.0,-1.0],[2.0,0.0,-2.0],[1.0,0.0,-1.0]],np.float32),
        'Sobel Hor': np.array([[1.0,2.0,1.0],[0.0,0.0,0.0],[-1.0,-2.0,-1.0]],np.float32)
    }.get(kernelName, np.ones((3,3),np.float32)/9.0)

interact(setKernelAndFilter3x3, kernelName=['Sobel Hor','Sobel Ver','laplacian high-pass','gaussian high-pass','gaussian blur']);<jupyter_output><empty_output><jupyter_text>Run HW filter2D + dilate on input frames from USB camera in real-timeNow we will run the 2D filter + dilate on input frames from the USB camera and display the results in real time. We will also respond to feedback from our pulldown menu which changes the 2D filter coefficients. While this interaction does not capture the performance acceleration of the HW implementation, it does allow us to receive real-time feedback of the accelerated hardware functions. NOTE: To allow for live interaction of the pull-down menu selection with the processing loop, the loop will be run as a separate thread for ~20 seconds. As soon as the kernel coefficient is changed, the image will instead appear under the widget cell along with the eventual reported FPS calculations. The normal feedback of when a cell is complete will therefore not function (* will disappear immediately). The indication when the loop is complete will therefore be when the FPS calculation is reported. Adjusting the loop time requires changing the num_frames variable.<jupyter_code>def loop_hw2_app():
    global kernel_g

    kernelD   = np.ones((3,3),np.uint8)
    frame_out = np.ones((height,width),np.uint8)
    xFin      = mem_manager.cma_array((height,width),np.uint8)
    xFbuf     = mem_manager.cma_array((height,width),np.uint8)
    xFout     = mem_manager.cma_array((height,width),np.uint8)

    num_frames = 60

    start = time.time()
    for _ in range(num_frames):
        # Capture frame-by-frame
        ret, frame_in = camera.read()
        if (not ret):
            # Release the Video Device if ret is false
            camera.release()
            # Message to be displayed after releasing the device
            print("Released Video Resource")
            break
        frame_in_gray = cv2.cvtColor(frame_in,cv2.COLOR_RGB2GRAY)
        xFin[:]       = frame_in_gray[:]
        xv2.filter2D(xFin, -1, kernel_g, xFbuf, borderType=cv2.BORDER_CONSTANT)
        xv2.dilate(xFbuf, kernelVoid, xFout, borderType=cv2.BORDER_CONSTANT)
        frame_out[:]  = xFout[:]
        imshow(frame_out)
        IPython.display.clear_output(wait=True)    

    end = time.time()
    print("Frames per second:  " + str(num_frames / (end - start)))  
from threading import Thread

t = Thread(target=loop_hw2_app, )
t.start()<jupyter_output><empty_output><jupyter_text>Release USB camera resource**NOTE**: This is needed to close the camera between subsequent runs. If the camera is unable to read a frame, be sure to call camera.release() and then try opening the VideoCapture again.<jupyter_code>camera.release()<jupyter_output><empty_output><jupyter_text>IMPORTANT NOTE: Be sure to run the cell below, shutting down the notebook, before starting a new one. The notebook interface shows "No Kernel", the cell below will incorrectly show a running status [ * ]. You can ignore this an safely close the tab of the notebook.<jupyter_code>%%javascript
Jupyter.notebook.session.delete();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10616268349762428, 'fraction_numerical': 0.0360258137720729, 'mean_word_length': 4.965313028764806, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '39565', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4961, 'n_tokens_neox': 4713, 'n_words': 1439, 'path': 'boards/Ultra96/notebooks/2__Overlays-filter2d_and_dilate.ipynb', 'repo_name': 'budaidai/PYNQ', 'size': '14101'}