<jupyter_start><jupyter_text>Video Super Resolution with OpenVINOSuper Resolution is the process of enhancing the quality of an image by increasing the pixel count using deep learning. This notebook applies Single Image Super Resolution (SISR) to frames in a 360p (480×360) video in 360p resolution. We use a model called [single-image-super-resolution-1032](https://github.com/openvinotoolkit/open_model_zoo/tree/develop/models/intel/single-image-super-resolution-1032) which is available from the Open Model Zoo. It is based on the research paper cited below. <NAME> et al., ["An Attention-Based Approach for Single Image Super Resolution,"](https://arxiv.org/abs/1807.06779) 2018 24th International Conference on Pattern Recognition (ICPR), 2018, pp. 2777-2784, doi: 10.1109/ICPR.2018.8545760.**NOTE:** The Single Image Super Resolution (SISR) model used in this demo is not optimized for video. Results may vary depending on the video. We are looking for a more suitable Multi Image Super Resolution (MISR) model, so if you know of a great open source model, please let us know! You can start a [discussion](https://github.com/openvinotoolkit/openvino_notebooks/discussions) or create an [issue](https://github.com/openvinotoolkit/openvino_notebooks/issues) on GitHub.   Preparation  Imports<jupyter_code>import os
import time
import urllib
from pathlib import Path

import cv2
import numpy as np
from IPython.display import (
    HTML,
    FileLink,
    Pretty,
    ProgressBar,
    Video,
    clear_output,
    display,
)
from openvino.inference_engine import IECore
from pytube import YouTube<jupyter_output><empty_output><jupyter_text>Settings<jupyter_code># Device to use for inference. For example, "CPU", or "GPU"
DEVICE = "CPU"
# 1032: 4x superresolution, 1033: 3x superresolution
MODEL_FILE = "model/single-image-super-resolution-1032.xml"
model_name = os.path.basename(MODEL_FILE)
model_xml_path = Path(MODEL_FILE).with_suffix(".xml")<jupyter_output><empty_output><jupyter_text>Functions<jupyter_code>def write_text_on_image(image: np.ndarray, text: str) -> np.ndarray:
    """
    Write the specified text in the top left corner of the image
    as white text with a black border.

    :param image: image as numpy arry with HWC shape, RGB or BGR
    :param text: text to write
    :return: image with written text, as numpy array
    """
    font = cv2.FONT_HERSHEY_PLAIN
    org = (20, 20)
    font_scale = 4
    font_color = (255, 255, 255)
    line_type = 1
    font_thickness = 2
    text_color_bg = (0, 0, 0)
    x, y = org

    image = cv2.UMat(image)
    (text_w, text_h), _ = cv2.getTextSize(
        text=text, fontFace=font, fontScale=font_scale, thickness=font_thickness
    )
    result_im = cv2.rectangle(
        img=image, pt1=org, pt2=(x + text_w, y + text_h), color=text_color_bg, thickness=-1
    )

    textim = cv2.putText(
        img=result_im,
        text=text,
        org=(x, y + text_h + font_scale - 1),
        fontFace=font,
        fontScale=font_scale,
        color=font_color,
        thickness=font_thickness,
        lineType=line_type,
    )
    return textim.get()


def load_image(path: str) -> np.ndarray:
    """
    Loads an image from `path` and returns it as BGR numpy array.

    :param path: path to an image filename or url
    :return: image as numpy array, with BGR channel order
    """
    if path.startswith("http"):
        # Set User-Agent to Mozilla because some websites block requests
        # with User-Agent Python
        request = urllib.request.Request(url=path, headers={"User-Agent": "Mozilla/5.0"})
        response = urllib.request.urlopen(url=request)
        array = np.asarray(bytearray(response.read()), dtype="uint8")
        image = cv2.imdecode(buf=array, flags=-1)  # Loads the image as BGR
    else:
        image = cv2.imread(filename=path)
    return image


def convert_result_to_image(result) -> np.ndarray:
    """
    Convert network result of floating point numbers to image with integer
    values from 0-255. Values outside this range are clipped to 0 and 255.

    :param result: a single superresolution network result in N,C,H,W shape
    """
    result = result.squeeze(0).transpose(1, 2, 0)
    result *= 255
    result[result < 0] = 0
    result[result > 255] = 255
    result = result.astype(np.uint8)
    return result<jupyter_output><empty_output><jupyter_text>Load the Superresolution Model Load the model in Inference Engine with `ie.read_network` and load it to the specified device with `ie.load_network`<jupyter_code>ie = IECore()
net = ie.read_network(model=model_xml_path)
exec_net = ie.load_network(network=net, device_name=DEVICE)<jupyter_output><empty_output><jupyter_text>Get information about network inputs and outputs. The Super Resolution model expects two inputs: 1) the input image, 2) a bicubic interpolation of the input image to the target size 1920x1080. It returns the super resolution version of the image in 1920x1800.<jupyter_code># Network inputs and outputs are dictionaries. Get the keys for the
# dictionaries.
original_image_key = list(exec_net.input_info)[0]
bicubic_image_key = list(exec_net.input_info)[1]
output_key = list(exec_net.outputs.keys())[0]

# Get the expected input and target shape. `.dims[2:]` returns the height
# and width. OpenCV's resize function expects the shape as (width, height),
# so we reverse the shape with `[::-1]` and convert it to a tuple
input_height, input_width = tuple(exec_net.input_info[original_image_key].tensor_desc.dims[2:])
target_height, target_width = tuple(exec_net.input_info[bicubic_image_key].tensor_desc.dims[2:])

upsample_factor = int(target_height / input_height)

print(f"The network expects inputs with a width of {input_width}, " f"height of {input_height}")
print(f"The network returns images with a width of {target_width}, " f"height of {target_height}")

print(
    f"The image sides are upsampled by a factor {upsample_factor}. "
    f"The new image is {upsample_factor**2} times as large as the "
    "original image"
)<jupyter_output><empty_output><jupyter_text>Superresolution on VideoDownload a YouTube\* video with PyTube and enhance the video quality with superresolution. By default only the first 100 frames of the video are processed. Change NUM_FRAMES in the cell below to modify this. **Note:**- The resulting video does not contain audio.- The input video should be a landscape video and have an an input resultion of 360p (640x360) for the 1032 model, or 480p (720x480) for the 1033 model.  Settings<jupyter_code>VIDEO_DIR = "data"
OUTPUT_DIR = "output"

os.makedirs(name=str(OUTPUT_DIR), exist_ok=True)
# Number of frames to read from the input video. Set to 0 to read all frames.
NUM_FRAMES = 100
# The format for saving the result video's
# vp09 is slow, but widely available. If you have FFMPEG installed, you can
# change the FOURCC to `*"THEO"` to improve video writing speed
FOURCC = cv2.VideoWriter_fourcc(*"vp09")<jupyter_output><empty_output><jupyter_text>Download and Prepare Video<jupyter_code># Use pytube to download a video. It downloads to the videos subdirectory.
# You can also place a local video there and comment out the following lines
VIDEO_URL = "https://www.youtube.com/watch?v=V8yS3WIkOrA"
yt = YouTube(VIDEO_URL)
# Use `yt.streams` to see all available streams. See the PyTube documentation
# https://python-pytube.readthedocs.io/en/latest/api.html for advanced
# filtering options
try:
    os.makedirs(name=VIDEO_DIR, exist_ok=True)
    stream = yt.streams.filter(resolution="360p").first()
    filename = Path(stream.default_filename.encode("ascii", "ignore").decode("ascii")).stem
    stream.download(output_path=OUTPUT_DIR, filename=filename)
    print(f"Video {filename} downloaded to {OUTPUT_DIR}")

    # Create Path objects for the input video and the resulting videos
    video_path = Path(stream.get_file_path(filename, OUTPUT_DIR))
except Exception:
    # If PyTube fails, use a local video stored in the VIDEO_DIR directory
    video_path = Path(rf"{VIDEO_DIR}/CEO <NAME> on Leading Intel.mp4")

# Path names for the result videos
superres_video_path = Path(f"{OUTPUT_DIR}/{video_path.stem}_superres.mp4")
bicubic_video_path = Path(f"{OUTPUT_DIR}/{video_path.stem}_bicubic.mp4")
comparison_video_path = Path(f"{OUTPUT_DIR}/{video_path.stem}_superres_comparison.mp4")
# Open the video and get the dimensions and the FPS
cap = cv2.VideoCapture(filename=str(video_path))
ret, image = cap.read()
if not ret:
    raise ValueError(f"The video at '{video_path}' cannot be read.")
fps = cap.get(cv2.CAP_PROP_FPS)
original_frame_height, original_frame_width = image.shape[:2]

cap.release()
print(
    f"The input video has a frame width of {original_frame_width}, "
    f"frame height of {original_frame_height} and runs at {fps:.2f} fps"
)<jupyter_output><empty_output><jupyter_text>Create superresolution video, bicubic video and comparison video. The superresolution video contains the enhanced video, upsampled with superresolution, the bicubic video is the input video upsampled with bicubic interpolation, the combination video sets the bicubic video and the superresolution side by side.<jupyter_code>superres_video = cv2.VideoWriter(
    filename=str(superres_video_path),
    fourcc=FOURCC,
    fps=fps,
    frameSize=(target_width, target_height),
)
bicubic_video = cv2.VideoWriter(
    filename=str(bicubic_video_path),
    fourcc=FOURCC,
    fps=fps,
    frameSize=(target_width, target_height),
)
comparison_video = cv2.VideoWriter(
    filename=str(comparison_video_path),
    fourcc=FOURCC,
    fps=fps,
    frameSize=(target_width * 2, target_height),
)<jupyter_output><empty_output><jupyter_text>Do InferenceRead video frames and enhance them with superresolution. Save the superresolution video, the bicubic video and the comparison video to file.The code in this cell reads the video frame by frame. Each frame is resized and reshaped to network input shape and upsampled with bicubic interpolation to target shape. Both the original and the bicubic image are propagated through the network. The network result is a numpy array with floating point values, with a shape of (1,3,1920,1080). This array is converted to an 8-bit image with shape (1080,1920,3) and written to `superres_video`. The bicubic image is written to `bicubic_video` for comparison. Lastly, the bicubic and result frames are combined side by side and written to `comparison_video`. A progress bar shows the progress of the process. Inference time is measured, as well as total time to process each frame, which includes inference time as well as the time it takes to process and write the video.<jupyter_code>start_time = time.perf_counter()
frame_nr = 1
total_inference_duration = 0
total_frames = cap.get(cv2.CAP_PROP_FRAME_COUNT) if NUM_FRAMES == 0 else NUM_FRAMES

progress_bar = ProgressBar(total=total_frames)
progress_bar.display()


cap = cv2.VideoCapture(filename=str(video_path))
try:
    while cap.isOpened():
        ret, image = cap.read()
        if not ret:
            cap.release()
            break

        if NUM_FRAMES > 0 and frame_nr == NUM_FRAMES:
            break

        # Resize the input image to network shape and convert from (H,W,C) to
        # (N,C,H,W)
        resized_image = cv2.resize(src=image, dsize=(input_width, input_height))
        input_image_original = np.expand_dims(resized_image.transpose(2, 0, 1), axis=0)

        # Resize and reshape the image to the target shape with bicubic
        # interpolation
        bicubic_image = cv2.resize(
            src=image, dsize=(target_width, target_height), interpolation=cv2.INTER_CUBIC
        )
        input_image_bicubic = np.expand_dims(bicubic_image.transpose(2, 0, 1), axis=0)

        # Do inference
        inference_start_time = time.perf_counter()
        result = exec_net.infer(
            inputs={
                original_image_key: input_image_original,
                bicubic_image_key: input_image_bicubic,
            }
        )[output_key]
        inference_stop_time = time.perf_counter()
        inference_duration = inference_stop_time - inference_start_time
        total_inference_duration += inference_duration

        # Transform inference result into an image
        result_frame = convert_result_to_image(result=result)

        # Write resulting image and bicubic image to video
        superres_video.write(image=result_frame)
        bicubic_video.write(image=bicubic_image)
        stacked_frame = np.hstack((bicubic_image, result_frame))
        comparison_video.write(image=stacked_frame)

        frame_nr = frame_nr + 1

        # Update progress bar and status message
        progress_bar.progress = frame_nr
        progress_bar.update()
        if frame_nr % 10 == 0:
            clear_output(wait=True)
            progress_bar.display()
            display(
                Pretty(
                    f"Processed frame {frame_nr}. Inference time: "
                    f"{inference_duration:.2f} seconds "
                    f"({1/inference_duration:.2f} FPS)"
                )
            )


except KeyboardInterrupt:
    print("Processing interrupted.")
finally:
    superres_video.release()
    bicubic_video.release()
    comparison_video.release()
    end_time = time.perf_counter()
    duration = end_time - start_time
    print(f"Video's saved to {comparison_video_path.parent} directory.")
    print(
        f"Processed {frame_nr} frames in {duration:.2f} seconds. Total FPS "
        f"(including video processing): {frame_nr/duration:.2f}. "
        f"Inference FPS: {frame_nr/total_inference_duration:.2f}."
    )<jupyter_output><empty_output><jupyter_text>Show Side-by-Side Video of Bicubic and Superresolution Version<jupyter_code>if not comparison_video_path.exists():
    raise ValueError("The comparison video does not exist.")
else:
    video_link = FileLink(comparison_video_path)
    video_link.html_link_str = "<a href='%s' download>%s</a>"
    display(
        HTML(
            f"Showing side by side comparison. If you cannot see the video in "
            "your browser, please click on the following link to download "
            f"the video<br>{video_link._repr_html_()}"
        )
    )
    display(Video(comparison_video_path, width=800, embed=True))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08886711916590861, 'fraction_numerical': 0.018753061367294103, 'mean_word_length': 3.955617198335645, 'pattern_counts': {'":': 1, '<': 47, '<?xml version=': 0, '>': 51, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '12360', 'license': 'Apache-2.0', 'n_tokens_mistral': 4664, 'n_tokens_neox': 4353, 'n_words': 1360, 'path': 'notebooks/202-vision-superresolution/202-vision-superresolution-video.ipynb', 'repo_name': 'SalnikovIgor/openvino_notebooks', 'size': '14298'}