<jupyter_start><jupyter_text>How to Use VTKOverlayWindowscikit-surgeryvtk provides a simple PySide2/VTK widget called VTKOverlayWindow for doing simple augmented reality (AR) overlays, using calibrated parameters, like those obtained from OpenCV. The video image is rendered as a background layer, with VTK models as a foreground layer. This is not very photo-realistic, but most researchers are focussing on registration/alignment rather than true 3D perception.In this page, we give a brief overview of the VTKOverlayWindow, a more detailed tutorial can be found in [SciKit-Surgery Augmented Reality Tutorial](https://scikit-surgerytutorial01.readthedocs.io/en/latest/)The Jupyter notebook for this page can be found in the `docs/tutorials` folder of the [code repository](https://github.com/SciKit-Surgery/scikit-surgeryvtk/). Required modulesThis tutorial requires numpy, scikit-surgeryvtk, scikit-surgeryutils and opencv-contrib-python. Quick implementationOne of the most common uses is to overlay some models on a camera feed, and a sample implementation is included in the scikit-surgeryutils module, which requires only a few lines of code.**Note** - Running Qt Widgets from a Jupyter notebook can be a bit temperemental. Remember to close any Qt windows once you have finished with them, before moving onto a different section. If there are any issues, restarting the Jupyter kernel should be sufficient to fix.The code can also be run as a standard .py script file.<jupyter_code>import sksurgeryutils.common_overlay_apps as coa
from PySide2 import QtWidgets, QtCore<jupyter_output><empty_output><jupyter_text>The below code will launch a Qt widget in a new window, where you should be able to interact with the loaded models (drag/rotate etc).<jupyter_code>app = QtCore.QCoreApplication.instance()
if app is None:
    app = QtWidgets.QApplication([])

camera_source = 0
wind = coa.OverlayOnVideoFeed(camera_source)
wind.add_vtk_models_from_dir('../../tests/data/models/Liver')
wind.start()

app.exec_()<jupyter_output><empty_output><jupyter_text>Detailed ImplementationThe `OverlayOnVideoFeed` class has an `update` method which is called to read a new frame from the webcam and display it in the widget. By writing our own `update` method, we can add additional functionality.<jupyter_code>import cv2
from PySide2 import QtWidgets, QtCore
import sksurgeryvtk.widgets.vtk_overlay_window as OW
import sksurgeryvtk.models.vtk_surface_model as SM
import sksurgeryvtk.models.vtk_surface_model_directory_loader as SMDL<jupyter_output><empty_output><jupyter_text>Now we define a new class, using a `QTimer` to call the `update` method, to which we can add whatever processing is desired. In this instance it just flips the video once it has been received.<jupyter_code>class OverlayApp():
    
    def __init__(self, video_source):
        self.vtk_overlay_window = OW.VTKOverlayWindow()
        self.video_source = cv2.VideoCapture(video_source)
        
        self.timer = QtCore.QTimer()
        self.timer.timeout.connect(self.update)

        update_frequency_ms = 100
        self.timer.start(update_frequency_ms)
        
        self.vtk_overlay_window.show()
        
    def update(self):
        ret, img = self.video_source.read()
        img = cv2.flip(img, 0)
        self.vtk_overlay_window.set_video_image(img)
        self.vtk_overlay_window._RenderWindow.Render()
app = QtCore.QCoreApplication.instance()
if app is None:
    app = QtWidgets.QApplication([])

camera_source = 0
overlay_app = OverlayApp(camera_source)

app.exec_()<jupyter_output><empty_output><jupyter_text>Overlay models can be loaded from a directory:<jupyter_code>app = QtCore.QCoreApplication.instance()
if app is None:
    app = QtWidgets.QApplication([])

camera_source = 0
overlay_app = OverlayApp(camera_source)

model_dir = '../../tests/data/models/Liver'
model_loader = SMDL.VTKSurfaceModelDirectoryLoader(model_dir)

overlay_app.vtk_overlay_window.add_vtk_models(model_loader.models)

app.exec_()<jupyter_output><empty_output><jupyter_text>Or from individual files. It is also possible to set the colour and opacity individually for each model:<jupyter_code>app = QtCore.QCoreApplication.instance()
if app is None:
    app = QtWidgets.QApplication([])

camera_source = 0
overlay_app = OverlayApp(camera_source)

liver_file= '../../tests/data/models/Liver/liver.vtk'
portal_vein_file = '../../tests/data/models/Liver/portal_vein.vtk'

liver_model = SM.VTKSurfaceModel(liver_file, colour = [1.0, 0.0, 0.0], opacity = 0.5)
vein_model = SM.VTKSurfaceModel(portal_vein_file, colour = [0.0, 1.0, 0.0])


overlay_app.vtk_overlay_window.add_vtk_models([liver_model, vein_model])

app.exec_()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08313891834570519, 'fraction_numerical': 0.006574761399787911, 'mean_word_length': 5.053915275994865, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '280459', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1479, 'n_tokens_neox': 1453, 'n_words': 440, 'path': 'docs/tutorials/overlay_window.ipynb', 'repo_name': 'SciKit-Surgery/scikit-surgeryvtk', 'size': '4715'}