TECHNICAL FIELD

The present disclosure generally relates to building surgical procedure video analysis tools and, more specifically, to systems, devices and techniques for synchronizing surgical videos of a given surgical procedure so that the surgical videos can be played back automatically synchronized at a sequence of predefined phases of the surgical procedure for comparative learning.

BACKGROUND

Recorded videos of medical procedures such as surgeries contain highly valuable and rich information for medical education and training, assessing and analyzing the quality of the surgeries and skills of the surgeons, and for improving the outcomes of the surgeries and skills of the surgeons. There are many surgical procedures which involve displaying and capturing video images of the surgical procedures. For example, almost all minimally invasive procedures (MIS), such as endoscopy, laparoscopy, and arthroscopy, involve using video cameras and video images to assist the surgeons. Furthermore, the state-of-the-art robotic-assisted surgeries require intraoperative video images being captured and displayed on the monitors for the surgeons. Consequently, for many of the aforementioned surgical procedures, e.g., a gastric sleeve or cholecystectomy, a large cache of surgical videos already exist and continue to be created as a result of a large number of surgical cases performed by many different surgeons from different hospitals.

It is well-accepted that a recorded surgical procedure video can be used to assess and analyze the quality of the surgery and skills of the surgeon. However, by merely watching a surgeon performing certain surgical tasks recorded on a surgical procedure video, it can be quite difficult to objectively assess the surgeon's performance because there is no clear reference or metric for gauging whether the surgeon did a good job or not in each surgical task. As an improvement to the above technique, a comparative-learning technique involves an expert simultaneously watching two surgical videos performed by two surgeons, visually comparing the techniques of the two surgeons, and more objectively assigning scores for the surgical tasks performed by the two surgeons. As a further improvement to the above techniques, another comparative-learning technique involves two recorded procedure videos playing side by side: the first video being a reference video performed by an expert of the surgical procedure; whereas the second video being an evaluation video performed by a second surgeon. Again, an expert is tasked to simultaneously watch the two side-by-side videos, compare the techniques of the second surgeon against those of the expert, and subsequently assign a skill score for the second surgeon based on the direct comparison of the second surgeon in the second video to the expert in the first video.

SUMMARY

This patent disclosure provides various embodiments of preparing two procedure videos, in particular two surgical procedure videos for comparative learning. In some embodiments, to allow comparative learning of two recorded surgical videos, each of the two recorded surgical videos is segmented into a sequence of predefined phases/steps. Next, corresponding phases/steps of the two segmented videos are individually time-synchronized in pair-wise manner so that a given phase/step of one segmented video and a corresponding phase/step of the other segmented video can have the same or substantially the same starting time and ending timing during comparative playbacks of the two recorded videos, e.g., on two side-by-side monitors.

In some embodiments, the proposed comparative learning techniques also include comparative playing and viewing of a recorded video and a live video feed side by side. To do so, the recorded video may be segmented into a sequence of predefined steps prior to the playback (i.e., processed offline). Next, during a live practice session, the proposed technique s provide a mechanism to identify the predefined steps in the live video feed in real-time and subsequently slave the playback of a predefined step in the recorded video to a corresponding predefined step identified in the live video feed. In this manner, the recorded video can pause, restart, and follow the speed of the live video feed depicting a person replicating the actions in the recorded video through the sequence of predefined steps.

In one aspect, a process for synchronizing the playback of a recorded video of a surgical procedure with a live video feed of a user performing the surgical procedure is disclosed. This process can begin by simultaneously receiving a recorded video of a surgical procedure and a live video feed of a user performing the surgical procedure in a training session. More specifically, the recorded video is shown to the user as a training reference, and the surgical procedure includes a set of surgical tasks. The process next simultaneously monitors the playback of a current surgical task in the set of surgical tasks in the recorded video and the live video feed depicting the user performing the current surgical task. Next, the process detects that the end of the current surgical task has been reached during the playback of the recorded video. In response to determining that the user has not completed the current surgical task in the live video feed, the process pauses the playback of the recorded video while awaiting the user to complete the current surgical task.

In some embodiments, the recorded video is a training video performed by an expert of the surgical procedure, whereas the live video feed depicts the user replicating actions of the expert performing the current surgical task displayed in the training video.

In some embodiments, the process simultaneously displays the playback of the recorded video and the live video feed of the surgical procedure side-by-side on a split-screen display or on a pair of dual monitors.

In some embodiments, the process detects that the end of the current surgical task has been reached during the playback of the recorded video by first receiving a set of landmark events, wherein each landmark event in the set of landmark events indicates the end of a corresponding surgical task in the set of surgical tasks. The process subsequently detects the occurrence of a landmark event in the set of landmark events corresponding to the current surgical task during the playback of the current surgical task.

In some embodiments, prior to monitoring the playback of the recorded video and the live video feed, the process generates the set of landmark events by: performing a segmentation operation on the recorded video to segment the recorded video into a set of video segments corresponding to the set of surgical tasks; and extracting a landmark event in the set of landmark events from the end of each video segment in the set of video segments.

In some embodiments, each landmark event in the set of landmark events includes one of: (1) a predefined action indicating the end of a corresponding surgical task; and (2) a timestamp indicating the end of the corresponding surgical task.

In some embodiments, the process determines whether the user has completed the current surgical task by determining whether the landmark event associated with the current surgical task has occurred in the live video feed.

In some embodiments, after pausing the playback of the recorded video while awaiting the user to complete the current surgical task, the process detects that the user has completed the current surgical task based on the live video feed. The process subsequently resumes the playback of the recorded video while continuing monitoring the playback of the subsequent surgical task in the recorded video and the live video feed depicting the user performing the subsequent surgical task. Note that automatically pausing and resuming the playback of the recorded video based on the live video feed facilitates automatically synchronizing the playback of the set of surgical tasks in the recorded video with the user's progress through the set of surgical tasks without any user intervention.

In some embodiments, the process further includes: determining that the user executes one or more surgical tasks consistently slower than the playback of the one or more surgical tasks in the recorded video; and decreasing the playback speed of the recorded video so that the playback of the recorded video becomes more synchronized with the execution of the surgical procedure by the user depicted in the live video feed.

In some embodiments, the process further includes: determining that the user executes one or more surgical tasks consistently faster than the playback of the one or more surgical tasks in the recorded video; and increasing the playback speed of the recorded video so that the playback of the recorded video becomes more synchronized with the execution of the surgical procedure by the user depicted in the live video feed.

In another aspect, a system for synchronizing the playback of a recorded video of a surgical procedure with a live video feed of a user performing the surgical procedure is disclosed. This system can include: one or more processors and a memory coupled to the one or more processors. The memory stores instructions that, when executed by the one or more processors, cause the system to: simultaneously receive a recorded video of a surgical procedure and a live video feed of a user performing the surgical procedure in a training session, wherein the recorded video is shown to the user as a training reference, and wherein the surgical procedure includes a set of surgical tasks; simultaneously monitor the playback of a current surgical task in the set of surgical tasks in the recorded video and the live video feed depicting the user performing the current surgical task; detect that the end of the current surgical task has been reached during the playback of the recorded video; and, responsive to determining that the user has not completed the current surgical task in the live video feed, pause the playback of the recorded video while awaiting the user to complete the current surgical task.

In yet another aspect, a virtual reality (VR) device for synchronizing the playback of a recorded video of a surgical procedure with a live video feed of a user performing the surgical procedure in an immersive display is disclosed. This VR device can include: a split-screen VR display comprising a left-half display and a right-half display, one or more processors, and a memory coupled to the one or more processors. The memory of the VR device stores instructions that, when executed by the one or more processors, cause the VR device to: simultaneously receive a recorded video of a surgical procedure and a live video feed of a user performing the surgical procedure in a training session, wherein the recorded video is shown to the user as a training reference, and wherein the surgical procedure includes a set of surgical tasks; simultaneously monitor the playback of a current surgical task in the set of surgical tasks in the recorded video and the live video feed depicting the user performing the current surgical task; detect that the end of the current surgical task has been reached during the playback of the recorded video; and, responsive to determining that the user has not completed the current surgical task in the live video feed, pause the playback of the recorded video while awaiting the user to complete the current surgical task.

