Automatic recognition and classification of tasks in robotic surgery is an important stepping stone toward automated surgery and surgical training. Recently, technical breakthroughs in gathering data make data-driven model development possible. In this paper, we propose a framework for high-level robotic surgery task recognition using motion data. We present a novel classification technique that is used to classify three important surgical tasks through quantitative analyses of motion: knot tying, needle passing and suturing. The proposed technique integrates state-of-the-art data mining and time series analysis methods. The first step of this framework consists of developing a time series distance-based similarity measure using derivative dynamic time warping (DDTW). The distance-weighted k-nearest neighbor algorithm was then used to classify task instances. The framework was validated using an extensive dataset. Our results demonstrate the strength of the proposed framework in recognizing fundamental robotic surgery tasks.
In recent years, recognizing and understanding surgical procedures at different levels of granularity has been a focus of research @cite_33 @cite_24 . Surgical procedures can be generally broken down to four main levels, from higher to lower: phases, steps, tasks and motions @cite_25 . At the higher level of surgical process modeling, statistical models have been proposed using recorded force and motion data @cite_20 @cite_0 , surgical tool usage @cite_4 and video data @cite_29 to classify surgery phases. Most existing work has addressed the recognition of activities using different techniques such as neural networks and Hidden Markov Models (HMM) @cite_8 @cite_15 . At the lower level, effort has been applied to detect surgical motion @cite_34 @cite_27 or model surgical gestures and classify them using different methods such as HMM and Linear Discriminant Analysis (LDA) @cite_21 . A common drawback in these methods is that they are time consuming and require significant human interaction and pre-processing.
@cite_4 @cite_33 @cite_8 @cite_29 @cite_21 @cite_24 @cite_0 @cite_27 @cite_15 @cite_34 @cite_25 @cite_20
2538822334 2064767749 2468278636 2054866636
The automatic analysis of the surgical process, from videos recorded during surgeries, could be very useful to surgeons, both for training and for acquiring new techniques. The training process could be optimized by automatically providing some targeted recommendations or warnings, similar to the expert surgeon’s guidance. In this paper, we propose to reuse videos recorded and stored during cataract surgeries to perform the analysis. The proposed system allows to automatically recognize, in real time, what the surgeon is doing: what surgical phase or, more precisely, what surgical step he or she is performing. This recognition relies on the inference of a multilevel statistical model which uses 1) the conditional relations between levels of description (steps and phases) and 2) the temporal relations among steps and among phases. The model accepts two types of inputs: 1) the presence of surgical tools, manually provided by the surgeons, or 2) motion in videos, automatically analyzed through the Content Based Video retrieval (CBVR) paradigm. Different data-driven statistical models are evaluated in this paper. For this project, a dataset of 30 cataract surgery videos was collected at Brest University hospital. The system was evaluated in terms of area under the ROC curve. Promising results were obtained using either the presence of surgical tools (A z = 0.983) or motion analysis (A z = 0.759). The generality of the method allows to adapt it to other kinds of surgeries. The proposed solution could be used in a computer assisted surgery tool to support surgeons during the surgery. In ophthalmology, it is now common practice to record every surgical procedure and to archive the resulting videos for documentation purposes. In this paper, we present a solution to automatically segment and categorize surgical tasks in real-time during the surgery, using the video recording. The goal would be to communicate information to the surgeon in due time, such as recommendations to the less experienced surgeons. The proposed solution relies on the content-based video retrieval paradigm: it reuses previously archived videos to automatically analyze the current surgery, by analogy reasoning. Each video is segmented, in real-time, into an alternating sequence of idle phases, during which no clinically-relevant motions are visible, and action phases. As soon as an idle phase is detected, the previous action phase is categorized and the next action phase is predicted. A conditional random field is used for categorization and prediction. The proposed system was applied to the automatic segmentation and categorization of cataract surgery tasks. A dataset of 186 surgeries, performed by ten different surgeons, was manually annotated: ten possibly overlapping surgical tasks were delimited in each surgery. Using the content of action phases and the duration of idle phases as sources of evidence, an average recognition performance of @math was achieved. Data recorded and stored during video-monitored surgeries are a relevant source of information for surgeons, especially during their training period. But today, this data is virtually unexploited. In this paper, we propose to reuse videos recorded during cataract surgeries to automatically analyze the surgical process with the real-time constraint, with the aim to assist the surgeon during the surgery. We propose to automatically recognize, in real-time, what the surgeon is doing: what surgical phase or, more precisely, what surgical step he or she is performing. This recognition relies on the inference of a multilevel statistical model which uses 1) the conditional relations between levels of description (steps and phases) and 2) the temporal relations among steps and among phases. The model accepts two types of inputs: 1) the presence of surgical instruments, manually provided by the surgeons, or 2) motion in videos, automatically analyzed through the CBVR paradigm. A dataset of 30 cataract surgery videos was collected at Brest University hospital. The system was evaluated in terms of mean area under the ROC curve. Promising results were obtained using either motion analysis (Az = 0.759) or the presence of surgical instruments (Az = 0.983). The need for a better integration of the new generation of computer-assisted-surgical systems has been recently emphasized. One necessity to achieve this objective is to retrieve data from the operating room (OR) with different sensors, then to derive models from these data. Recently, the use of videos from cameras in the OR has demonstrated its efficiency. In this paper, we propose a framework to assist in the development of systems for the automatic recognition of high-level surgical tasks using microscope videos analysis. We validated its use on cataract procedures. The idea is to combine state-of-the-art computer vision techniques with time series analysis. The first step of the framework consisted in the definition of several visual cues for extracting semantic information, therefore, characterizing each frame of the video. Five different pieces of image-based classifiers were, therefore, implemented. A step of pupil segmentation was also applied for dedicated visual cue detection. Time series classification algorithms were then applied to model time-varying data. Dynamic time warping and hidden Markov models were tested. This association combined the advantages of all methods for better understanding of the problem. The framework was finally validated through various studies. Six binary visual cues were chosen along with 12 phases to detect, obtaining accuracies of 94 .