Surgical tool segmentation is the task of labelling each pixel of an image as belonging to a specific class among background and tool, using a single instrument class in case of binary segmentation.
First attempts to solve this problem used hand-crafted image features, machine learning models (Support Vector Machine, Random Forests) and template matching using tool shape-priors ([1]}, [2]}). Nowadays, research works mostly address the problem of surgical tool segmentation using fully-supervised Deep Learning approaches, which have proved to outperform other existing methods ([3]}). In particular, Convolutional Neural Network (CNN) architectures have been widely adopted. [4]} propose a multi-scale and holistically nested CNN light-weight architecture trained with Dice loss function; [5]} modify a VGG16 architecture by adding skip connections, winning the 2017 MICCAI EndoVis Robotic Instrument Segmentation challenge ([6]}). [7]} leverage deep residual learning and dilated convolutions for binary and part tool segmentation. [8]} propose a variation of the standard U-Net architecture ([9]}) having a modified decoder and an improved augmentation pipeline. Multi-task learning has also been explored by [10]}, by simultaneously learning segmentation and tool landmarks localization, and by [11]}, by introducing the Spatio-Temporal Multi-Task Learning (ST-MTL) model, for surgical instrument segmentation and task-oriented saliency detection. [12]} propose an attention based approach leveraging motion information, in the form of optical-flow, to improve segmentation accuracy. [13]} propose a bilinear attention network with an adaptive receptive field to tackle the challenges of scale and illumination inter-frames variability. [14]} propose an alternative to standard semantic segmentation, first extracting instrument instances and then independently classifying them, reaching state-of-the-art results for this task. Despite the good results obtained by fully-supervised methods, their application is inherently limited by the need for manual annotations, which prevents their scalability. In order to mitigate this problem, [15]} produce semi-synthetic samples, merging automatically segmented tools from green-screen recordings and real surgical background images. In the context of robotic surgery, [16]} propose the combined use of recorded kinematics and green-screen, in order to cheaply obtain ground-truth segmentation masks. Several works have also tried to tackle the segmentation problem by including synthetic or unlabelled data, in combination with generative approaches. [17]} propose Endo-Sim2Real, a consistency-based framework for joint training from simulated and unlabelled real data. [18]} propose a cycle Generative Adversarial Network (cycle-GAN) approach to convert synthetic tools into real-looking ones, to be then blended with surgical background images, to form semi-synthetic samples.
[19]} pre-train a CNN on unlabelled data, by means of a pretext task carried out using a cycle-GAN architecture, showing a significant boost in segmentation accuracy.
[20]} incorporate unlabelled data in the training process, by mapping annotated frames to the unlabelled data domain using a cycle-GAN architecture, allowing for better generalization to the unlabelled domain. [21]} use a pix2pix GAN to generate synthetic surgical images from rough segmentation mask of surgical instruments and tissues. In the context of robotic surgery, [22]} record synchronized surgical videos and kinematic joint values and then use the letter to generate synthetic annotations, projecting the estimated tool 3D shapes, obtained via forward kinematics, onto the image space; in order to take into account the possible inaccuracy of the tool model, the segmentation problem is formulated as unpaired image-to-image translation, using a cycle-GAN architecture. An alternative proposed solution to reduce the need for manual annotations is represented by semi-supervision using label propagation. [23]} propose a flow prediction and compensation framework for semi-supervised tool segmentation, propagating low hertz annotations to unlabelled data using optical-flow. Finally, an unsupervised approach is proposed by [24]}, which generate tool pseudo-labels using handcrafted cues, such as color distribution, and then refine segmentation results exploiting feature correlation between adjacent video frames.
