We aim to detect all instances of a category in an image and, for each instance, mark the pixels that belong to it. We call this task Simultaneous Detection and Segmentation (SDS). Unlike classical bounding box detection, SDS requires a segmentation and not just a box. Unlike classical semantic segmentation, we require individual object instances. We build on recent work that uses convolutional neural networks to classify category-independent region proposals (R-CNN [16]), introducing a novel architecture tailored for SDS. We then use category-specific, top- down figure-ground predictions to refine our bottom-up proposals. We show a 7 point boost (16 relative) over our baselines on SDS, a 5 point boost (10 relative) over state-of-the-art on semantic segmentation, and state-of-the-art performance in object detection. Finally, we provide diagnostic tools that unpack performance and provide directions for future work.
In many of these approaches, segmentation is used only to localize the detections better. Other authors have explored using segmentation as a stronger cue. Fidler al @cite_2 use the output of a state-of-the-art semantic segmentation approach @cite_15 to score detections better. Mottaghi @cite_19 uses detectors based on non-rectangular patches to both detect and segment objects.
@cite_19 @cite_15 @cite_2
2112357820 78159342
The performance of part-based object detectors generally degrades for highly flexible objects. The limited topological structure of models and pre-specified part shapes are two main factors preventing these detectors from fully capturing large deformations. To better capture the deformations, we propose a novel approach to integrate the detections from a family of part-based detectors with patches of objects that have irregular shape. This integration is formulated as MAP inference in a Conditional Random Field (CRF). The energy function defined over the CRF takes into account the information provided by an object patch classifier and the object detector, and the goal is to augment the partial detections with missing patches, and also to refine the detections that include background clutter. The proposed method is evaluated on the object detection task of PASCAL VOC. Our experimental results show significant improvement over a base part-based detector (which is among the current state-of-the-art methods) especially for the deformable object classes. Feature extraction, coding and pooling, are important components on many contemporary object recognition paradigms. In this paper we explore novel pooling techniques that encode the second-order statistics of local descriptors inside a region. To achieve this effect, we introduce multiplicative second-order analogues of average and max-pooling that together with appropriate non-linearities lead to state-of-the-art performance on free-form region recognition, without any type of feature coding. Instead of coding, we found that enriching local descriptors with additional image information leads to large performance gains, especially in conjunction with the proposed pooling methodology. We show that second-order pooling over free-form regions produces results superior to those of the winning systems in the Pascal VOC 2011 semantic segmentation challenge, with models that are 20,000 times faster.