We present a simple, fully-convolutional model for real-time instance segmentation that achieves 29.8 mAP on MS COCO at 33 fps evaluated on a single Titan Xp, which is significantly faster than any previous competitive approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. Finally, we also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty.
While real-time object detection @cite_43 @cite_29 @cite_38 @cite_0 , and semantic segmentation @cite_22 @cite_8 @cite_20 @cite_35 @cite_13 methods exist, few works have focused on real-time instance segmentation. Straight to Shapes @cite_37 can perform instance segmentation with learned encodings of shapes at 30 fps, but its accuracy is far from that of modern baselines. Box2Pix @cite_47 relies on an extremely light-weight backbone detector (GoogLeNet v1 @cite_27 and SSD @cite_43 ) combined with a hand-engineered algorithm to obtain 10.9 fps on Cityscapes @cite_23 and 35 fps on KITTI @cite_42 . However, they report no results on the more challenging and semantically-rich COCO dataset, which has 80 classes compared to the 8 of KITTI and Cityscapes. Furthermore, they observe a large drop in relative performance going from a semantically simple dataset (KITTI) to a more complex one (Cityscapes), so an even more difficult dataset (COCO) would pose a challenge. In fact, Mask R-CNN @cite_14 remains one of the fastest instance segmentation methods on semantically challenging datasets (13.5 fps on @math px images; see Table ).
@cite_38 @cite_35 @cite_37 @cite_14 @cite_22 @cite_8 @cite_29 @cite_42 @cite_0 @cite_43 @cite_27 @cite_23 @cite_47 @cite_13 @cite_20
2963342610 2809110088 2774561273 2580683366
An increasing need of running Convolutional Neural Network (CNN) models on mobile devices with limited computing power and memory resource encourages studies on efficient model design. A number of efficient architectures have been proposed in recent years, for example, MobileNet, ShuffleNet, and NASNet-A. However, all these models are heavily dependent on depthwise separable convolution which lacks efficient implementation in most deep learning frameworks. In this study, we propose an efficient architecture named PeleeNet, which is built with conventional convolution instead. On ImageNet ILSVRC 2012 dataset, our proposed PeleeNet achieves a higher accuracy by 1.8 (72.4 vs. 70.6 ) and 23 faster speed than MobileNet, the state-of-the-art efficient architecture. Meanwhile, PeleeNet is only 66 of the model size of MobileNet. We then propose a real-time object detection system by combining PeleeNet with Single Shot MultiBox Detector (SSD) method and optimizing the architecture for fast speed. Our proposed detection system, named Pelee, achieves 76.4 mAP (mean average precision) on PASCAL VOC2007 and 22.4 mAP on MS COCO dataset at the speed of 23.6 FPS on iPhone 8 and 74 FPS on NVIDIA TX2. The result on COCO outperforms YOLOv2 in consideration of a higher precision, 13.6 times lower computational cost and 11.3 times smaller model size. The code and models are open sourced. The task of semantic instance segmentation has gained a large interest within academia as well as industry, especially in the context of autonomous driving. While several published approaches achieve very strong results, only few of them achieve frame rates that are sufficient for the automotive domain. We present an approach that achieves competitive results on the Cityscapes [1] and KITTI [2] datasets, while being twice as fast as any other existing approach. Our method relies on a single fully-convolutional network (FCN [3]) predicting object bounding boxes, as well as pixel-wise semantic object classes and an offset vector pointing to corresponding object centers. Using those outputs, we present an efficient and simple post-processing that assigns each object pixel to its best matching object detection, resulting in an instance segmentation obtained at real-time speeds. We propose a novel single shot object detection network named Detection with Enriched Semantics (DES). Our motivation is to enrich the semantics of object detection features within a typical deep detector, by a semantic segmentation branch and a location-agnostic module. The segmentation branch is supervised by weak segmentation ground-truth, i.e., no extra annotation is required. In conjunction with that, we employ a location-agnostic module which learns relationship between channels and object classes in a self-supervised manner. Comprehensive experimental results on both PASCAL VOC and MS COCO detection datasets demonstrate the effectiveness of the proposed method. In particular, with a VGG16 based DES, we achieve an mAP of 81.6 on VOC2007 test and an mmAP of 32.8 on COCO test-dev with an inference speed of 36.7 milliseconds per image on a Titan X Pascal GPU. With a lower resolution version, we achieve an mAP of 79.5 on VOC2007 with an inference speed of 14.7 milliseconds per image. To understand driving environments effectively, it is important to achieve accurate detection and classification of objects detected by sensor-based intelligent vehicle systems, which are significantly important tasks. Object detection is performed for the localization of objects, whereas object classification recognizes object classes from detected object regions. For accurate object detection and classification, fusing multiple sensor information into a key component of the representation and perception processes is necessary. In this paper, we propose a new object-detection and classification method using decision-level fusion. We fuse the classification outputs from independent unary classifiers, such as 3D point clouds and image data using a convolutional neural network (CNN). The unary classifiers for the two sensors are the CNN with five layers, which use more than two pre-trained convolutional layers to consider local to global features as data representation. To represent data using convolutional layers, we apply region of interest (ROI) pooling to the outputs of each layer on the object candidate regions generated using object proposal generation to realize color flattening and semantic grouping for charge-coupled device and Light Detection And Ranging (LiDAR) sensors. We evaluate our proposed method on a KITTI benchmark dataset to detect and classify three object classes: cars, pedestrians and cyclists. The evaluation results show that the proposed method achieves better performance than the previous methods. Our proposed method extracted approximately 500 proposals on a 1226 × 370 image, whereas the original selective search method extracted approximately 10 6 × n proposals. We obtained classification performance with 77.72 mean average precision over the entirety of the classes in the moderate detection level of the KITTI benchmark dataset.