Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Object localization <s> An object recognition system has been developed that uses a new class of local image features. The features are invariant to image scaling, translation, and rotation, and partially invariant to illumination changes and affine or 3D projection. These features share similar properties with neurons in inferior temporal cortex that are used for object recognition in primate vision. Features are efficiently detected through a staged filtering approach that identifies stable points in scale space. Image keys are created that allow for local geometric deformations by representing blurred image gradients in multiple orientation planes and at multiple scales. The keys are used as input to a nearest neighbor indexing method that identifies candidate object matches. Final verification of each match is achieved by finding a low residual least squares solution for the unknown model parameters. Experimental results show that robust object recognition can be achieved in cluttered partially occluded images with a computation time of under 2 seconds. <s> BIB001 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Object localization <s> Object detection performance, as measured on the canonical PASCAL VOC dataset, has plateaued in the last few years. The best-performing methods are complex ensemble systems that typically combine multiple low-level image features with high-level context. In this paper, we propose a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012 -- achieving a mAP of 53.3%. Our approach combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost. Since we combine region proposals with CNNs, we call our method R-CNN: Regions with CNN features. We also present experiments that provide insight into what the network learns, revealing a rich hierarchy of image features. Source code for the complete system is available at http://www.cs.berkeley.edu/~rbg/rcnn. <s> BIB002 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Object localization <s> We introduce a novel method for 3D object detection and pose estimation from color images only. We first use segmentation to detect the objects of interest in 2D even in presence of partial occlusions and cluttered background. By contrast with recent patch-based methods, we rely on a "holistic" approach: We apply to the detected objects a Convolutional Neural Network (CNN) trained to predict their 3D poses in the form of 2D projections of the corners of their 3D bounding boxes. This, however, is not sufficient for handling objects from the recent T-LESS dataset: These objects exhibit an axis of rotational symmetry, and the similarity of two images of such an object under two different poses makes training the CNN challenging. We solve this problem by restricting the range of poses used for training, and by introducing a classifier to identify the range of a pose at run-time before estimating it. We also use an optional additional step that refines the predicted poses. We improve the state-of-the-art on the LINEMOD dataset from 73.7% to 89.3% of correctly registered RGB frames. We are also the first to report results on the Occlusion dataset using color images only. We obtain 54% of frames passing the Pose 6D criterion on average on several sequences of the T-LESS dataset, compared to the 67% of the state-of-the-art on the same sequences which uses both color and depth. The full approach is also scalable, as a single network can be trained for multiple objects simultaneously. <s> BIB003 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Object localization <s> This paper develops an enhanced teaching interface tested on both a Baxter robot and a KUKA iiwa robot. Movements are collected from a human demonstrator by using a Kinect v2 sensor, and then the data is sent to a remote PC for the teleoperation with Baxter. Meanwhile, data is saved locally for the playback process of the Baxter. The dynamic movement primitive (DMP) is used to model and generalize the movements. In order to learn from multiple demonstrations accurately, dynamic time warping (DTW), is used to pretreat the data recorded by the robot platform and Gaussian mixture model (GMM), aiming to generate multiple patterns after the teaching process, are employed for the calculation of the DMP. Then the Gaussian mixture regression (GMR) algorithm is applied to generate a synthesized trajectory with smaller position errors in 3D space. This proposed approach is tested by performing two tasks on a KUKA iiwa and a Baxter robot. <s> BIB004 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Object localization <s> Object detection and classification in 3D is a key task in Automated Driving (AD). LiDAR sensors are employed to provide the 3D point cloud reconstruction of the surrounding environment, while the task of 3D object bounding box detection in real time remains a strong algorithmic challenge. In this paper, we build on the success of the one-shot regression meta-architecture in the 2D perspective image space and extend it to generate oriented 3D object bounding boxes from LiDAR point cloud. Our main contribution is in extending the loss function of YOLO v2 to include the yaw angle, the 3D box center in Cartesian coordinates and the height of the box as a direct regression problem. This formulation enables real-time performance, which is essential for automated driving. Our results are showing promising figures on KITTI benchmark, achieving real-time performance (40 fps) on Titan X GPU. <s> BIB005 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Object localization <s> Tool manipulation is vital for facilitating robots to complete challenging task goals. It requires reasoning about the desired effect of the task and thus properly grasping and manipulating the tool to achieve the task. Task-agnostic grasping optimizes for grasp robustness while ignoring crucial task-specific constraints. In this paper, we propose the Task-Oriented Grasping Network (TOG-Net) to jointly optimize both task-oriented grasping of a tool and the manipulation policy for that tool. The training process of the model is based on large-scale simulated self-supervision with procedurally generated tool objects. We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering. Our model achieves overall 71.1% task success rate for sweeping and 80.0% task success rate for hammering. Supplementary material is available at: this http URL <s> BIB006
Most grasping approaches require the computation of the target object's location in the input image data first. This involves object detection and segmentation techniques. Object detection provides the rectangular bounding box of the target object, and object segmentation provides the precise boundary of the target object. The latter provides more accurate descriptions of the object area, while its computation is more time consuming. The representative works of the two methods are shown in Table 1 . BIB004 Object detection Typical functional flow-chart of 2D object detection is illustrated in Fig. 4 . Traditional 2D object detection methods rely on template matching, which utilized manually designed descriptors, such as SIFT BIB001 , SURF BIB005 , Bag of Words BIB006 and so on. Researchers trained classifiers, such as neural networks, support vector machine or Adaboost, according to the descriptors to conduct object detection. Although the descriptors have been widely used in various vision-related tasks, deep learning-based methods have become popular in recent years since the proposal of deep convolutional neural network(DCNN)  . These methods can be further divided into two-stage methods and one-stage methods. Two-stage methods include the pre-processing for region proposal, making the overall pipeline two stages which are region proposals generation and ranking of the best. Onestage methods utilize a unified pipeline to output the detection results directly and skip the separate proposal detection. Representative works of two-stage methods include RCNN BIB002 , Mask RCNN  and one-stage methods include YOLO BIB003 , SSD , etc. The YOLO detection system is illustrated in Fig. 5 . Detailed review of these works please refer to a recent survey . Typical functional flow-chart of 3D object detection is illustrated in Fig. 6 . Except the RGB image data, the depth image can also be used to conduct the detection task by mapping to 3D point cloud and using 3D local shape descriptors, such as FPFH  , SHOT [Salti et al., 2014] , etc. Various deep learning-based 3D object detection methods [Simon et al., 2018; BIB004 Song and Xiao, 2014;  on 3D data are also proposed in recent years. The framework of the Complex-YOLO method is illustrated in Fig. 7 .