Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> 6D Pose Estimation <s> An object recognition system has been developed that uses a new class of local image features. The features are invariant to image scaling, translation, and rotation, and partially invariant to illumination changes and affine or 3D projection. These features share similar properties with neurons in inferior temporal cortex that are used for object recognition in primate vision. Features are efficiently detected through a staged filtering approach that identifies stable points in scale space. Image keys are created that allow for local geometric deformations by representing blurred image gradients in multiple orientation planes and at multiple scales. The keys are used as input to a nearest neighbor indexing method that identifies candidate object matches. Final verification of each match is achieved by finding a low residual least squares solution for the unknown model parameters. Experimental results show that robust object recognition can be achieved in cluttered partially occluded images with a computation time of under 2 seconds. <s> BIB001 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> 6D Pose Estimation <s> The gold standard method for tridimensional reconstruction and camera localization from a set of images is well known to be Bundle Adjustment (BA). Although BA was regarded for years as a costly method restricted to the offline domain, several real time algorithms based on BA flourished in the last decade. However those algorithms were limited to perform SLAM in small scenes or only Visual Odometry. In this work we built on excellent algorithms of the last years to design from scratch a Monocular SLAM system that operates in real time, in small and large, indoor and outdoor environments, with the capability of wide baseline loop closing and relocalization, and including full automatic initialization. Our survival of the fittest approach to select the points and keyframes of the reconstruction generates a compact and trackable map that only grows if the scene content changes, enhancing lifelong operation. We present an exhaustive evaluation in 27 sequences from the most popular datasets achieving unprecedented performance with a typical localization accuracy from 0.2% to 1% of the trajectory dimension in scenes from a desk to several city blocks. We make public a ROS implementation. <s> BIB002 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> 6D Pose Estimation <s> Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at this https URL. <s> BIB003 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> 6D Pose Estimation <s> Object detection and classification in 3D is a key task in Automated Driving (AD). LiDAR sensors are employed to provide the 3D point cloud reconstruction of the surrounding environment, while the task of 3D object bounding box detection in real time remains a strong algorithmic challenge. In this paper, we build on the success of the one-shot regression meta-architecture in the 2D perspective image space and extend it to generate oriented 3D object bounding boxes from LiDAR point cloud. Our main contribution is in extending the loss function of YOLO v2 to include the yaw angle, the 3D box center in Cartesian coordinates and the height of the box as a direct regression problem. This formulation enables real-time performance, which is essential for automated driving. Our results are showing promising figures on KITTI benchmark, achieving real-time performance (40 fps) on Titan X GPU. <s> BIB004 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> 6D Pose Estimation <s> We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. To train our network, we collected over 800,000 grasp attempts over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and hardware. Our experimental evaluation demonstrates that our method achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing. <s> BIB005 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> 6D Pose Estimation <s> We present a new approach to transfer grasp configurations from prior example objects to novel objects. We assume the novel and example objects have the same topology and similar shapes. We perform 3D segmentation on these objects using geometric and semantic shape characteristics. We compute a grasp space for each part of the example object using active learning. We build bijective contact mapping between these model parts and compute the corresponding grasps for novel objects. Finally, we assemble the individual parts and use local replanning to adjust grasp configurations while maintaining its stability and physical constraints. Our approach is general, can handle all kind of objects represented using mesh or point cloud and a variety of robotic hands. <s> BIB006
6D pose estimation plays a pivotal role in many areas such as augmented reality, robotic manipulation, and autonomous driving, et al. It helps a robot to get aware of the position and orientation of the object to grasp. The pose estimation methods can be roughly divided into four kinds, which are based on correspondence, template, voting, and regression respectively, as shown in Table 2 . The first three kinds of methods which rely on the detected or segmented objects will be discussed in this section, and the regression-based methods which accomplish object detection and 6D pose estimation together will be discussed in the next section.  Correspondence-based methods Typical functional flow-chart of correspondence-based object pose estimation methods without object detection is illustrated in Fig. 12 . This kind of method mainly targets on the pose estimation of objects with rich textures for the matching of 2D feature points. Multiple images are firstly rendered by projecting the existing 3D models from various angles. By finding the matchings between 2D feature points on the observed image and the rendered images BIB006 , as shown in Fig. 13 , the 2D pixel to 3D point correspondences are established. The common 2D descriptors such as SIFT BIB001 , SURF BIB004 , ORB BIB002 , etc., are usually utilized for the 2D feature extraction. The 6D pose of the object can be calculated with Perspective-n-Point(PnP) algorithms BIB005 . BIB005 ] present a survey about monocular model-based 3D tracking methods of rigid objects. When the depth image is available, the problem turns into a partial registration problem. The popular 3D descriptors, such as FPFH  , Spin image  , and SHOT  , can be utilized to find correspondences between the partial 3D point cloud and the full object to obtain a rough pose. It is then refined through the iterative closest points(ICP) algorithm  . The influence region diagram for the FPFH descriptor is illustrated in Fig. 14. This kind of methods rely on the geometry of the target object and are widely used in pose estimation of robotic grasping. Wong et al. BIB003 proposed a method which integrated RGB-based object segmentation and depth image-based partial registration to obtain the pose of the target object. They presented a novel metric for scoring model registration quality, and conducted multi-hypothesis registration, which achieved accurate pose estimation with 1cm position error and < 5 • angle error.