Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Evaluation metrics <s> The Pascal Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection.This paper describes the dataset and evaluation procedure. We review the state-of-the-art in evaluated methods for both classification and detection, analyse whether the methods are statistically different, what they are learning from the images (e.g. the object or its context), and what the methods find easy or confuse. The paper concludes with lessons learnt in the three year history of the challenge, and proposes directions for future improvement and extension. <s> BIB001 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Evaluation metrics <s> Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at this https URL. <s> BIB002
Evaluation metrics about 2D object detection are also summarized in Liu et al. , and here we briefly introduced the mean Average Precision (mAP) BIB001 metric. The AverageP recision (AP) metric is derived from precision and recall, and is usually used to evaluate a specific category. The meanAverageP recision (mAP) BIB001 is averaged over all object categories, which is adopted as the measure of the general performance. The standard outputs of a detector applied to a testing image I are the predicted detections {(b j , c j , p j )} j , indexed by j. Here b denotes the predicted location such as the bounding box, c denotes the category label and the confidence level is p. A  Here area(b ∩ b g ) denotes the intersection of the predicted and ground truth bounding boxes, and area(b ∪ b g ) represents their union. The typical value of ε is 0.5. If IOU is smaller than ε, it is considered as False Positive(FP). The confidence level p is usually compared with a threshold β to determine whether the predicted class label c is accepted. Each detection is either a TP or FP, and the precision P (β) with confidence threshold β can be computed. The Average Precision (AP) can thus be achieved by varying β. In order to estimate the 6D pose of an object in the image, the pose error is measured. The object pose can be represented by a 4 × 4 matrix P = [R, t; 0, 1], where R is a 3 × 3 rotation matrix and t is a 3 × 1 translation vector. Anyway, direct comparison of the variances between rotations can not provide an intuitive visual understanding. Therefore, other evaluation metrics are proposed. A commonly used metric is the Average Distance of Model Points(ADD)  for non-symmetric objects and the average closest point distances(ADD-S) BIB002 for symmetric objects. Given a 3D model M , the ground truth rotation R and translation T , and the estimated rotationR and translationT , we compute the average distance of all model points x from their transformed versions. The visible surface discrepancy(VSD)  is another metric which measures the two rendered distance mapsŜ andS of the object model according to the estimated poseP w.r.t. the ground truth poseP . Given the visibility masks which contain the sets of pixels where the model M is visible in the image I, the VSD error is calculated according to the two distance maps as: where τ is a misalignment tolerance. The pose error e V SD is calculated over the visible part of the model surface and thus the indistinguishable poses are treated as equivalent.