The motivation is that surface normal can provide strong shape prior of objects. For extracting features from photometrically challenging objects, especially for transparent objects, color and position are not enough, as background and transparent parts can look similar in RGB. However, their orientations represented by the surface normals are different. Introducing the normal into the function can make \(f\left(q_i, q_j\right)\)  reweighted and optimized.
Inspired by (2), we crop local regions centered on each keypoint to let the feature encoder only focus on the respective regions, as our task does not care about depth estimation for the whole map. Besides, we further extend (2) to the grasping width and angle regression. In practice, we achieve this by concatenating RGB normalized crops and normal crops along the feature channel. \(f\left(q_i, q_j\right)\)  can be represented by various existed feature encoder, such as [1]}, [2]}, [3]}. Before sending feature crops into the encoder, we perform RoI Align [4]} on each crop, as some of the keypoints may appear near the boundaries of RGB images, resulting in these crops with different sizes. Moreover, RoI Align allows flexible feature output adjustment. After obtaining the high-dimension feature map, we employ fully connected layers to regress depth \(d\) , grasping width \(w\) , and rotated angle \(\phi \)  for each 2D keypoint.
<FIGURE>