PoseNet [1]} is the first work to tackle camera relocalization problem by training a ConvNet to predict camera pose from single RGB images in an end-to-end manner. PoseNet is based on the main structure of GoogleNet [2]} to extract visual features,, but removes the last softmax layers. Instead, a fully connected layer was introduced to output a 7 dimensional global pose, consisting of position and orientation vector in 3 and 4 dimensions respectively. However, PoseNet was designed with a naive regression loss function without any consideration for geometry, in which the hyper-parameters inside requires expensive hand-engineering to be tuned. Furthermore, it also suffers from the overfitting problem due to the high dimensionality of the feature embedding and limited training data. Thus various extensions enhance the original pipeline by exploiting LSTM units to reduce the dimensionality [3]}, applying synthetic generation to augment training data [4]}, [5]}, [6]}, replacing backbone with ResNet34 [7]}, modelling pose uncertainty [8]}, [9]} and introducing geometry-aware loss function [10]}. Alternatively, Atloc [11]} associates the features in spatial domain with attention mechanism, which encourages the network to focus on parts of the image that are temporally consistent and robust. Similarly, a prior guided dropout mask is additionally adopted in RVL [12]} to further eliminate the uncertainty caused by dynamic objects. Different such methods only considering spatial connections, VidLoc [13]} incorporates temporal constraints of image sequences to model the temporal connections of input images for visual localization. Moreover, additional motion constraints, including spatial constraints and other sensor constraints from GPS or SLAM systems are exploited in MapNet [14]}, to enforce the motion consistency between predicted poses. Similar motion constraints are also added by jointly optimizing a relocalization network and visual odometry network [15]}, [16]}. However, being application-specific, scene representations learned from localization tasks may ignore some useful features they are not designed for. Out of this, VLocNet++ [17]} and FGSN [18]} additionally exploits the inter-task relationship among learning semantics and regressing poses, achieving impressive results.
