We train a simple Siamese-VGG [1]} network following the modified triplet loss in [2]}. During the training, if the image pairs are not aligned (e.g. randomly rotate the aerial view images), the activation map can be used for orientation estimation. Specifically, we train the Siamese-VGG with randomly rotated aerial images so that the model is approximately rotation-invariant and so is the overall activation map for aerial image. In the example of Fig. REF , the overall activation maps for the original aligned pair are first generated to show the highlighted regions contributing to the similarity score, and both views focus on the road areas in this example. The most activated regions are highly relevant in both views and most of them are similar patterns. When we randomly rotate the aerial image (\(30^{\circ }\)  in this example and is denoted as the ground truth rotation angle in Fig. REF ), the aerial view activation map still focuses on the road area which is relevant to the street view.
<FIGURE>