Learning the similarity metrics between arbitrary images is a fundamental problem for a variety of tasks, such as image retrieval [1]}, verification [2]}, [3]}, localization [4]}, video tracking [5]}, etc. Recently the deep Siamese network [6]} based framework has become a standard architecture for metric learning and achieves exciting results on a wide range of applications [7]}. However, there are surprisingly few works conducting visual analysis to explain why the learned similarity of a given image pair is high or low. Specifically, which part contributes the most to the similarity is a straightforward question and the answer can reveal important hidden information about the model as well as the data.
