1. An apparatus for matching images across from heterogeneous sources, the apparatus comprising: a memory; and at least one processor device, coupled to the memory, operative to: (a) parse each of the images into different spatially connected semantic labeled regions; (b) create a list of potential matches by matching spatial structure-free representations of the images based on two or more of the images having same semantic labeled regions; and (c) prune the list of potential matches created in step (b) by then taking into consideration spatial arrangements of the semantic labeled regions in the images by analyzing spatial relationships of the semantic labeled regions in the images, wherein the images are parsed into the different semantic labeled regions based on a semantic visual content of the images, and the images from the heterogeneous sources are matched based on the semantic visual content of the images, and wherein the images from the heterogeneous sources comprise photographs from different perspectives and from different modalities.