Abstract The task of a visual landmark recognition system is to identify photographed buildings or objects in query photos and to provide the user with relevant information on them. With their increasing coverage of the world’s landmark buildings and objects, Internet photo collections are now being used as a source for building such systems in a fully automatic fashion. This process typically consists of three steps: clustering large amounts of images by the objects they depict; determining object names from user-provided tags; and building a robust, compact, and efficient recognition index. To this date, however, there is little empirical information on how well current approaches for those steps perform in a large-scale open-set mining and recognition task. Furthermore, there is little empirical information on how recognition performance varies for different types of landmark objects and where there is still potential for improvement. With this paper, we intend to fill these gaps. Using a dataset of 500 k images from Paris, we analyze each component of the landmark recognition pipeline in order to answer the following questions: How many and what kinds of objects can be discovered automatically? How can we best use the resulting image clusters to recognize the object in a query? How can the object be efficiently represented in memory for recognition? How reliably can semantic information be extracted? And finally: What are the limiting factors in the resulting pipeline from query to semantics? We evaluate how different choices of methods and parameters for the individual pipeline steps affect overall system performance and examine their effects for different query categories such as buildings, paintings or sculptures.
Several works have addressed this problem at the , by removing redundant images from the index. Li al @cite_13 summarize the input image collection in a set of images by applying k-means clustering based on Gist descriptors, and use only these images to represent a landmark in retrieval. Gammeter al @cite_27 identify sets of very similar images using complete-link hierarchical agglomerative clustering and replace them by just one image. This step yields a slight compression of the index without loss in performance. Instead of performing clustering Yang al @cite_11 only determine a set of canonical views by applying PageRank on the matching graph of the image collection. They then discard all other views and match the query only against the canonical views.
@cite_27 @cite_13 @cite_11
1826823045 2058650633 2153697991
Most of the recent work on image-based object recognition and 3D reconstruction has focused on improving the underlying algorithms. In this paper we present a method to automatically improve the quality of the reference database, which, as we will show, also affects recognition and reconstruction performances significantly. Starting out from a reference database of clustered images we expand small clusters. This is done by exploiting cross-media information, which allows for crawling of additional images. For large clusters redundant information is removed by scene analysis. We show how these techniques make object recognition and 3D reconstruction both more efficient and more precise - we observed up to 14.8 improvement for the recognition task. Furthermore, the methods are completely data-driven and fully automatic. This article presents an approach for modeling landmarks based on large-scale, heavily contaminated image collections gathered from the Internet. Our system efficiently combines 2D appearance and 3D geometric constraints to extract scene summaries and construct 3D models. In the first stage of processing, images are clustered based on low-dimensional global appearance descriptors, and the clusters are refined using 3D geometric constraints. Each valid cluster is represented by a single iconic view, and the geometric relationships between iconic views are captured by an iconic scene graph. Using structure from motion techniques, the system then registers the iconic images to efficiently produce 3D models of the different aspects of the landmark. To improve coverage of the scene, these 3D models are subsequently extended using additional, non-iconic views. We also demonstrate the use of iconic images for recognition and browsing. Our experimental results demonstrate the ability to process datasets containing up to 46,000 images in less than 20 hours, using a single commodity PC equipped with a graphics card. This is a significant advance towards Internet-scale operation. We study the problem of place recognition. Given a photo, we estimate its location by scene matching to a large database of internet photos of known locations. Traditional strategies, which involve a linear scan of the database to find matching scenes, fail to scale. On the other hand, internet photos contain a massive amount of noise and redundancy, which is of little help for place recognition. By exploiting the scene distribution of photos, we summarize the database by a set of canonical views. The set of canonical views eliminates the noise and redundancy in internet photos, and provides a compact representation for the database. By restricting scene matching to the set of canonical views, we observe a good tradeoff between efficiency and recall: the average processing time for a query photo is reduced by 97 , while the recall rate for place recognition remains at 75 .