Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> Generative probability models such as hidden Markov models provide a principled way of treating missing information and dealing with variable length sequences. On the other hand, discriminative methods such as support vector machines enable us to construct flexible decision boundaries and often result in classification performance superior to that of the model based approaches. An ideal classifier should combine these two complementary approaches. In this paper, we develop a natural way of achieving this combination by deriving kernel functions for use in discriminative methods such as support vector machines from generative probability models. We provide a theoretical justification for this combination as well as demonstrate a substantial improvement in the classification performance in the context of DNA and protein sequence analysis. <s> BIB001 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> In k-means clustering, we are given a set of n data points in d-dimensional space R/sup d/ and an integer k and the problem is to determine a set of k points in Rd, called centers, so as to minimize the mean squared distance from each data point to its nearest center. A popular heuristic for k-means clustering is Lloyd's (1982) algorithm. We present a simple and efficient implementation of Lloyd's k-means clustering algorithm, which we call the filtering algorithm. This algorithm is easy to implement, requiring a kd-tree as the only major data structure. We establish the practical efficiency of the filtering algorithm in two ways. First, we present a data-sensitive analysis of the algorithm's running time, which shows that the algorithm runs faster as the separation between clusters increases. Second, we present a number of empirical studies both on synthetically generated data and on real data sets from applications in color quantization, data compression, and image segmentation. <s> BIB002 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> We describe an approach to object and scene retrieval which searches for and localizes all the occurrences of a user outlined object in a video. The object is represented by a set of viewpoint invariant region descriptors so that recognition can proceed successfully despite changes in viewpoint, illumination and partial occlusion. The temporal continuity of the video within a shot is used to track the regions in order to reject unstable regions and reduce the effects of noise in the descriptors. The analogy with text retrieval is in the implementation where matches on descriptors are pre-computed (using vector quantization), and inverted file systems and document rankings are used. The result is that retrieved is immediate, returning a ranked list of key frames/shots in the manner of Google. The method is illustrated for matching in two full length feature films. <s> BIB003 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> In this paper, we present a large-scale object retrieval system. The user supplies a query object by selecting a region of a query image, and the system returns a ranked list of images that contain the same object, retrieved from a large corpus. We demonstrate the scalability and performance of our system on a dataset of over 1 million images crawled from the photo-sharing site, Flickr [3], using Oxford landmarks as queries. Building an image-feature vocabulary is a major time and performance bottleneck, due to the size of our dataset. To address this problem we compare different scalable methods for building a vocabulary and introduce a novel quantization method based on randomized trees which we show outperforms the current state-of-the-art on an extensive ground-truth. Our experiments show that the quantization has a major effect on retrieval quality. To further improve query performance, we add an efficient spatial verification stage to re-rank the results returned from our bag-of-words model and show that this consistently improves search quality, though by less of a margin when the visual vocabulary is large. We view this work as a promising step towards much larger, "web-scale " image corpora. <s> BIB004 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> Within the field of pattern classification, the Fisher kernel is a powerful framework which combines the strengths of generative and discriminative approaches. The idea is to characterize a signal with a gradient vector derived from a generative probability model and to subsequently feed this representation to a discriminative classifier. We propose to apply this framework to image categorization where the input signals are images and where the underlying generative model is a visual vocabulary: a Gaussian mixture model which approximates the distribution of low-level features in images. We show that Fisher kernels can actually be understood as an extension of the popular bag-of-visterms. Our approach demonstrates excellent performance on two challenging databases: an in-house database of 19 object/scene categories and the recently released VOC 2006 database. It is also very practical: it has low computational needs both at training and test time and vocabularies trained on one set of categories can be applied to another set without any significant loss in performance. <s> BIB005 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> This paper describes a probabilistic approach to the problem of recognizing places based on their appearance. The system we present is not limited to localization, but can determine that a new observation comes from a previously unseen place, and so augment its map. Effectively this is a SLAM system in the space of appearance. Our probabilistic approach allows us to explicitly account for perceptual aliasing in the environment—identical but indistinctive observations receive a low probability of having come from the same place. We achieve this by learning a generative model of place appearance. By partitioning the learning problem into two parts, new place models can be learned online from only a single observation of a place. The algorithm complexity is linear in the number of places in the map, and is particularly suitable for online loop closure detection in mobile robotics. <s> BIB006 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> Continuous, real-time mapping of an environment using a camera requires a constanttime estimation engine. This rules out optimal global solving such as bundle adjustment. In this article, we investigate the precision that can be achieved with only local estimation of motion and structure provided by a stereo pair. We introduce a simple but novel representation of the environment in terms of a sequence of relative locations. We demonstrate precise local mapping and easy navigation using the relative map, and importantly show that this can be done without requiring a global minimisation after loop closure. We discuss some of the issues that arise from using a relative representation, and evaluate our system on long sequences processed at a constant 30-45 Hz, obtaining precisions down to a few metres over distances of a few kilometres. <s> BIB007 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> We address the problem of image search on a very large scale, where three constraints have to be considered jointly: the accuracy of the search, its efficiency, and the memory usage of the representation. We first propose a simple yet efficient way of aggregating local image descriptors into a vector of limited dimension, which can be viewed as a simplification of the Fisher kernel representation. We then show how to jointly optimize the dimension reduction and the indexing algorithm, so that it best preserves the quality of vector comparison. The evaluation shows that our approach significantly outperforms the state of the art: the search accuracy is comparable to the bag-of-features approach for an image representation that fits in 20 bytes. Searching a 10 million image dataset takes about 50ms. <s> BIB008 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB009 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> Recent years have witnessed a surge of interest in graph-based transductive image classification. Existing simple graph-based transductive learning methods only model the pairwise relationship of images, however, and they are sensitive to the radius parameter used in similarity calculation. Hypergraph learning has been investigated to solve both difficulties. It models the high-order relationship of samples by using a hyperedge to link multiple samples. Nevertheless, the existing hypergraph learning methods face two problems, i.e., how to generate hyperedges and how to handle a large set of hyperedges. This paper proposes an adaptive hypergraph learning method for transductive image classification. In our method, we generate hyperedges by linking images and their nearest neighbors. By varying the size of the neighborhood, we are able to generate a set of hyperedges for each image and its visual neighbors. Our method simultaneously learns the labels of unlabeled images and the weights of hyperedges. In this way, we can automatically modulate the effects of different hyperedges. Thorough empirical studies show the effectiveness of our approach when compared with representative baselines. <s> BIB010 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> This paper is about long-term navigation in environments whose appearance changes over time, suddenly or gradually. We describe, implement and validate an approach which allows us to incrementally learn a model whose complexity varies naturally in accordance with variation of scene appearance. It allows us to leverage the state of the art in pose estimation to build over many runs, a world model of sufficient richness to allow simple localisation despite a large variation in conditions. As our robot repeatedly traverses its workspace, it accumulates distinct visual experiences that in concert, implicitly represent the scene variation: each experience captures a visual mode. When operating in a previously visited area, we continually try to localise in these previous experiences while simultaneously running an independent vision-based pose estimation system. Failure to localise in a sufficient number of prior experiences indicates an insufficient model of the workspace and instigates the laying down of the live image sequence as a new distinct experience. In this way, over time we can capture the typical time-varying appearance of an environment and the number of experiences required tends to a constant. Although we focus on vision as a primary sensor throughout, the ideas we present here are equally applicable to other sensor modalities. We demonstrate our approach working on a road vehicle operating over a 3-month period at different times of day, in different weather and lighting conditions. We present extensive results analysing different aspects of the system and approach, in total processing over 136,000 frames captured from 37 km of driving. <s> BIB011 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> Scene recognition is one of the hallmark tasks of computer vision, allowing definition of a context for object recognition. Whereas the tremendous recent progress in object recognition tasks is due to the availability of large datasets like ImageNet and the rise of Convolutional Neural Networks (CNNs) for learning high-level features, performance at scene recognition has not attained the same level of success. This may be because current deep features trained from ImageNet are not competitive enough for such tasks. Here, we introduce a new scene-centric database called Places with over 7 million labeled pictures of scenes. We propose new methods to compare the density and diversity of image datasets and show that Places is as dense as other scene datasets and has more diversity. Using CNN, we learn deep features for scene recognition tasks, and establish new state-of-the-art results on several scene-centric datasets. A visualization of the CNN layers' responses allows us to show differences in the internal representations of object-centric and scene-centric networks. <s> BIB012 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> Recent results indicate that the generic descriptors extracted from the convolutional neural networks are very powerful. This paper adds to the mounting evidence that this is indeed the case. We report on a series of experiments conducted for different recognition tasks using the publicly available code and model of the OverFeat network which was trained to perform object classification on ILSVRC13. We use features extracted from the OverFeat network as a generic image representation to tackle the diverse range of recognition tasks of object image classification, scene recognition, fine grained recognition, attribute detection and image retrieval applied to a diverse set of datasets. We selected these tasks and datasets as they gradually move further away from the original task and data the OverFeat network was trained to solve. Astonishingly, we report consistent superior results compared to the highly tuned state-of-the-art systems in all the visual classification tasks on various datasets. For instance retrieval it consistently outperforms low memory footprint methods except for sculptures dataset. The results are achieved using a linear SVM classifier (or L2 distance in case of retrieval) applied to a feature representation of size 4096 extracted from a layer in the net. The representations are further modified using simple augmentation techniques e.g. jittering. The results strongly suggest that features obtained from deep learning with convolutional nets should be the primary candidate in most visual recognition tasks. <s> BIB013 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> The inconsistency between textual features and visual contents can cause poor image search results. To solve this problem, click features, which are more reliable than textual information in justifying the relevance between a query and clicked images, are adopted in image ranking model. However, the existing ranking model cannot integrate visual features, which are efficient in refining the click-based search results. In this paper, we propose a novel ranking model based on the learning to rank framework. Visual features and click features are simultaneously utilized to obtain the ranking model. Specifically, the proposed approach is based on large margin structured output learning and the visual consistency is integrated with the click features through a hypergraph regularizer term. In accordance with the fast alternating linearization method, we design a novel algorithm to optimize the objective function. This algorithm alternately minimizes two different approximations of the original objective function by keeping one function unchanged and linearizing the other. We conduct experiments on a large-scale dataset collected from the Microsoft Bing image search engine, and the results demonstrate that the proposed learning to rank models based on visual features and user clicks outperforms state-of-the-art algorithms. <s> BIB014 </s> Place recognition: An Overview of Vision Perspective <s> INTRODUCTION <s> We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection. <s> BIB015
Place recognition has attracted a significant amount of attention in computer vision and robotics communities, as evidenced by related citations and a number of workshops dedicated to improve long-term robot navigation and autonomy. It has a number of applications ranging from autonomous driving, robot navigation to augmented reality, geo-localizing archival imagery. The process of identifying the location of a given image by querying the locations of images belonging to the same place in a large geotagged database, usually known as place recognition, is still an open problem. One major characteristic that separates place recognition from other visual recognition tasks is that place recognition has to solve condition invariant recognition to a degree that many other fields haven't. How can we robustly identify the same real-world place undergoing major changes in appearance, e.g., illumination variation (Figure 1 ), change of seasons (Figure 1 ) or weather, structural modifications over time, and viewpoint change. To be clear, above changes in appearance are summarized as conditional variations, but excludes the viewpoint change. Moreover, how can we distinguish true images from similarly looking images without supervision? Since collecting geotagged datasets is time-consuming and labor intensive, and situations like indoor places do not have GPS information necessarily. Place recognition task has been traditionally cast as an image retrieval task BIB014 where image representations for places are essential. The fundamental scientific question is that what is the appropriate representation of a place that is informative enough to recognize real-world places, yet compact enough to satisfy the real-time processing requirement on a terminal, such as mobile phone and a robot. At early stages, place recognition was dominated by sophisticated local-invariant feature extractors such as SIFT Lowe (2004) and SURF Bay et al. (2006) , hand-crafted global image descriptors such as GIST Oliva and Torralba (2001; , and bag-of-visual-words BIB004 ; BIB003 approach. These traditional feature extraction techniques have led to a great step towards the ultimate outcome. on many category-level recognition tasks such as object classification BIB009 ; Simonyan and Zisserman (2014); BIB015 , scene recognition BIB012 ; ?, image classification BIB010 . Principle ideas of CNNs can date back to 1980s, and the major two reasons why CNNs are so successful in computer vision are the advances of GPU-based computation power and data volume respectively. Recent studies show that general features extracted by CNNs can be transferable Sharif BIB013 and generalized well to other visual recognition tasks. Semantic gap is a well-known problem in place recognition, where different semantics of places may share common low-level features extracted by SIFT, e.g., colors, textures. Convolutional Neural Networks may bridge this semantic gap by treating an image as an high-level feature vector, extracted through deep stacked layers. This paper provides an overview of both traditional and deep learning based descriptive techniques widely applied to place recognition task, which by no means is exhaustive. The remainder of this paper is organized as follows : section 1 gives an introduction of place recognition literature. Section 2 talks about local and global image descriptors widely applied to place recognition. Section 3 presents a brief view of convolutional neural networks and corresponding techniques used in place recognition. Section 4 discusses the future work in place recognition. (2004) and SURF Bay et al. (2006) have been widely applied to visual localization and place recognition tasks. They achieve an outstanding performance on viewpoint invariance. SIFT and SURF describes the local appearance of individual patches or key points within an image, for an image, T represents the set of local invariant features, in which x i ∈ R d is the i th local feature. The length of X depends on the number of key points of the image. These local invariant features are then usually aggregated into a compact single vector for the entire image, using techniques such as bag-of-visual-words BIB004 ; BIB003 , VLAD Jégou et al. (2010) and Fisher kernel BIB001 ; BIB005 . Since local feature extraction consists of two phases : detection and description, a number of variations and extensions of techniques for the two phases are developed. For example, BIB007 used FAST Rosten and Drummond (2006) to detect the interested patches of an image, which were then described by SIFT. BIB011 used FAST as well during detection phase, whereas they use BRIEF Calonder et al. (2012) to describe the key points instead of SIFT. State-of-the-art visual simultaneous localization and mapping (SLAM) systems such as FAB-MAP BIB006 used bag-of-visual-words to construct the final image representation. Each image may contain hundreds of local features, which is impractical in large-scale and real-time processing place recognition task, moreover, they require an enormous amount of memory to store the high dimensional features. The bag-of-visual-words mimics the bag-of-words technique used in efficient text retrieval field. It typically needs to form a codebook C = [c 1 , c 2 , ..., c k ] with K visual words, each visual word c i ∈ R d is a centroid of a cluster, usually gained by k-means BIB002 . Then each local invariant feature x i is assigned to its nearest cluster centroid, i.e., the visual word. A histogram vector of k dimension containing the frequency of each visual word being assigned can be formed this way. The bag-of-visual-words model and local image descriptors generally ignore the geometric structure of the image, that is, different orders of local invariant features in X will not impact on the histogram vector, thus the resulting image representation is viewpoint invariant. There are several variations on how to normalize the histogram vector, a common choice is L 2 normalization. Components of the vector are then weighted by inverse document frequency (idf). However, the codebook is dataset dependent and needs to be retrained if a robot moves into a new region it has never seen before. The lack of structural information of the image can also weaken the performance of the model. (2007) is a powerful tool in pattern classification combining the strengths of generative models and discriminative classifiers. It defines a generative probability model p, which is the probability density function with parameter λ. Then one can characterize the set of local invariant features T with the following gradient vector : where the gradient of the log-likelihood describes the direction in which parameters should be modified to best fit the observed data intuitively. It transforms a variable length sample X into a fixed length vector whose size is only dependent on the number of parameters in the model. BIB005 applied Fisher kernel in the context of image classification with a Gaussian Mixture Model to model the visual words. In comparison with the bag-of-visual-words representation, they obtain a (2d + 1) * k − 1 dimensional image representation of a local invariant feature set, while k dimensional image representation using bag-of-visual-words. Thus, fisher kernel can provide richer information under the circumstance that their size of codebook is equal, or, fewer visual words are required by this more sophisticated representation. BIB008 propose a new local aggregation method called Vectors of locally aggregated descriptors (VLAD), which becomes state-of-the-art technique compared to bag-of-visual-words and fisher vector. The final representation is computed as follows: The VLAD vector is represented by v i,j where the indices i = 1...k and j = 1...d respectively index the i th visual word and the j th component of local invariant feature. The vector is subsequently L 2 normalized. We can see that the final representation stores the sum of all the residuals between local invariant features and its nearest visual word. One excellent property of VLAD vector is that it's relatively sparse and very structured, BIB008 show that a principle component analysis is likely to capture this structure for dimensionality reduction without much degradation of representation. They obtain a comparable search quality to bag-of-visual-words and Fisher kernel with at least an order of magnitude less memory.