This paper addresses the problem of very large-scale image retrieval, focusing on improving its accuracy and robustness. We target enhanced robustness of search to factors, such as variations in illumination, object appearance and scale, partial occlusions, and cluttered backgrounds—particularly important when a search is performed across very large datasets with significant variability. We propose a novel CNN-based global descriptor, called REMAP, which learns and aggregates a hierarchy of deep features from multiple CNN layers, and is trained end-to-end with a triplet loss. REMAP explicitly learns discriminative features which are mutually supportive and complementary at various semantic levels of visual abstraction. These dense local features are max-pooled spatially at each layer, within multi-scale overlapping regions, before aggregation into a single image-level descriptor. To identify the semantically useful regions and layers for retrieval, we propose to measure the information gain of each region and layer using KL-divergence. Our system effectively learns during training how useful various regions and layers are and weights them accordingly. We show that such relative entropy-guided aggregation outperforms classical CNN-based aggregation controlled by SGD. The entire framework is trained in an end-to-end fashion, outperforming the latest state-of-the-art results. On image retrieval datasets Holidays, Oxford, and MPEG, the REMAP descriptor achieves mAP of 95.5 , 91.5 , and 80.1 , respectively, outperforming any results published to date. REMAP also formed the core of the winning submission to the Google Landmark Retrieval Challenge on Kaggle.
Early approaches typically involve extracting multiple local descriptors (usually hand-crafted), and combining them into a fixed length image-level representation for fast matching. Local descriptors may be scale-invariant and centered on image feature points, such as for SIFT @cite_6 , or extracted on regular, dense grids, possibly at multiple scales independently of the image content @cite_20 . An impressive number of local descriptors have been developed over years, each claiming superiority, making it difficult to select the best one for the job - an attempt at comparative study can be found in @cite_18 . It should be noted that descriptor dimension (for hand-crafted features) is typically between 32 and 192, which is an order or two less than the number of deep features available for each image region.
@cite_18 @cite_20 @cite_6
2528954356 2037031790 2151103935
Scale invariant feature detectors often find stable scales in only a few image pixels. Consequently, methods for feature matching typically choose one of two extreme options: matching a sparse set of scale invariant features, or dense matching using arbitrary scales. In this paper we turn our attention to the overwhelming majority of pixels, those where stable scales are not found by standard techniques. We ask, is scale-selection necessary for these pixels, when dense, scale-invariant matching is required and if so, how can it be achieved? We make the following contributions: (i) We show that features computed over different scales, even in low-contrast areas, can be different; selecting a single scale, arbitrarily or otherwise, may lead to poor matches when the images have different scales. (ii) We show that representing each pixel as a set of SIFTs, extracted at multiple scales, allows for far better matches than single-scale descriptors, but at a computational price. Finally, (iii) we demonstrate that each such set may be accurately represented by a low-dimensional, linear subspace. A subspace-to-point mapping may further be used to produce a novel descriptor representation, the Scale-Less SIFT (SLS), as an alternative to single-scale descriptors. These claims are verified by quantitative and qualitative tests, demonstrating significant improvements over existing methods. Visual search and image retrieval underpin numerous applications, however the task is still challenging predominantly due to the variability of object appearance and ever increasing size of the databases, often exceeding billions of images. Prior art methods rely on aggregation of local scale-invariant descriptors, such as SIFT, via mechanisms including Bag of Visual Words (BoW), Vector of Locally Aggregated Descriptors (VLAD) and Fisher Vectors (FV). However, their performance is still short of what is required. This paper presents a novel method for deriving a compact and distinctive representation of image content called Robust Visual Descriptor with Whitening (RVD-W). It significantly advances the state of the art and delivers world-class performance. In our approach local descriptors are rank-assigned to multiple clusters. Residual vectors are then computed in each cluster, normalized using a direction-preserving normalization function and aggregated based on the neighborhood rank. Importantly, the residual vectors are de-correlated and whitened in each cluster before aggregation, leading to a balanced energy distribution in each dimension and significantly improved performance. We also propose a new post-PCA normalization approach which improves separability between the matching and non-matching global descriptors. This new normalization benefits not only our RVD-W descriptor but also improves existing approaches based on FV and VLAD aggregation. Furthermore, we show that the aggregation framework developed using hand-crafted SIFT features also performs exceptionally well with Convolutional Neural Network (CNN) based features. The RVD-W pipeline outperforms state-of-the-art globa The region descriptors using local intensity ordering patterns have become more popular recent years for image matching due to its enhanced discriminative ability. However, the dimension of these descriptors increases rapidly with the slight increase in the number of local neighbors under consideration and becomes unreasonable for image matching due to time constraint. In this paper, we reduce the dimension of the descriptor and matching time significantly while keeping up the comparable performance by considering the number of neighboring sample points in an interleaved manner. The proposed interleaved order based local descriptor (IOLD) considers the local neighbors of a pixel as a set of interleaved neighbors and constructs the descriptor over each set separately and finally combines them to produce a single pattern. We extract the local ordering pattern to cope up with the illumination effect in an inherent rotation invariant manner. The novelty lies with using multiple neighboring sets in an interleaved fashion. We also explored the local intensity order pattern in a multisupport-region scenario. Results are compared over three challenging and widely adopted image matching data sets with other prominent descriptors under various image transformations. Results based on experiments suggest that the proposed IOLD descriptor outperforms in terms of both improved matching performance and reduced matching time. We also found that the amount of improvement is significant under complex illumination difference while showing more robustness toward noise.