Traditional Bag-of-visual Words (BoWs) model is commonly generated with many steps including local feature extraction, codebook generation, and feature quantization, etc. Those steps are relatively independent with each other and are hard to be jointly optimized. Moreover, the dependency on hand-crafted local feature makes BoWs model not effective in conveying high-level semantics. These issues largely hinder the performance of BoWs model in large-scale image applications. To conquer these issues, we propose an End-to-End BoWs (E @math BoWs) model based on Deep Convolutional Neural Network (DCNN). Our model takes an image as input, then identifies and separates the semantic objects in it, and finally outputs the visual words with high semantic discriminative power. Specifically, our model firstly generates Semantic Feature Maps (SFMs) corresponding to different object categories through convolutional layers, then introduces Bag-of-Words Layers (BoWL) to generate visual words for each individual feature map. We also introduce a novel learning algorithm to reinforce the sparsity of the generated E @math BoWs model, which further ensures the time and memory efficiency. We evaluate the proposed E @math BoWs model on several image search datasets including CIFAR-10, CIFAR-100, MIRFLICKR-25K and NUS-WIDE. Experimental results show that our method achieves promising accuracy and efficiency compared with recent deep learning based retrieval works.
As a fundamental task in multimedia content analysis and computer vision @cite_15 @cite_13 @cite_18 , CBIR aims to search for images similar with the query in an image gallery. Since directly computing similarity between two images with raw image pixels is infeasible, BoWs model is widely used as an image representation for large-scale image retrieval. Over the past decade, various BoWs models @cite_38 @cite_27 @cite_35 @cite_6 have been proposed based on local descriptors, such as SIFT @cite_24 and SURF @cite_32 . Those BoWs models have shown promising performance in large-scale image retrieval. Conventional BoWs models consider an image as a collection of visual words and is generated by many stages, , feature extraction, codebook generation and feature quantization @cite_38 @cite_27 @cite_35 @cite_6 . For instnace, Nister @cite_38 extract SIFT @cite_24 descriptors from MSER regions @cite_21 and then hierarchically quantize SIFT descriptors by the vocabulary tree. As individual visual word cannot depict the spatial cues in images, some works combine visual words with spatial information @cite_23 @cite_36 to make the resulting BoWs model more discriminative to the spatial cues. Some other works aim to generate more effective and discriminative vocabularies @cite_41 @cite_5 .
@cite_38 @cite_18 @cite_35 @cite_36 @cite_41 @cite_21 @cite_32 @cite_6 @cite_24 @cite_27 @cite_23 @cite_5 @cite_15 @cite_13
2342423902 2031332477 2059304719 2364255476
Local descriptors are mostly extracted over the raw intensity values of the image and requires the analysis of patterns in local pixel neighborhoods. The raw images don't provide much local relationship information required for the local descriptors. To resolve this problem, a new method of local image feature description is proposed in this paper based on the patterns extracted from the multiple filtered images. The whole process is structured as follows: First, the images are filtered with the bag of filters (BoF) and then local binary pattern (LBP) is computed over each filtered image and finally concatenated to find out the single BoF-LBP descriptor. The local information enriches in the filtered images improves the discriminative ability of the final descriptor. Content based image retrieval (CBIR) experiments are performed to observe the effectiveness of the proposed approach and compared with the recent state-of-the-art methods. The experiments are conducted over four benchmark databases having natural and texture images such as Corel-1k, Corel-10k, MIT-VisTex and STex-512S. The experimental results confirm that the introduced approach is able to improve the retrieval performance of the CBIR system. In Bag-of-Words (BoW) based image retrieval, the SIFT visual word has a low discriminative power, so false positive matches occur prevalently. Apart from the information loss during quantization, another cause is that the SIFT feature only describes the local gradient distribution. To address this problem, this paper proposes a coupled Multi-Index (c-MI) framework to perform feature fusion at indexing level. Basically, complementary features are coupled into a multi-dimensional inverted index. Each dimension of c-MI corresponds to one kind of feature, and the retrieval process votes for images similar in both SIFT and other feature spaces. Specifically, we exploit the fusion of local color feature into c-MI. While the precision of visual match is greatly enhanced, we adopt Multiple Assignment to improve recall. The joint cooperation of SIFT and color features significantly reduces the impact of false positive matches. Extensive experiments on several benchmark datasets demonstrate that c-MI improves the retrieval accuracy significantly, while consuming only half of the query time compared to the baseline. Importantly, we show that c-MI is well complementary to many prior techniques. Assembling these methods, we have obtained an mAP of 85.8 and N-S score of 3.85 on Holidays and Ukbench datasets, respectively, which compare favorably with the state-of-the-arts. Content-based image retrieval (CBIR) systems require users to query images by their low-level visual content; this not only makes it hard for users to formulate queries, but also can lead to unsatisfied retrieval results. To this end, image annotation was proposed. The aim of image annotation is to automatically assign keywords to images, so image retrieval users are able to query images by keywords. Image annotation can be regarded as the image classification problem: that images are represented by some low-level features and some supervised learning techniques are used to learn the mapping between low-level features and high-level concepts (i.e., class labels). One of the most widely used feature representation methods is bag-of-words (BoW). This paper reviews related works based on the issues of improving and or applying BoW for image annotation. Moreover, many recent works (from 2006 to 2012) are compared in terms of the methodology of BoW feature generation and experimental design. In addition, several different issues in using BoW are discussed, and some important issues for future research are discussed. Abstract With the growing amount of inappropriate content on the Internet, such as pornography, arises the need to detect and filter such material. The reason for this is given by the fact that such content is often prohibited in certain environments (e.g., schools and workplaces) or for certain publics (e.g., children). In recent years, many works have been mainly focused on detecting pornographic images and videos based on visual content, particularly on the detection of skin color. Although these approaches provide good results, they generally have the disadvantage of a high false positive rate since not all images with large areas of skin exposure are necessarily pornographic images, such as people wearing swimsuits or images related to sports. Local feature based approaches with Bag-of-Words models (BoW) have been successfully applied to visual recognition tasks in the context of pornography detection. Even though existing methods provide promising results, they use local feature descriptors that require a high computational processing time yielding high-dimensional vectors. In this work, we propose an approach for pornography detection based on local binary feature extraction and BossaNova image representation, a BoW model extension that preserves more richly the visual information. Moreover, we propose two approaches for video description based on the combination of mid-level representations namely BossaNova Video Descriptor (BNVD) and BoW Video Descriptor (BoW-VD). The proposed techniques are promising, achieving an accuracy of 92.40 , thus reducing the classification error by 16 over the current state-of-the-art local features approach on the Pornography dataset.