We propose a novel method for predicting image labels by fusing image content descriptors with the social media context of each image. An image uploaded to a social media site such as Flickr often has meaningful, associated information, such as comments and other images the user has uploaded, that is complementary to pixel content and helpful in predicting labels. Prediction challenges such as ImageNet imagenet_cvpr09 and MSCOCO LinMBHPRDZ:ECCV14 use only pixels, while other methods make predictions purely from social media context McAuleyECCV12 . Our method is based on a novel fully connected Conditional Random Field (CRF) framework, where each node is an image, and consists of two deep Convolutional Neural Networks (CNN) and one Recurrent Neural Network (RNN) that model both textual and visual node image information. The edge weights of the CRF graph represent textual similarity and link-based metadata such as user sets and image groups. We model the CRF as an RNN for both learning and inference, and incorporate the weighted ranking loss and cross entropy loss into the CRF parameter optimization to handle the training data imbalance issue. Our proposed approach is evaluated on the MIR-9K dataset and experimentally outperforms current state-of-the-art approaches.
Social media context for labeling . A set of tags associated with each image is commonly used in multimodal classification settings. Guillamumin @cite_23 explored the relationship between tags and manual annotations to recover annotations using a combination of tags and image content. Lindstaedt @cite_27 and Sigurbjornsson @cite_20 studied the problem of recommending tags that were obtained from similar images and similar users. Sawant @cite_7 and Stone @cite_10 investigates friendship information between users for tag recommendation in social networks. EXIF and GPS are two commonly used sources of metadata that come directly from the camera @cite_6 @cite_14 @cite_28 @cite_13 . Such metadata can be used to help determine who captured the photo and where, and also provide informative signals for image labeling tasks. Our method differs from all these and also @cite_11 in that we use a much larger range of social media information, including free-form text as well as links, with deep learning based pixel descriptors incorporated into our novel deep learning fully connected CRF framework.
@cite_13 @cite_14 @cite_7 @cite_28 @cite_6 @cite_27 @cite_23 @cite_10 @cite_20 @cite_11
2062258476 2294432969 2057508887 2029731618
Social image tagging is becoming increasingly popular with the development of social website, where images are annotated with arbitrary keywords called tags. Most of present image tagging approaches are mainly based on the visual similarity or mapping between visual feature and tags. However, in the social media environment, images are always associated with multi-type of object information (i.e., visual content, tags, and user contact information) which makes this task more challenging. In this paper, we propose to fuse multi-type of information to tag social image. Specifically, we model social image tagging as a ''ranking and reinforcement'' problem, and a novel graph-based reinforcement algorithm for interrelated multi-type objects is proposed. When a user issue a tagging request for a query image, a candidate tag set is derived and a set of friends of the query user is selected. Then a graph which contains three types of objects (i.e., visual features of the query image, candidate tags, and friend users) is constructed, and each type of objects are initially ranked based on their weight and intra-relation. Finally, candidate tags are re-ranked by our graph-based reinforcement algorithm which takes into consideration both inter-relation with visual features and friend users, and the top ranked tags are saved. Experiments on real-life dataset demonstrate that our algorithm significantly outperforms state-of-the-art algorithms. Our approach locates the temporal positions of tags in videos at the keyframe level.We deal with a scenario in which there is no pre-defined set of tags.We report experiments about the use of different web sources (Flickr, Google, Bing).We show state-of-the-art results on DUT-WEBV, a large dataset of YouTube videos.We show results in a real-world scenario to perform open vocabulary tag annotation. Tagging of visual content is becoming more and more widespread as web-based services and social networks have popularized tagging functionalities among their users. These user-generated tags are used to ease browsing and exploration of media collections, e.g.?using tag clouds, or to retrieve multimedia content. However, not all media are equally tagged by users. Using the current systems is easy to tag a single photo, and even tagging a part of a photo, like a face, has become common in sites like Flickr and Facebook. On the other hand, tagging a video sequence is more complicated and time consuming, so that users just tag the overall content of a video. In this paper we present a method for automatic video annotation that increases the number of tags originally provided by users, and localizes them temporally, associating tags to keyframes. Our approach exploits collective knowledge embedded in user-generated tags and web sources, and visual similarity of keyframes and images uploaded to social sites like YouTube and Flickr, as well as web sources like Google and Bing. Given a keyframe, our method is able to select "on the fly" from these visual sources the training exemplars that should be the most relevant for this test sample, and proceeds to transfer labels across similar images. Compared to existing video tagging approaches that require training classifiers for each tag, our system has few parameters, is easy to implement and can deal with an open vocabulary scenario. We demonstrate the approach on tag refinement and localization on DUT-WEBV, a large dataset of web videos, and show state-of-the-art results. Image annotation is usually formulated as a multi-label semi-supervised learning problem. Traditional graph-based methods only utilize the data (images) graph induced from image similarities, while ignore the label (semantic terms) graph induced from label correlations of a multi-label image data set. In this paper, we propose a novel Bi-relational Graph (BG) model that comprises both the data graph and the label graph as subgraphs, and connect them by an additional bipartite graph induced from label assignments. By considering each class and its labeled images as a semantic group, we perform random walk on the BG to produce group-to-vertex relevance, including class-to-image and class-to-class relevances. The former can be used to predict labels for unannotated images, while the latter are new class relationships, called as Causal Relationships (CR), which are asymmetric. CR is learned from input data and has better semantic meaning to enhance the label prediction for unannotated images. We apply the proposed approaches to automatic image annotation and semantic image retrieval tasks on four benchmark multi-label image data sets. The superior performance of our approaches compared to state-of-the-art multi-label classification methods demonstrate their effectiveness. We address the task of learning a semantic segmentation from weakly supervised data. Our aim is to devise a system that predicts an object label for each pixel by making use of only image level labels during training – the information whether a certain object is present or not in the image. Such coarse tagging of images is faster and easier to obtain as opposed to the tedious task of pixelwise labeling required in state of the art systems. We cast this task naturally as a multiple instance learning (MIL) problem. We use Semantic Texton Forest (STF) as the basic framework and extend it for the MIL setting. We make use of multitask learning (MTL) to regularize our solution. Here, an external task of geometric context estimation is used to improve on the task of semantic segmentation. We report experimental results on the MSRC21 and the very challenging VOC2007 datasets. On MSRC21 dataset we are able, by using 276 weakly labeled images, to achieve the performance of a supervised STF trained on pixelwise labeled training set of 56 images, which is a significant reduction in supervision needed.