We propose an entirely data-driven approach to estimating the 3D pose of a hand given a depth image. We show that we can correct the mistakes made by a Convolutional Neural Network trained to predict an estimate of the 3D pose by using a feedback loop. The components of this feedback loop are also Deep Networks, optimized using training data. They remove the need for fitting a 3D model to the input data, which requires both a carefully designed fitting function and algorithm. We show that our approach outperforms state-of-the-art methods, and is efficient as our implementation runs at over 400 fps on a single GPU.
Different modalities were proposed for the similarity function, which are coupled to the used model. The modalities include, , depth values @cite_14 @cite_37 @cite_34 , salient points, edges, color @cite_12 , or combinations of these @cite_23 @cite_26 @cite_29 @cite_31 .
@cite_37 @cite_14 @cite_26 @cite_29 @cite_23 @cite_31 @cite_34 @cite_12
2112553126 2273249530 2022398331 2885775891 2793381735 2077311337 2098894541 2131151781
In this paper, we introduce a method to detect co-saliency from an image pair that may have some objects in common. The co-saliency is modeled as a linear combination of the single-image saliency map (SISM) and the multi-image saliency map (MISM). The first term is designed to describe the local attention, which is computed by using three saliency detection techniques available in literature. To compute the MISM, a co-multilayer graph is constructed by dividing the image pair into a spatial pyramid representation. Each node in the graph is described by two types of visual descriptors, which are extracted from a representation of some aspects of local appearance, e.g., color and texture properties. In order to evaluate the similarity between two nodes, we employ a normalized single-pair SimRank algorithm to compute the similarity score. Experimental evaluation on a number of image pairs demonstrates the good performance of the proposed method on the co-saliency detection task. Recent studies have demonstrated that biomarkers from multiple modalities contain complementary information for the diagnosis of Alzheimer's disease AD and its prodromal stage mild cognitive impairment MCI. In order to fuse data from multiple modalities, most previous approaches calculate a mixed kernel or a similarity matrix by linearly combining kernels or similarities from multiple modalities. However, the complementary information from multi-modal data are not necessarily linearly related. In addition, this linear combination is also sensitive to the weights assigned to each modality. In this paper, we propose a nonlinear graph fusion method to efficiently exploit the complementarity in the multi-modal data for the classification of AD. Specifically, a graph is first constructed for each modality individually. Afterwards, a single unified graph is obtained via a nonlinear combination of the graphs in an iterative cross diffusion process. Using the unified graphs, we achieved classification accuracies of 91.8 between AD subjects and normal controls NC, 79.5 between MCI subjects and NC and 60.2 in a three-way classification, which are competitive with state-of-the-art results. The cross-modal feature matching has gained much attention in recent years, which has many practical applications, such as the text-to-image retrieval. The most difficult problem of cross-modal matching is how to eliminate the heterogeneity between modalities. The existing methods (e.g., CCA and PLS) try to learn a common latent subspace, where the heterogeneity between two modalities is minimized so that cross-matching is possible. However, most of these methods require fully paired samples and suffer difficulties when dealing with unpaired data. Besides, utilizing the class label information has been found as a good way to reduce the semantic gap between the low-level image features and high-level document descriptions. Considering this, we propose a novel and effective supervised algorithm, which can also deal with the unpaired data. In the proposed formulation, the basis matrices of different modalities are jointly learned based on the training samples. Moreover, a local group-based priori is proposed in the formulation to make a better use of popular block based features (e.g., HOG and GIST). Extensive experiments are conducted on four public databases: Pascal VOC2007, LabelMe, Wikipedia, and NUS-WIDE. We also evaluated the proposed algorithm with unpaired data. By comparing with existing state-of-the-art algorithms, the results show that the proposed algorithm is more robust and achieves the best performance, which outperforms the second best algorithm by about 5 on both the Pascal VOC2007 and NUS-WIDE databases. We present an approach named JSFusion (Joint Sequence Fusion) that can measure semantic similarity between any pairs of multimodal sequence data (e.g. a video clip and a language sentence). Our multimodal matching network consists of two key components. First, the Joint Semantic Tensor composes a dense pairwise representation of two sequence data into a 3D tensor. Then, the Convolutional Hierarchical Decoder computes their similarity score by discovering hidden hierarchical matches between the two sequence modalities. Both modules leverage hierarchical attention mechanisms that learn to promote well-matched representation patterns while prune out misaligned ones in a bottom-up manner. Although the JSFusion is a universal model to be applicable to any multimodal sequence data, this work focuses on video-language tasks including multimodal retrieval and video QA. We evaluate the JSFusion model in three retrieval and VQA tasks in LSMDC, for which our model achieves the best performance reported so far. We also perform multiple-choice and movie retrieval tasks for the MSR-VTT dataset, on which our approach outperforms many state-of-the-art methods. RGB-D images captured by consumer camera can provide pair-wise color and depth information but depth image usually contains strong noise and large holes. Due to different modalities of RGB-D, the intensity-guided depth enhancement easily causes artifacts and blurred edges. To solve this problem, we propose the multi-scale mutual feature convolutional neural network (MSMF-CNN) to learn essential mutual features of RGB-D. This end-to-end framework has two stages, i.e., mutual feature learning and depth regeneration. Firstly, we design two parallel separated subnetworks as intensity and depth mutual feature generator. Specifically, we design a multi-scale mutual feature generator to enforce the depth mutual feature learning and incorporate various structure characteristic. Secondly, depth transformation sub-network combines two mutual features to recover clean depth image. We tested our model on two well-known datasets in terms of PSNR, visual effect and computing speed. Compared with state-of-art algorithms, our method provides better performance of all three criteria. In this paper, we present a simple and effective topic correlation model (TCM) for cross-modal multimedia retrieval by jointly modeling the text and image components in multimedia documents. In this model, the image component is represented by the bag-of-features model based on local scale-invariant feature transform features, meanwhile the text component is described by a topic distribution learned from a latent topic model. Statistical correlations between these two mid-level features are investigated by mapping them into a semantic space. These cross-modality correlations are used to calculate the conditional probabilities of answers in one modality while given query in the other modality. The model is tested on three cross-modal retrieval benchmark problems including Wikipedia documents in both English and Chinese. Experimental results have demonstrated that the new TCM model achieves the best performance compared to recent state-of-the-art cross-modal retrieval models on the given benchmarks. We investigate how to discover all common visual patterns within two sets of feature points. Common visual patterns generally share similar local features as well as similar spatial layout. In this paper these two types of information are integrated and encoded into the edges of a graph whose nodes represent potential correspondences, and the common visual patterns then correspond to those strongly connected subgraphs. All such strongly connected subgraphs correspond to large local maxima of a quadratic function on simplex, which is an approximate measure of the average intra-cluster affinity score of these subgraphs. We find all large local maxima of this function, thus discover all common visual patterns and recover the correct correspondences, using replicator equation and through a systematic way of initialization. The proposed algorithm possesses two characteristics: 1) robust to outliers, and 2) being able to discover all common visual patterns, no matter the mappings among the common visual patterns are one to one, one to many, or many to many. Extensive experiments on both point sets and real images demonstrate the properties of our proposed algorithm in terms of robustness to outliers, tolerance to large spatial deformations, and simplicity in implementation. We present an approach for the construction of text similarity functions using a parameterized resemblance coefficient in combination with a softened cardinality function called soft cardinality. Our approach provides a consistent and recursive model, varying levels of granularity from sentences to characters. Therefore, our model was used to compare sentences divided into words, and in turn, words divided into q-grams of characters. Experimentally, we observed that a performance correlation function in a space defined by all parameters was relatively smooth and had a single maximum achievable by "hill climbing." Our approach used only surface text information, a stop-word remover, and a stemmer to tackle the semantic text similarity task 6 at SEMEVAL 2012. The proposed method ranked 3rd (average), 5th (normalized correlation), and 15th (aggregated correlation) among 89 systems submitted by 31 teams.