Video learning is an important task in computer vision and has experienced increasing interest over the recent years. Since even a small amount of videos easily comprises several million frames, methods that do not rely on a frame-level annotation are of special importance. In this work, we propose a novel learning algorithm with a Viterbi-based loss that allows for online and incremental learning of weakly annotated video data. We moreover show that explicit context and length modeling leads to huge improvements in video segmentation and labeling tasks andinclude these models into our framework. On several action segmentation benchmarks, we obtain an improvement of up to 10 compared to current state-of-the-art methods.
Kuehne al @cite_5 interpret the task of learning an action segmentation system just given an ordered list of occurring actions as supervision as an instance of a speech recognition problem, where the videos correspond to the audio signal and the action classes correspond to words. They apply a standard HMM-GMM system using a speech recognition toolkit. Building upon this idea, Richard al @cite_28 replace the GMM by a recurrent neural network but still rely on an HMM for a coarse temporal modeling. A similar approach has been proposed by Koller al @cite_35 @cite_2 for sign language recognition, which is a problem closely related to temporal action segmentation, but with the most significant difference that the actual temporal boundaries of the recognized words classes are not relevant. Note that in contrast to our proposed method, @cite_5 @cite_28 @cite_35 @cite_2 use a two-step optimization scheme that does not allow for direct, sequence-wise training.
@cite_28 @cite_5 @cite_35 @cite_2
2283723945 2015551990 2081773958 2104480821
We propose a hierarchical approach to multi-action recognition that performs joint classification and segmentation. Ai¾źgiven video containing several consecutive actions is processed via a sequence of overlapping temporal windows. Each frame in a temporal window is represented through selective low-level spatio-temporal features which efficiently capture relevant local dynamics. Features from each window are represented as a Fisher vector, which captures first and second order statistics. Instead of directly classifying each Fisher vector, it is converted into a vector of class probabilities. The final classification decision for each frame is then obtained by integrating the class probabilities at the frame level, which exploits the overlapping of the temporal windows. Experiments were performed on two datasets: s-KTH ai¾źstitched version of the KTH dataset to simulate multi-actions, and the challenging CMU-MMAC dataset. On s-KTH, the proposed approach achieves an accuracy of 85.0i¾ź , significantly outperforming two recent approaches based on GMMs and HMMs which obtained 78.3i¾ź and 71.2i¾ź , respectively. On CMU-MMAC, the proposed approach achieves an accuracy of 40.9i¾ź , outperforming the GMM and HMM approaches which obtained 33.7i¾ź and 38.4i¾ź , respectively. Furthermore, the proposed system is on average 40 times faster than the GMM based approach. Recently, great progress has been made in sign language recognition. Most approaches are based on the Hidden Markov Model (HMM) with various features, such as motion trajectory. Recognition for sign sentences is obtained from optimal path by Viterbi algorithm, however, some wrong jumps are usually caused by transitional movements between signs. To address the problem, in this paper, we propose an approach consisting of two stages: offline training and online recognition. In the offline training stage, we propose a threshold matrix and rate thresholds. Each element of the threshold matrix describes the minimal probability when a segment belongs to a sign, and rate thresholds are defined as the average probability for signs. So, if certain segment's evaluation is smaller than all the thresholds, it is regarded as a transitional movement and then it should be removed. In the online recognition stage, coarse segmentation, based on the threshold matrix, records the time interval for fine segmentation, and fine segmentation, based on Dynamic Time Warping(DTW) and Length-Root method, determines the endpoint for each candidate sign and selects the most possible one. The final recognition is obtained by concatenating the most possible signs. We evaluate our approach with Kinect-based dataset and the experiments demonstrate the effectiveness of our approach. In this paper we propose a novel method for human action recognition based on boosted key-frame selection and correlated pyramidal motion feature representations. Instead of using an unsupervised method to detect interest points, a Pyramidal Motion Feature (PMF), which combines optical flow with a biologically inspired feature, is extracted from each frame of a video sequence. The AdaBoost learning algorithm is then applied to select the most discriminative frames from a large feature pool. In this way, we obtain the top-ranked boosted frames of each video sequence as the key frames which carry the most representative motion information. Furthermore, we utilise the correlogram which focuses not only on probabilistic distributions within one frame but also on the temporal relationships of the action sequence. In the classification phase, a Support-Vector Machine (SVM) is adopted as the final classifier for human action recognition. To demonstrate generalizability, our method has been systematically tested on a variety of datasets and shown to be more effective and accurate for action recognition compared to the previous work. We obtain overall accuracies of: 95.5 , 93.7 , and 36.5 with our proposed method on the KTH, the multiview IXMAS and the challenging HMDB51 datasets, respectively. In this paper, we propose an HMM trajectory-guided, real image sample concatenation approach to photo-realistic talking head synthesis. An audio-visual database of a person is recorded first for training a statistical Hidden Markov Model (HMM) of Lips movement. The HMM is then used to generate the dynamic trajectory of lips movement for given speech signals in the maximum probability sense. The generated trajectory is then used as a guide to select, from the original training database, an optimal sequence of lips images which are then stitched back to a background head video. We also propose a minimum generation error (MGE) training method to refine the audio-visual HMM to improve visual speech trajectory synthesis. Compared with the traditional maximum likelihood (ML) estimation, the proposed MGE training explicitly optimizes the quality of generated visual speech trajectory, where the audio-visual HMM modeling is jointly refined by using a heuristic method to find the optimal state alignment and a probabilistic descent algorithm to optimize the model parameters under the MGE criterion. In objective evaluation, compared with the ML-based method, the proposed MGE-based method achieves consistent improvement in the mean square error reduction, correlation increase, and recovery of global variance. For as short as 20 min recording of audio video footage, the proposed system can synthesize a highly photo-realistic talking head in sync with the given speech signals (natural or TTS synthesized). This system won the first place in the A V consistency contest in LIPS Challenge, perceptually evaluated by recruited human subjects.