In this paper we propose a novel approach to multi-action recognition that performs joint segmentation and classification. This approach models each action using a Gaussian mixture using robust low-dimensional action features. Segmentation is achieved by performing classification on overlapping temporal windows, which are then merged to produce the final result. This approach is considerably less complicated than previous methods which use dynamic programming or computationally expensive hidden Markov models (HMMs). Initial experiments on a stitched version of the KTH dataset show that the proposed approach achieves an accuracy of 78.3 , outperforming a recent HMM-based approach which obtained 71.2 .
Hidden Markov models (HMMs) provide joint segmentation and classification of sequential data by efficient inference algorithms and have therefore been employed in fields as diverse as speech recognition, document processing, and genomics. However, conventional HMMs do not suit action segmentation in video due to the nature of the measurements which are often irregular in space and time, high dimensional and affected by outliers. For this reason, in this paper we present a joint action segmentation and classification approach based on an extended model: the hidden Markov model for multiple, irregular observations (HMM-MIO). Experiments performed over a concatenated version of the popular KTH action dataset and the challenging CMU multi-modal activity dataset (CMU-MMAC) report accuracies comparable to or higher than those of a bag-of-features approach, showing the usefulness of improved sequential models for joint action segmentation and classification tasks. In this paper, we propose a robust recognition and segmentation method for daily actions with a novel multi-task sequence labeling algorithm called multi-task conditional random field (MT-CRF). Multi-Task sequence labeling is a task of assigning input sequence to sequence of multi-labels that consist of one or multiple symbols in single frame. Multi-Task sequence labeling is essential for action recognition, since motions can be often classified into multi-labels, e.g. he is folding arms while sitting. The MT-CRFs: extensions of conditional random fields (CRFs), incorporate jointly interaction between action labels as well as Markov property of actions, to improve the performance of the joint accuracy: the accuracy for whole labels at specific time. The MT-CRFs offer several advantages over the generative dynamic Bayesian networks (DBNs), which are often utilized as multi-task sequence labelers. First, the MT-CRFs allow relaxing the strong assumption of conditional independence of observed motion, which is used in DBNs. Second, the MT-CRFs exploit the power of non-Markovian discriminative classification frameworks instead of generative models in DBNs. With deep insight of the problem Multi-Task sequence labeling, the inference process of the classifier gains more efficiency than the previous Markov random fields that tackle multi-task sequence labeling. The experimental results show that classifiers with MT-CRFs have better performance than cascaded classifiers with a couple of CRFs. Automatic video segmentation and action recognition has been a long-standing problem in computer vision. Much work in the literature treats video segmentation and action recognition as two independent problems; while segmentation is often done without a temporal model of the activity, action recognition is usually performed on pre-segmented clips. In this paper we propose a novel method that avoids the limitations of the above approaches by jointly performing video segmentation and action recognition. Unlike standard approaches based on extensions of dynamic Bayesian networks, our method is based on a discriminative temporal extension of the spatial bag-of-words model that has been very popular in object recognition. The classification is performed robustly within a multi-class SVM framework whereas the inference over the segments is done efficiently with dynamic programming. Experimental results on honeybee, Weizmann, and Hollywood datasets illustrate the benefits of our approach compared to state-of-the-art methods.