We present a novel, reusable and task-agnostic primitive for assessing the outcome of a force-interaction robotic skill, useful e.g. for applications such as quality control in industrial manufacturing. The proposed method is easily programmed by kinesthetic teaching, and the desired adaptability and reusability are achieved by machine learning models. The primitive records sensory data during both demonstrations and reproductions of a movement. Recordings include the end-effector's Cartesian pose and exerted wrench at each time step. The collected data are then used to train Gaussian Processes which create models of the wrench as a function of the robot's pose. The similarity between the wrench models of the demonstration and the movement's reproduction is derived by measuring their Hellinger distance. This comparison creates features that are fed as inputs to a Naive Bayes classifier which estimates the movement's probability of success. The evaluation is performed on two diverse robotic assembly tasks -- snap-fitting and screwing -- with a total of 5 use cases, 11 demonstrations, and more than 200 movement executions. The performance metrics prove the proposed method's capability of generalization to different demonstrations and movements.
Haidu al @cite_13 segment the task in smaller pose trajectories and train a Hidden Markov Model (HMM) using each segment as a state. They identify promising states by aligning the segmented samples to the HMM, which contains key information for failure detection. A predictor is trained to assess the task from the aligned data at those promising states. Similarly, Di Lello al @cite_2 segment the wrench signatures by the control strategy of a finite state machine. Each wrench signal is treated as a Bayesian time-series model. A HMM is trained via a Bayesian non-parametric method to detect the deviation from the successful execution. Both of those two methods require the predefined segmentation of the trajectory. However, there is often no such clear segmentation such as Zero Velociy Crossing point @cite_15 or contact events that are plausible in human demonstrations.
@cite_15 @cite_13 @cite_2
2161491846 2096925292 1973026036
This paper proposes an effective segmentation-free approach using a hybrid neural network hidden Markov model (NN-HMM) for offline handwritten Chinese text recognition (HCTR). In the general Bayesian framework, the handwritten Chinese text line is sequentially modeled by HMMs with each representing one character class, while the NN-based classifier is adopted to calculate the posterior probability of all HMM states. The key issues in feature extraction, character modeling, and language modeling are comprehensively investigated to show the effectiveness of NN-HMM framework for offline HCTR. First, a conventional deep neural network (DNN) architecture is studied with a well-designed feature extractor. As for the training procedure, the label refinement using forced alignment and the sequence training can yield significant gains on top of the frame-level cross-entropy criterion. Second, a deep convolutional neural network (DCNN) with automatically learned discriminative features demonstrates its superiority to DNN in the HMM framework. Moreover, to solve the challenging problem of distinguishing quite confusing classes due to the large vocabulary of Chinese characters, NN-based classifier should output 19900 HMM states as the classification units via a high-resolution modeling within each character. On the ICDAR 2013 competition task of CASIA-HWDB database, DNN-HMM yields a promising character error rate (CER) of 5.24 by making a good trade-off between the computational complexity and recognition accuracy. To the best of our knowledge, DCNN-HMM can achieve a best published CER of 3.53 . We present a novel probabilistic method for topic segmentation on unstructured text. One previous approach to this problem utilizes the hidden Markov model (HMM) method for probabilistically modeling sequence data [7]. The HMM treats a document as mutually independent sets of words generated by a latent topic variable in a time series. We extend this idea by embedding Hofmann's aspect model for text [5] into the segmenting HMM to form an aspect HMM (AHMM). In doing so, we provide an intuitive topical dependency between words and a cohesive segmentation model. We apply this method to segment unbroken streams of New York Times articles as well as noisy transcripts of radio programs on SpeechBot , an online audio archive indexed by an automatic speech recognition engine. We provide experimental comparisons which show that the AHMM outperforms the HMM for this task. This paper presents the application of a Bayesian nonparametric time-series model to process monitoring and fault classification for industrial robotic tasks. By means of an alignment task performed with a real robot, we show how the proposed approach allows to learn a set of sensor signature models encoding the spatial and temporal correlations among wrench measurements recorded during a number of successful task executions. Using these models, it is possible to detect continuously and on-line deviations from the expected sensor readings. Separate models are learned for a set of possible error scenarios involving a human modifying the workspace configuration. These non-nominal task executions are correctly detected and classified with an on-line algorithm, which opens the possibility for the development of error-specific recovery strategies. Our work is complementary to previous approaches in robotics, where process monitors based on probabilistic models, but limited to contact events, were developed for control purposes. Instead, in this paper we focus on capturing dynamic models of sensor signatures throughout the whole task, therefore allowing continuous monitoring and extending the system ability to interpret and react to errors. We present a novel method for generating sports video summary highlights. Specifically, our method localizes semantically important events in sport programs by detecting slow motion replays of these events, and then generates highlights of these events at multiple levels. In our method, a hidden Markov model (HMM) is used to model slow motion replays, and an inference algorithm is introduced which computes the probability of a slow motion replay segment, and localizes the boundaries of the segment as well. An effective new feature is used in our HMM, based on a moving measure of the number of zero-crossings and the amplitudes of variations over time of video field differences. Furthermore, the method is capable of filtering out slow motion play segments in commercials. As compared with existing methods for video event detection, our method is more generic (ie, domain independent), and has the ability to capture inherently important events. We present a new approach to recognizing events in videos. We first detect and track moving objects in the scene. Based on the shape and motion properties of these objects, we infer probabilities of primitive events frame-by-frame by using Bayesian networks. Composite events, consisting of multiple primitive events, over extended periods of time are analyzed by using a hidden, semi-Markov finite state model. This results in more reliable event segmentation compared to the use of standard HMMs in noisy video sequences at the cost of some increase in computational complexity. We describe our approach to reducing this complexity. We demonstrate the effectiveness of our algorithm using both real-world and perturbed data. This paper proposes a method for transcribing drums from polyphonic music using a network of connected hidden Markov models (HMMs). The task is to detect the temporal locations of unpitched percussive sounds (such as bass drum or hi-hat) and recognise the instruments played. Contrary to many earlier methods, a separate sound event segmentation is not done, but connected HMMs are used to perform the segmentation and recognition jointly. Two ways of using HMMs are studied: modelling combinations of the target drums and a detector-like modelling of each target drum. Acoustic feature parametrisation is done with mel-frequency cepstral coefficients and their first-order temporal derivatives. The effect of lowering the feature dimensionality with principal component analysis and linear discriminant analysis is evaluated. Unsupervised acoustic model parameter adaptation with maximum likelihood linear regression is evaluated for compensating the differences between the training and target signals. The performance of the proposed method is evaluated on a publicly available data set containing signals with and without accompaniment, and compared with two reference methods. The results suggest that the transcription is possible using connected HMMs, and that using detector-like models for each target drum provides a better performance than modelling drum combinations. Existing gesture segmentations use the backward spotting scheme that first detects the end point, then traces back to the start point and sends the extracted gesture segment to the hidden Markov model (HMM) for gesture recognition. This makes an inevitable time delay between the gesture segmentation and recognition and is not appropriate for continuous gesture recognition. To solve this problem, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously. The start and end points of gestures are determined by zero crossing from negative to positive (or from positive to negative) of a competitive differential observation probability that is defined by the difference of observation probability between the maximal gesture and the non-gesture. We also propose the sliding window and accumulative HMMs. The former is used to alleviate the effect of incomplete feature extraction on the observation probability and the latter improves the gesture recognition rate greatly by accepting all accumulated gesture segments between the start and end points and deciding the gesture type by a majority vote of all intermediate recognition results. We use the predetermined association mapping to determine the 3D articulation data, which reduces the feature extraction time greatly. We apply the proposed simultaneous gesture segmentation and recognition method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment. Experimental results show that the proposed method has a good recognition rate of 95.42 for continuously changing gestures. We present algorithms for coupling and training hidden Markov models (HMMs) to model interacting processes, and demonstrate their superiority to conventional HMMs in a vision task classifying two-handed actions. HMMs are perhaps the most successful framework in perceptual computing for modeling and classifying dynamic behaviors, popular because they offer dynamic time warping, a training algorithm and a clear Bayesian semantics. However the Markovian framework makes strong restrictive assumptions about the system generating the signal-that it is a single process having a small number of states and an extremely limited state memory. The single-process model is often inappropriate for vision (and speech) applications, resulting in low ceilings on model performance. Coupled HMMs provide an efficient way to resolve many of these problems, and offer superior training speeds, model likelihoods, and robustness to initial conditions. This paper presents a novel method for facial expression recognition using only sequence level labeling. With facial image sequences containing multiple peaks of expression, our method aims to label these sequences and identifies expressional peaks automatically. We formulate this weakly labeled expression recognition as a multi-instance learning (MIL) problem. First, image sequences are clustered into multiple segments. After segmentation, image sequences are regarded as bags in MIL and the segments in the bags are viewed as instances. Second, bags data are used to train a discriminative classifier which combined multi-instance learning and discriminative Hidden Markov Model (HMM) learning. In our method, HMM is used to model temporal variation within segments. We conducted experiments on CK+ database and UNBC-McMaster Shoulder Pain Database. Experimental results on both databases show that our method can not only label the sequences effectively, but also locate apex frames of multi-peak sequences. Besides, the experiments demonstrate that our method outperforms state of the art. Hidden Markov models (HMMs) are a powerful probabilistic tool for modeling sequential data, and have been applied with success to many text-related tasks, such as part-of-speech tagging, text segmentation and information extraction. In these cases, the observations are usually modeled as multinomial distributions over a discrete vocabulary, and the HMM parameters are set to maximize the likelihood of the observations. This paper presents a new Markovian sequence model, closely related to HMMs, that allows observations to be represented as arbitrary overlapping features (such as word, capitalization, formatting, part-of-speech), and defines the conditional probability of state sequences given observation sequences. It does this by using the maximum entropy framework to fit a set of exponential models that represent the probability of a state given an observation and the previous state. We present positive experimental results on the segmentation of FAQ’s. In this work, we tackle the problem of transform-invariant unsupervised learning in the space of Covariance matrices and applications thereof. We begin by introducing the Spectral Polytope Covariance Matrix (SPCM) Similarity function; a similarity function for Covariance matrices, invariant to any type of transformation. We then derive the SPCM-CRP mixture model, a transform-invariant non-parametric clustering approach for Covariance matrices that leverages the proposed similarity function, spectral embedding and the distance-dependent Chinese Restaurant Process (dd-CRP) (Blei and Frazier, 2011). The scalability and applicability of these two contributions is extensively validated on real-world Covariance matrix datasets from diverse research fields. Finally, we couple the SPCM-CRP mixture model with the Bayesian non-parametric Indian Buffet Process (IBP) - Hidden Markov Model (HMM) (, 2009), to jointly segment and discover transform-invariant action primitives from complex sequential data. Resulting in a topic-modeling inspired hierarchical model for unsupervised time-series data analysis which we call ICSC-HMM (IBP Coupled SPCM-CRP Hidden Markov Model). The ICSC-HMM is validated on kinesthetic demonstrations of uni-manual and bi-manual cooking tasks; achieving unsupervised human-level decomposition of complex sequential tasks. Recently, great progress has been made in sign language recognition. Most approaches are based on the Hidden Markov Model (HMM) with various features, such as motion trajectory. Recognition for sign sentences is obtained from optimal path by Viterbi algorithm, however, some wrong jumps are usually caused by transitional movements between signs. To address the problem, in this paper, we propose an approach consisting of two stages: offline training and online recognition. In the offline training stage, we propose a threshold matrix and rate thresholds. Each element of the threshold matrix describes the minimal probability when a segment belongs to a sign, and rate thresholds are defined as the average probability for signs. So, if certain segment's evaluation is smaller than all the thresholds, it is regarded as a transitional movement and then it should be removed. In the online recognition stage, coarse segmentation, based on the threshold matrix, records the time interval for fine segmentation, and fine segmentation, based on Dynamic Time Warping(DTW) and Length-Root method, determines the endpoint for each candidate sign and selects the most possible one. The final recognition is obtained by concatenating the most possible signs. We evaluate our approach with Kinect-based dataset and the experiments demonstrate the effectiveness of our approach. Automatic facial action unit (AU) detection from video is a long-standing problem in computer vision. Two main approaches have been pursued: (1) static modeling — typically posed as a discriminative classification problem in which each video frame is evaluated independently; (2) temporal modeling — frames are segmented into sequences and typically modeled with a variant of dynamic Bayesian networks. We propose a segment-based approach, kSeg-SVM, that incorporates benefits of both approaches and avoids their limitations. kSeg-SVM is a temporal extension of the spatial bag-of-words. kSeg-SVM is trained within a structured output SVM framework that formulates AU detection as a problem of detecting temporal events in a time series of visual features. Each segment is modeled by a variant of the BoW representation with soft assignment of the words based on similarity. Our framework has several benefits for AU detection: (1) both dependencies between features and the length of action units are modeled; (2) all possible segments of the video may be used for training; and (3) no assumptions are required about the underlying structure of the action unit events (e.g., i.i.d.). Our algorithm finds the best k-or-fewer segments that maximize the SVM score. Experimental results suggest that the proposed method outperforms state-of-the-art static methods for AU detection. Classification of musical genres gives a useful measure of similarity and is often the most useful descriptor of a musical piece. Previous techniques to use hidden Markov models (HMMs) for automatic genre classification have used a single HMM to model an entire song or genre. This paper provides a framework to give finer segmentation of HMMs through acoustic segment modeling. Modeling each of these acoustic segments with an HMM builds a timbral dictionary in the same fashion that one would create a phonetic dictionary for speech. A symbolic transcription is created by finding the most likely sequence of symbols. These transcriptions then serve as inputs into an efficient text classifier utilized to provide a solution to the genre classification problem. This paper demonstrates that language-ignorant approaches provide results that are consistent with the current state-of-the-art for the genre classification problem. However, the finer segmentation potentially allows for “musical language”-based syntactic rules to enhance performance. Hidden Markov models (HMMs) provide joint segmentation and classification of sequential data by efficient inference algorithms and have therefore been employed in fields as diverse as speech recognition, document processing, and genomics. However, conventional HMMs do not suit action segmentation in video due to the nature of the measurements which are often irregular in space and time, high dimensional and affected by outliers. For this reason, in this paper we present a joint action segmentation and classification approach based on an extended model: the hidden Markov model for multiple, irregular observations (HMM-MIO). Experiments performed over a concatenated version of the popular KTH action dataset and the challenging CMU multi-modal activity dataset (CMU-MMAC) report accuracies comparable to or higher than those of a bag-of-features approach, showing the usefulness of improved sequential models for joint action segmentation and classification tasks. Many alternative models have been proposed to address some of the shortcomings of the hidden Markov model (HMM), which is currently the most popular approach to speech recognition. In particular, a variety of models that could be broadly classified as segment models have been described for representing a variable-length sequence of observation vectors in speech recognition applications. Since there are many aspects in common between these approaches, including the general recognition and training problems, it is useful to consider them in a unified framework. The paper describes a general stochastic model that encompasses most of the models proposed in the literature, pointing out similarities of the models in terms of correlation and parameter tying assumptions, and drawing analogies between segment models and HMMs. In addition, we summarize experimental results assessing different modeling assumptions and point out remaining open questions. In this paper, we propose an HMM trajectory-guided, real image sample concatenation approach to photo-realistic talking head synthesis. An audio-visual database of a person is recorded first for training a statistical Hidden Markov Model (HMM) of Lips movement. The HMM is then used to generate the dynamic trajectory of lips movement for given speech signals in the maximum probability sense. The generated trajectory is then used as a guide to select, from the original training database, an optimal sequence of lips images which are then stitched back to a background head video. We also propose a minimum generation error (MGE) training method to refine the audio-visual HMM to improve visual speech trajectory synthesis. Compared with the traditional maximum likelihood (ML) estimation, the proposed MGE training explicitly optimizes the quality of generated visual speech trajectory, where the audio-visual HMM modeling is jointly refined by using a heuristic method to find the optimal state alignment and a probabilistic descent algorithm to optimize the model parameters under the MGE criterion. In objective evaluation, compared with the ML-based method, the proposed MGE-based method achieves consistent improvement in the mean square error reduction, correlation increase, and recovery of global variance. For as short as 20 min recording of audio video footage, the proposed system can synthesize a highly photo-realistic talking head in sync with the given speech signals (natural or TTS synthesized). This system won the first place in the A V consistency contest in LIPS Challenge, perceptually evaluated by recruited human subjects. We propose a new blind segmentation approach to acoustic event detection (AED) based on i-vectors. Conventional approaches to AED often required well-segmented data with non-overlapping boundaries for competing events. Inspired by block-based automatic image annotation in image retrieval tasks, we blindly segment audio streams into equal-length pieces, label the underlying observed acoustic events with multiple categories and with no event boundary information, extract i-vector for them, and perform classification using support vector machine and maximal figure-of-merit based classifiers. Experiments on various sets of audio data show promising results with an average of 8 absolute gain in F1 over the conventional hidden Markov model based approach. An enhanced robustness at different noise levels is also observed. The key to the success lies in the enhanced discrimination power offered by the i-vector representation of the acoustic data. Index Terms: acoustic event detection, i-vector, blind segmentation, support vector machine, maximal figure-of-merit In this work, we address the task of weakly-supervised human action segmentation in long, untrimmed videos. Recent methods have relied on expensive learning models, such as Recurrent Neural Networks (RNN) and Hidden Markov Models (HMM). However, these methods suffer from expensive computational cost, thus are unable to be deployed in large scale. To overcome the limitations, the keys to our design are efficiency and scalability. We propose a novel action modeling framework, which consists of a new temporal convolutional network, named Temporal Convolutional Feature Pyramid Network (TCFPN), for predicting frame-wise action labels, and a novel training strategy for weakly-supervised sequence modeling, named Iterative Soft Boundary Assignment (ISBA), to align action sequences and update the network in an iterative fashion. The proposed framework is evaluated on two benchmark datasets, Breakfast and Hollywood Extended, with four different evaluation metrics. Extensive experimental results show that our methods achieve competitive or superior performance to state-of-the-art methods. Background Profile Hidden Markov Models (HMM) are statistical representations of protein families derived from patterns of sequence conservation in multiple alignments and have been used in identifying remote homologues with considerable success. These conservation patterns arise from fold specific signals, shared across multiple families, and function specific signals unique to the families. The availability of sequences pre-classified according to their function permits the use of negative training sequences to improve the specificity of the HMM, both by optimizing the threshold cutoff and by modifying emission probabilities to minimize the influence of fold-specific signals. A protocol to generate family specific HMMs is described that first constructs a profile HMM from an alignment of the family's sequences and then uses this model to identify sequences belonging to other classes that score above the default threshold (false positives). Ten-fold cross validation is used to optimise the discrimination threshold score for the model. The advent of fast multiple alignment methods enables the use of the profile alignments to align the true and false positive sequences, and the resulting alignments are used to modify the emission probabilities in the original model.