A novel method to identify trampoline skills using a single video camera is proposed herein. Conventional computer vision techniques are used for identification, estimation, and tracking of the gymnast's body in a video recording of the routine. For each frame, an open source convolutional neural network is used to estimate the pose of the athlete's body. Body orientation and joint angle estimates are extracted from these pose estimates. The trajectories of these angle estimates over time are compared with those of labelled reference skills. A nearest neighbour classifier utilising a mean squared error distance metric is used to identify the skill performed. A dataset containing 714 skill examples with 20 distinct skills performed by adult male and female gymnasts was recorded and used for evaluation of the system. The system was found to achieve a skill identification accuracy of 80.7 for the dataset.
In @cite_11 , inertial sensors were used to measure body point acceleration and orientation. The gymnast was required to wear a body suit containing ten inertial measurement units. The sensor data streams were transformed into a feature sequence for classification. For each skill, a motion template was learned. The feature sequence of the unknown trampoline motions were compared with a set of skill templates using a variant of dynamic time warping. The best accuracy achieved was 84.7 A survey of vision-based methods for general human motion representation, segmentation and recognition can be found in @cite_5 . In @cite_9 , judging of rhythmic gymnastics skills from video was investigated. The movement of the gymnast was tracked using optical flow. Velocity field information was extracted across all frames of a skill and projected into a velocity covariance eigenspace. Similar movements were found to trace out unique, but similar, trajectories. New video recordings were classified based on their distance from reference trajectories. The system's specificity was approximately 85
@cite_5 @cite_9 @cite_11
2545779397 2087617234 2195342085 1670326883 2056358962 1578415806 2066864006 1968194699 2029904017 2057343530 2567705927 2040378481 2732126315 2032375919 2081773958 2538522345 2553098005 2140924050 2333544554 2056639157
The ability to accurately observe human motion and identify human activities is essential for developing automatic rehabilitation and sports training systems. In this paper, large-scale exercise motion data obtained from a forearm-worn wearable sensor are classified with a convolutional neural network (CNN). Time series data consisting of accelerometer and orientation measurements are formatted as "images", allowing the CNN to automatically extract discriminative features. The resulting CNN classifies 50 gym exercises with 92.14 accuracy. A comparative study on the effects of image formatting and different CNN architectures is also presented. Abstract We describe a conceptually simple algorithm for assigning judgement scores to rhythmic gymnastic movements, which could improve scoring objectivity and reduce judgemental bias during competitions. Our method, implemented as a real-time computer vision software, takes a video shot or a live performance video stream as input and extracts detailed velocity field information from body movements, transforming them into specialized spatio-temporal image templates. The collection of such images over time, when projected into a velocity covariance eigenspace, trace out unique but similar trajectories for a particular gymnastic movement type. By comparing separate executions of the same atomic gymnastic routine, our method assigns a quality judgement score that is related to the distance between the respective spatio-temporal trajectories. For several standard gymnastic movements, the method accurately assigns scores that are comparable to those assigned by expert judges. We also describe our rhythmic gymnastic video shot database, which we have made freely available to the human movement research community. The database can be obtained at http: www.milegroup.net apps gymdb . This paper presents a review of different classification techniques used to recognize human activities from wearable inertial sensor data. Three inertial sensor units were used in this study and were worn by healthy subjects at key points of upper lower body limbs (chest, right thigh and left ankle). Three main steps describe the activity recognition process: sensors’ placement, data pre-processing and data classification. Four supervised classification techniques namely, k-Nearest Neighbor (k-NN), Support Vector Machines (SVM), Gaussian Mixture Models (GMM), and Random Forest (RF) as well as three unsupervised classification techniques namely, k-Means, Gaussian mixture models (GMM) and Hidden Markov Model (HMM), are compared in terms of correct classification rate, F-measure, recall, precision, and specificity. Raw data and extracted features are used separately as inputs of each classifier. The feature selection is performed using a wrapper approach based on the RF algorithm. Based on our experiments, the results obtained show that the k-NN classifier provides the best performance compared to other supervised classification algorithms, whereas the HMM classifier is the one that gives the best results among unsupervised classification algorithms. This comparison highlights which approach gives better performance in both supervised and unsupervised contexts. It should be noted that the obtained results are limited to the context of this study, which concerns the classification of the main daily living human activities using three wearable accelerometers placed at the chest, right shank and left ankle of the subject. This paper proposes an ego-motion tracking method that utilizes visual-inertial sensors for wearable blind navigation. The unique challenge of wearable motion tracking is to cope with arbitrary body motion and complex environmental dynamics. We introduce a visual sanity check to select accurate visual estimations by comparing visually estimated rotation with measured rotation by a gyroscope. The movement trajectory is recovered through adaptive fusion of visual estimations and inertial measurements, where the visual estimation outputs motion transformation between consecutive image captures, and inertial sensors measure translational acceleration and angular velocities. The frame rates of visual and inertial sensors are different, and vary with respect to time owning to visual sanity checks. We hence employ a multirate extended Kalman filter (EKF) to fuse visual and inertial estimations. The proposed method was tested in different indoor environments, and the results show its effectiveness and accuracy in ego-motion tracking. In this paper, we focus on the problem of motion tracking in unknown environments using visual and inertial sensors. We term this estimation task visual-inertial odometry (VIO), in analogy to the well-known visual-odometry problem. We present a detailed study of extended Kalman filter (EKF)-based VIO algorithms, by comparing both their theoretical properties and empirical performance. We show that an EKF formulation where the state vector comprises a sliding window of poses (the multi-state-constraint Kalman filter (MSCKF)) attains better accuracy, consistency, and computational efficiency than the simultaneous localization and mapping (SLAM) formulation of the EKF, in which the state vector contains the current pose and the features seen by the camera. Moreover, we prove that both types of EKF approaches are inconsistent, due to the way in which Jacobians are computed. Specifically, we show that the observability properties of the EKF's linearized system models do not match those of the underlying system, which causes the filters to underestimate the uncertainty in the state estimates. Based on our analysis, we propose a novel, real-time EKF-based VIO algorithm, which achieves consistent estimation by (i) ensuring the correct observability properties of its linearized system model, and (ii) performing online estimation of the camera-to-inertial measurement unit (IMU) calibration parameters. This algorithm, which we term MSCKF 2.0, is shown to achieve accuracy and consistency higher than even an iterative, sliding-window fixed-lag smoother, in both Monte Carlo simulations and real-world testing. We present an approach for visual discrimination of children from adults in video using characteristic regularities present in their locomotion patterns. The framework employs computer vision to analyze correlated, scale invariant locomotion properties for classifying different styles of walking. Male and female subjects for the experiments include six children (3-5 yrs) and nine adults (30-52 yrs). For the analysis, we coordinate a minimalist point-representation of the human body with a space-time analysis of head and ankle trajectories to characterize the modality. Together the properties of relative stride length and stride frequency are shown to clearly differentiate children from adult walkers. The highly correlated log-linear relationships for the stride properties are exploited to reduce the categorization problem to a linear discrimination task. Using a trained two-class linear perceptron, we were able to achieve a correct classification rate of 93-95 on our dataset. Our approach emphasizing the natural modal behavior in human motion offers a useful and general methodology as the basis for designing efficient motion recognition systems using limited visual features. Abstract This article reports an experimental study that aimed to quantitatively analyze motion coordination patterns across digits 2–5 (index to little finger), and examine the kinematic synergies during manipulative and gestic acts. Twenty-eight subjects (14 males and 14 females) performed two types of tasks, both right-handed: (1) cylinder-grasping that involved concurrent voluntary flexion of digits 2–5, and (2) voluntary flexion of individual fingers from digit 2 to 5 (i.e., one at a time). A five-camera opto-electronic motion capture system measured trajectories of 21 miniature reflective markers strategically placed on the dorsal surface landmarks of the hand. Joint angular profiles for 12 involved flexion–extension degrees of freedom (DOF's) were derived from the measured coordinates of surface markers. Principal components analysis (PCA) was used to examine the temporal covariation between joint angles. A mathematical modeling procedure, based on hyperbolic tangent functions, characterized the sigmoidal shaped angular profiles with four kinematically meaningful parameters. The PCA results showed that for all the movement trials ( n =280), two principal components accounted for at least 98 of the variance. The angular profiles ( n =2464) were accurately characterized, with the mean (±SD) coefficient of determination ( R 2 ) and root-mean-square-error (RMSE) being 0.95 (±0.12) and 1.03° (±0.82°), respectively. The resulting parameters which quantified both the spatial and temporal aspects of angular profiles revealed stereotypical patterns including a predominant (87 of all trials) proximal-to-distal flexion sequence and characteristic interdependence – involuntary joint flexion induced by the voluntarily flexed joint. The principal components' weights and the kinematic parameters also exhibited qualitatively similar variation patterns. Motor control interpretations and new insights regarding the underlying synergistic mechanisms, particularly in relation to previous findings on force synergies, are discussed. A generic framework for the automated classification of human movements using an accelerometry monitoring system is introduced. The framework was structured around a binary decision tree in which movements were divided into classes and subclasses at different hierarchical levels. General distinctions between movements were applied in the top levels, and successively more detailed subclassifications were made in the lower levels of the tree. The structure was modular and flexible: parts of the tree could be reordered, pruned or extended, without the remainder of the tree being affected. This framework was used to develop a classifier to identify basic movements from the signals obtained from a single, waist-mounted triaxial accelerometer. The movements were first divided into activity and rest. The activities were classified as falls, walking, transition between postural orientations, or other movement. The postural orientations during rest were classified as sitting, standing or lying. In controlled laboratory studies in which 26 normal, healthy subjects carried out a set of basic movements, the sensitivity of every classification exceeded 87 , and the specificity exceeded 94 ; the overall accuracy of the system, measured as the number of correct classifications across all levels of the hierarchy, was a sensitivity of 97.7 and a specificity of 98.7 over a data set of 1309 movements. Generating a visually appealing human motion sequence using low-dimensional control signals is a major line of study in the motion research area in computer graphics. We propose a novel approach that allows us to reconstruct full body human locomotion using a single inertial sensing device, a smartphone. Smartphones are among the most widely used devices and incorporate inertial sensors such as an accelerometer and a gyroscope. To find a mapping between a full body pose and smartphone sensor data, we perform low dimensional embedding of full body motion capture data, based on a Gaussian Process Latent Variable Model. Our system ensures temporal coherence between the reconstructed poses by using a state decomposition model for automatic phase segmentation. Finally, application of the proposed nonlinear regression algorithm finds a proper mapping between the latent space and the sensor data. Our framework effectively reconstructs plausible 3D locomotion sequences. We compare the generated animation to ground truth data obtained using a commercial motion capture system. This paper presents a tracking system for ego-motion estimation which fuses vision and inertial measurements using EKF and UKF (Extended and Unscented Kalman Filters), where a comparison of their performance has been done. It also considers the multi-rate nature of the sensors: inertial sensing is sampled at a fast sampling frequency while the sampling frequency of vision is lower. the proposed approach uses a constant linear acceleration model and constant angular velocity model based on quaternions, which yields a non-linear model for states and a linear model in measurement equations. Results show that a significant improvement is obtained on the estimation when fusing both measurements with respect to just vision or just inertial measurements. It is also shown that the proposed system can estimate fast-motions even when vision system fails. Moreover, a study of the influence of the noise covariance is also performed, which aims to select their appropriate values at the tuning process. The setup is an end-effector mounted camera, which allow us to pre-define basic rotational and translational motions for validating results. Walking impairment resulted by various chronic diseases, disorders and injuries have been investigated using recent emerging wearable technology, for instance, gait assessment using inertial body sensors in 6-minute walk (6MW) for persons with Multiple Sclerosis (PwMS) to identify spatiotemporal features useful to assess MS progression. However, most studies to date have investigated the features extracted from movements of the lower limbs and do not provide a holistic gait assessment. A recent pilot study demonstrated that the holistic gait assessment such as evaluating the associations among lower and upper limbs provided better discrimination between healthy controls and PwMS. This paper is motivated by this and further aim to answer the following question: can we identify the temporal gait patterns in terms of the holistic gait assessment? Traditionally this suffers from the statistical property of the causality inference method adopted by previous study. We proposed a deep convolutional neural network (CNN) to learn the temporal and spectral associations among the time-series motion data captured by the inertial body sensors. A simulated model was developed to train the CNN, and then the trained CNN was adopted to assess the gait performance from a pilot dataset with 41 subjects (28 PwMS and 13 healthy controls). Experimental results are reported to illustrate the performance of the proposed approach. Real-time tracking of human body motion is an important technology in synthetic environments, robotics, and other human-computer interaction applications. This paper presents an extended Kalman filter designed for real-time estimation of the orientation of human limb segments. The filter processes data from small inertial magnetic sensor modules containing triaxial angular rate sensors, accelerometers, and magnetometers. The filter represents rotation using quaternions rather than Euler angles or axis angle pairs. Preprocessing of the acceleration and magnetometer measurements using the Quest algorithm produces a computed quaternion input for the filter. This preprocessing reduces the dimension of the state vector and makes the measurement equations linear. Real-time implementation and testing results of the quaternion-based Kalman filter are presented. Experimental results validate the filter design, and show the feasibility of using inertial magnetic sensor modules for real-time human body motion tracking We demonstrate accurate spatio-temporal gait data classification from raw tomography sensor data without the need to reconstruct images. This is based on a simple yet efficient machine learning methodology based on a convolutional neural network architecture for learning spatio-temporal features, automatically end-to-end from raw sensor data. In a case study on a floor pressure tomography sensor, experimental results show an effective gait pattern classification F -score performance of 97.88 @math 1.70 . It is shown that the automatic extraction of classification features from raw data leads to a substantially better performance, compared to features derived by shallow machine learning models that use the reconstructed images as input, implying that for the purpose of automatic decision-making it is possible to eliminate the image reconstruction step. This approach is portable across a range of industrial tasks that involve tomography sensors. The proposed learning architecture is computationally efficient, has a low number of parameters and is able to achieve reliable classification F -score performance from a limited set of experimental samples. We also introduce a floor sensor dataset of 892 samples, encompassing experiments of 10 manners of walking and 3 cognitive-oriented tasks to yield a total of 13 types of gait patterns. Humans interact with their environment through sensory information and motor actions. These interactions may be understood via the underlying geometry of both perception and action. While the motor space is typically considered by default to be Euclidean, persistent behavioral observations point to a different underlying geometric structure. These observed regularities include the “two-thirds power law”, which connects path curvature with velocity, and “local isochrony”, which prescribes the relation between movement time and its extent. Starting with these empirical observations, we have developed a mathematical framework based on differential geometry, Lie group theory and Cartan’s moving frame method for the analysis of human hand trajectories. We also use this method to identify possible motion primitives, i.e., elementary building blocks from which more complicated movements are constructed. We show that a natural geometric description of continuous repetitive hand trajectories is not Euclidean but equi-affine. Specifically, equi-affine velocity is piecewise constant along movement segments, and movement execution time for a given segment is proportional to its equi-affine arc-length. Using this mathematical framework, we then analyze experimentally recorded drawing movements. To examine movement segmentation and classification, the two fundamental equi-affine differential invariants—equi-affine arc-length and curvature are calculated for the recorded movements. We also discuss the possible role of conic sections, i.e., curves with constant equi-affine curvature, as motor primitives and focus in more detail on parabolas, the equi-affine geodesics. Finally, we explore possible schemes for the internal neural coding of motor commands by showing that the equi-affine framework is compatible with the common model of population coding of the hand velocity vector when combined with a simple assumption on its dynamics. We then discuss several alternative explanations for the role that the equi-affine metric may play in internal representations of motion perception and production. In this paper we propose a novel method for human action recognition based on boosted key-frame selection and correlated pyramidal motion feature representations. Instead of using an unsupervised method to detect interest points, a Pyramidal Motion Feature (PMF), which combines optical flow with a biologically inspired feature, is extracted from each frame of a video sequence. The AdaBoost learning algorithm is then applied to select the most discriminative frames from a large feature pool. In this way, we obtain the top-ranked boosted frames of each video sequence as the key frames which carry the most representative motion information. Furthermore, we utilise the correlogram which focuses not only on probabilistic distributions within one frame but also on the temporal relationships of the action sequence. In the classification phase, a Support-Vector Machine (SVM) is adopted as the final classifier for human action recognition. To demonstrate generalizability, our method has been systematically tested on a variety of datasets and shown to be more effective and accurate for action recognition compared to the previous work. We obtain overall accuracies of: 95.5 , 93.7 , and 36.5 with our proposed method on the KTH, the multiview IXMAS and the challenging HMDB51 datasets, respectively. In recent years there have been excellent results in visual-inertial odometry techniques, which aim to compute the incremental motion of the sensor with high accuracy and robustness. However, these approaches lack the capability to close loops and trajectory estimation accumulates drift even if the sensor is continually revisiting the same place. In this letter, we present a novel tightly coupled visual-inertial simultaneous localization and mapping system that is able to close loops and reuse its map to achieve zero-drift localization in already mapped areas. While our approach can be applied to any camera configuration, we address here the most general problem of a monocular camera, with its well-known scale ambiguity. We also propose a novel IMU initialization method, which computes the scale, the gravity direction, the velocity, and gyroscope and accelerometer biases, in a few seconds with high accuracy. We test our system in the 11 sequences of a recent micro-aerial vehicle public dataset achieving a typical scale factor error of 1 and centimeter precision. We compare to the state-of-the-art in visual-inertial odometry in sequences with revisiting, proving the better accuracy of our method due to map reuse and no drift accumulation. The correct execution of well-defined movements plays a crucial role in physical rehabilitation and sports. While there is an extensive number of well-established approaches for human action recognition, the task of assessing the quality of actions and providing feedback for correcting inaccurate movements has remained an open issue in the literature. We present a learning-based method for efficiently providing feedback on a set of training movements captured by a depth sensor. We propose a novel recursive neural network that uses growing self-organization for the efficient learning of body motion sequences. The quality of actions is then computed in terms of how much a performed movement matches the correct continuation of a learned sequence. The proposed system provides visual assistance to the person performing an exercise by displaying real-time feedback, thus enabling the user to correct inaccurate postures and motion intensity. We evaluate our approach with a data set containing 3 powerlifting exercises performed by 17 athletes. Experimental results show that our novel architecture outperforms our previous approach for the correct prediction of routines and the detection of mistakes both in a single- and multiple-subject scenario. Vision-aided inertial navigation systems (V-INSs) can provide precise state estimates for the 3-D motion of a vehicle when no external references (e.g., GPS) are available. This is achieved by combining inertial measurements from an inertial measurement unit (IMU) with visual observations from a camera under the assumption that the rigid transformation between the two sensors is known. Errors in the IMU-camera extrinsic calibration process cause biases that reduce the estimation accuracy and can even lead to divergence of any estimator processing the measurements from both sensors. In this paper, we present an extended Kalman filter for precisely determining the unknown transformation between a camera and an IMU. Contrary to previous approaches, we explicitly account for the time correlation of the IMU measurements and provide a figure of merit (covariance) for the estimated transformation. The proposed method does not require any special hardware (such as spin table or 3-D laser scanner) except a calibration target. Furthermore, we employ the observability rank criterion based on Lie derivatives and prove that the nonlinear system describing the IMU-camera calibration process is observable. Simulation and experimental results are presented that validate the proposed method and quantify its accuracy. In this work, we present an approach to fuse video with sparse orientation data obtained from inertial sensors to improve and stabilize full-body human motion capture. Even though video data is a strong cue for motion analysis, tracking artifacts occur frequently due to ambiguities in the images, rapid motions, occlusions or noise. As a complementary data source, inertial sensors allow for accurate estimation of limb orientations even under fast motions. However, accurate position information cannot be obtained in continuous operation. Therefore, we propose a hybrid tracker that combines video with a small number of inertial units to compensate for the drawbacks of each sensor type: on the one hand, we obtain drift-free and accurate position information from video data and, on the other hand, we obtain accurate limb orientations and good performance under fast motions from inertial sensors. In several experiments we demonstrate the increased performance and stability of our human motion tracker. This paper presents a novel method for learning object manipulation such as rotating an object or placing one object on another. In this method, motions are learned using reference-point-dependent probabilistic models, which can be used for the generation and recognition of motions. The method estimates (i) the reference point, (ii) the intrinsic coordinate system type, which is the type of coordinate system intrinsic to a motion, and (iii) the probabilistic model parameters of the motion that is considered in the intrinsic coordinate system. Motion trajectories are modeled by a hidden Markov model (HMM), and an HMM-based method using static and dynamic features is used for trajectory generation. The method was evaluated in physical experiments in terms of motion generation and recognition. In the experiments, users demonstrated the manipulation of puppets and toys so that the motions could be learned. A recognition accuracy of 90 was obtained for a test set of motions performed by three subjects. Furthe...