We propose the use of a proportional-derivative (PD) control based policy learned via reinforcement learning (RL) to estimate and forecast 3D human pose from egocentric videos. The method learns directly from unsegmented egocentric videos and motion capture data consisting of various complex human motions (e.g., crouching, hopping, bending, and motion transitions). We propose a video-conditioned recurrent control technique to forecast physically-valid and stable future motions of arbitrary length. We also introduce a value function based fail-safe mechanism which enables our method to run as a single pass algorithm over the video data. Experiments with both controlled and in-the-wild data show that our approach outperforms previous art in both quantitative metrics and visual quality of the motions, and is also robust enough to transfer directly to real-world scenarios. Additionally, our time analysis shows that the combined use of our pose estimation and forecasting can run at 30 FPS, making it suitable for real-time applications.
The idea of using reference motions has existed for a long time in computer animation. Early work has applied this idea to bipedal locomotions with planar characters @cite_44 @cite_58 . Model-based methods @cite_56 @cite_10 @cite_57 generate locomotions with 3D humanoid characters by tracking reference motions. Sampling-based control methods @cite_36 @cite_67 @cite_16 have also shown great success in generating highly dynamic humanoid motions. DeepRL based approaches have utilized reference motions to shape the reward function @cite_38 @cite_17 . Approaches based on GAIL @cite_69 have also been proposed to eliminate the need for manual reward engineering @cite_5 @cite_52 @cite_35 . The work most relevant to ours is DeepMimic @cite_70 and its video variant @cite_45 . DeepMimic has shown beautiful results on human locomotion skills with manually designed reward and is able to combine learned skills to achieve different tasks. However, it is only able to learn skills from segmented motion clips and relies on the phase of motion as input to the policy. In contrast, our approach can learn from unsegmented MoCap data and use the visual context as a natural alternative to the phase variable.
@cite_38 @cite_67 @cite_69 @cite_35 @cite_36 @cite_70 @cite_52 @cite_16 @cite_56 @cite_44 @cite_57 @cite_45 @cite_5 @cite_58 @cite_10 @cite_17
2894766094 2201235033 2081773958 2007104354
Data-driven character animation based on motion capture can produce highly naturalistic behaviors and, when combined with physics simulation, can provide for natural procedural responses to physical perturbations, environmental changes, and morphological discrepancies. Motion capture remains the most popular source of motion data, but collecting mocap data typically requires heavily instrumented environments and actors. In this paper, we propose a method that enables physically simulated characters to learn skills from videos (SFV). Our approach, based on deep pose estimation and deep reinforcement learning, allows data-driven animation to leverage the abundance of publicly available video clips from the web, such as those from YouTube. This has the potential to enable fast and easy design of character controllers simply by querying for video recordings of the desired behavior. The resulting controllers are robust to perturbations, can be adapted to new settings, can perform basic object interactions, and can be retargeted to new morphologies via reinforcement learning. We further demonstrate that our method can predict potential human motions from still images, by forward simulation of learned controllers initialized from the observed pose. Our framework is able to learn a broad range of dynamic skills, including locomotion, acrobatics, and martial arts. (Video1) In this paper we consider the motion planning problem of humanoid with articulated-spine for object manipulation in double support phase. Complexity of motion planning increases due to higher degrees-of-freedom (DOF), redundancy arising due to articulated spine, inherent underactuation and stability constraint. Additionally, loop-closure constraints arise during double-support phase, and self-collision constraints exist independent of the task to be performed. These problems make motion planning of humanoids a challenging task. In this work, we address the above issues by proposing a sampling based approach for planning the motion of the humanoid. Our approach to the problem is based on RRT* which can generate asymptotically optimal paths. The proposed approach deals with the stability constraints by rejective sampling approach which divides random configurations generated into valid and invalid configurations. Additionally, the loop-closure constraints are tackled by separating the humanoid into two open-kinematic sub-chains, and then generating random configurations in one sub-chain whereas the remaining sub-chain uses inverse kinematics for closure. Efficacy of the proposed approach is demonstrated for whole-body motion planning of a 25-DOF humanoid in generating asymptotically optimal end-effector paths. In this paper we propose a novel method for human action recognition based on boosted key-frame selection and correlated pyramidal motion feature representations. Instead of using an unsupervised method to detect interest points, a Pyramidal Motion Feature (PMF), which combines optical flow with a biologically inspired feature, is extracted from each frame of a video sequence. The AdaBoost learning algorithm is then applied to select the most discriminative frames from a large feature pool. In this way, we obtain the top-ranked boosted frames of each video sequence as the key frames which carry the most representative motion information. Furthermore, we utilise the correlogram which focuses not only on probabilistic distributions within one frame but also on the temporal relationships of the action sequence. In the classification phase, a Support-Vector Machine (SVM) is adopted as the final classifier for human action recognition. To demonstrate generalizability, our method has been systematically tested on a variety of datasets and shown to be more effective and accurate for action recognition compared to the previous work. We obtain overall accuracies of: 95.5 , 93.7 , and 36.5 with our proposed method on the KTH, the multiview IXMAS and the challenging HMDB51 datasets, respectively. Tracking the articulated 3D motion of the hand has important applications, for example, in human-computer interaction and teleoperation. We present a novel method that can capture a broad range of articulated hand motions at interactive rates. Our hybrid approach combines, in a voting scheme, a discriminative, part-based pose retrieval method with a generative pose estimation method based on local optimization. Color information from a multi-view RGB camera setup along with a person-specific hand model are used by the generative method to find the pose that best explains the observed images. In parallel, our discriminative pose estimation method uses fingertips detected on depth data to estimate a complete or partial pose of the hand by adopting a part-based pose retrieval strategy. This part-based strategy helps reduce the search space drastically in comparison to a global pose retrieval strategy. Quantitative results show that our method achieves state-of-the-art accuracy on challenging sequences and a near-real time performance of 10 fps on a desktop computer.