In this paper, we introduce a method to automatically reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person and the object, contact positions, and forces and torques actuated by the human limbs. The main contributions of this work are three-fold. First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of their interactions. This is cast as a large-scale trajectory optimization problem. Second, we develop a method to automatically recognize from the input video the position and timing of contacts between the person and the object or the ground, thereby significantly simplifying the complexity of the optimization. Third, we validate our approach on a recent MoCap dataset with ground truth contact forces and demonstrate its performance on a new dataset of Internet videos showing people manipulating a variety of tools in unconstrained environments.
There is also related work on modeling person-object interactions in robotics @cite_13 and computer animation @cite_66 . Similarly to people, humanoid robots interact with the environment by creating and breaking contacts @cite_7 , for example, during walking. Typically, generating artificial motion is formulated as an optimal control problem, transcribed into a high-dimensional numerical optimization problem, seeking to minimize an objective function under contact and feasibility constraints @cite_4 @cite_44 . A known difficulty is handling the non-smoothness of the resulting optimization problem introduced by the creation and breaking of contacts @cite_22 . Due to this difficulty, the sequence of contacts is often computed separately and not treated as a decision variable in the optimizer @cite_65 @cite_27 . Recent work has shown that it may be possible to decide both the continuous movement and the contact sequence together, either by implicitly formulating the contact constraints @cite_5 or by using invariances to smooth the resulting optimization problem @cite_58 @cite_37 .
@cite_37 @cite_4 @cite_22 @cite_7 @cite_65 @cite_44 @cite_27 @cite_5 @cite_58 @cite_13 @cite_66
2331982411 2415458238 2570393549 2411183766
In this letter, we propose a two-stage strategy for optimal control problems of robotic mechanical systems that proves to be more robust, and yet more efficient, than straightforward solution strategies. Specifically, we focus on a simplified humanoid model, represented as a two-dimensional articulated serial chain of rigid bodies, in the tasks of getting up (sitting down) from (to) the supine and prone postures. Interactions with the environment are integral parts of these motions, and a priori unscheduled contact sequences are discovered by the solver itself, opportunistically making or breaking contacts with the ground through feet, knees, hips, elbows, and hands. The present investigation analyzes the effects on the computational performance of: 1) the explicit introduction of contact forces among the optimization variables, 2) the substitution of undesired contact forces with geometric constraints that prevent interpenetrations, and 3) the splitting of the planning problem into two consecutive phases of increasing complexity. To the best of our knowledge, these tests represent the only quantitative analysis of the performances achievable with different solution strategies for optimization-based, whole-body dynamic motion planning in the presence of contacts. This paper presents a generic and efficient approach to generate dynamically consistent motions for under-actuated systems like humanoid or quadruped robots. The main contribution is a walking pattern generator, able to compute a stable trajectory of the center of mass of the robot along with the angular momentum, for any given configuration of contacts (e.g. on uneven, sloppy or slippery terrain, or with closed-gripper). Unlike existing methods, our solver is fast enough to be applied as a model-predictive controller. We then integrate this pattern generator in a complete framework: an acyclic contact planner is first used to automatically compute the contact sequence from a 3D model of the environment and a desired final posture; a stable walking pattern is then computed by the proposed solver; a dynamically-stable whole-body trajectory is finally obtained using a second-order hierarchical inverse kinematics. The implementation of the whole pipeline is fast enough to plan a step while the previous one is executed. The interest of the method is demonstrated by real experiments on the HRP-2 robot, by performing long-step walking and climbing a staircase with handrail support. We propose a humanoid robot navigation planning framework that reuses previous experience to decrease planning time. The framework is intended for navigating complex unstructured environments using both palm and foot contacts. In a complex environment, discrete-search-based contact space planners trade-off between high branching factor and action flexibility. Although approaches such as weighted A∗, ARA∗ and ANA∗ could speed up the search by compromising on optimality, they can be very slow when the heuristic is inaccurate. In the proposed framework, an experience-retrieval module is added in parallel to ANA∗. This module collects previously-generated motion plans and clusters them based on contact pose similarity to form a motion plan library. To retrieve an appropriate plan from the library for a given environment, the framework uses a distance between the contact poses in the plan and environment surfaces. Candidate plans are then modified with local trajectory optimization until a plan fitting the query environment is found. Our experiments show that the proposed framework outperforms planning-from-scratch in success rate in unstructured environments by at least 28 and can navigate difficult environments such as rubble and narrow corridors. Most animal and human locomotion behaviors for solving complex tasks involve dynamic motions and rich contact interaction. In fact, complex maneuvers need to consider dynamic movement and contact events at the same time. We present a hierarchical trajectory optimization approach for planning dynamic movements with unscheduled contact sequences. We compute whole-body motions that achieve goals that cannot be reached in a kinematic fashion. First, we find a feasible CoM motion according to the centroidal dynamics of the robot. Then, we refine the solution by applying the robot's full-dynamics model, where the feasible CoM trajectory is used as a warm-start point. To accomplish the unscheduled contact behavior, we use complementarity constraints to describe the contact model, i.e. environment geometry and non-sliding active contacts. Both optimization phases are posed as Mathematical Program with Complementarity Constraints (MPCC). Experimental trials demonstrate the performance of our planning approach in a set of challenging tasks.