The Dense Trajectories concept is one of the most successful approaches in action recognition, suitable for scenarios involving a significant amount of motion. However, due to noise and background motion, many generated trajectories are irrelevant to the actual human activity and can potentially lead to performance degradation. In this paper, we propose Localized Trajectories as an improved version of Dense Trajectories where motion trajectories are clustered around human body joints provided by RGB-D cameras and then encoded by local Bag-of-Words. As a result, the Localized Trajectories concept provides a more discriminative representation of actions as compared to Dense Trajectories. Moreover, we generalize Localized Trajectories to 3D by using the modalities offered by RGB-D cameras. One of the main advantages of using RGB-D data to generate trajectories is that they include radial displacements that are perpendicular to the image plane. Extensive experiments and analysis are carried out on five different datasets.
Finally, @cite_47 had the idea of focusing on trajectory groups which contribute more importantly to a specific action by defining an optimization problem. Towards the same direction, @cite_37 proposed the extraction of features around joint trajectories, increasing the discriminative power of the original Dense Trajectories approach @cite_58 . Although all the aforementioned methods have shown their effectiveness, they unfortunately lack locality information related to the human body. This piece of information is crucial when actions include similar motion patterns performed by different body parts. For this reason, we propose a novel dense trajectory-based approach by taking into consideration the local spatial repartition of motion with respect of the human body.
@cite_47 @cite_58 @cite_37
1932274155 2613401634 2620105270 2067977053 2587415290 2107221262 2543099233 2101340954 2201235033 2774366155 1973595287 2418222507 2162235402 1982838671 2012451022 2796664602 2054661770 2010985691 2964304299 2004972618
Dense local trajectories have been successfully used in action recognition. However, for most actions only a few local motion features (e.g., critical movement of hand, arm, leg etc.) are responsible for the action label. Therefore, highlighting the local features which are associated with important motion parts will lead to a more discriminative action representation. Inspired by recent advances in sentence regularization for text classification, we introduce a Motion Part Regularization framework to mine for discriminative groups of dense trajectories which form important motion parts. First, motion part candidates are generated by spatio-temporal grouping of densely extracted trajectories. Second, an objective function which encourages sparse selection for these trajectory groups is formulated together with an action class discriminative term. Then, we propose an alternative optimization algorithm to efficiently solve this objective function by introducing a set of auxiliary variables which correspond to the discriminativeness weights of each motion part (trajectory group). These learned motion part weights are further utilized to form a discriminativeness weighted Fisher vector representation for each action sample for final classification. The proposed motion part regularization framework achieves the state-of-the-art performances on several action recognition benchmarks. We study the sparsity and optimality properties of crowd navigation and find that existing techniques do not satisfy both criteria simultaneously: either they achieve optimality with a prohibitive number of samples or tractability assumptions make them fragile to catastrophe. For example, if the human and robot are modeled independently, then tractability is attained but the planner is prone to overcautious or overaggressive behavior. For sampling based motion planning of joint human-robot cost functions, for @math agents and @math step lookahead, @math samples are needed for coverage of the action space. Advanced approaches statically partition the action space into free-space and then sample in those convex regions. However, if the human is into free-space, then the partition is misleading and sampling is unsafe: free space will soon be occupied. We diagnose the cause of these deficiencies---optimization happens over space---and propose a novel solution: optimize over trajectory space by using a Gaussian process (GP) basis. We exploit the "kernel trick" of GPs, where a continuum of trajectories are captured with a mean and covariance function. By using the mean and covariance as proxies for a trajectory family we reason about collective trajectory behavior without resorting to sampling. The GP basis is sparse and optimal with respect to collision avoidance and robot and crowd intention and flexibility. GP sparsity leans heavily on the insight that joint action space decomposes into free regions; however, the decomposition contains feasible solutions only if the partition is dynamically generated. We call our approach . In this paper we propose an approach for articulated tracking of multiple people in unconstrained videos. Our starting point is a model that resembles existing architectures for single-frame pose estimation but is substantially faster. We achieve this in two ways: (1) by simplifying and sparsifying the body-part relationship graph and leveraging recent methods for faster inference, and (2) by offloading a substantial share of computation onto a feed-forward convolutional architecture that is able to detect and associate body joints of the same person even in clutter. We use this model to generate proposals for body joint locations and formulate articulated tracking as spatio-temporal grouping of such proposals. This allows to jointly solve the association problem for all people in the scene by propagating evidence from strong detections through time and enforcing constraints that each proposal can be assigned to one person only. We report results on a public MPII Human Pose benchmark and on a new MPII Video Pose dataset of image sequences with multiple people. We demonstrate that our model achieves state-of-the-art results while using only a fraction of time and is able to leverage temporal information to improve state-of-the-art for crowded scenes. The seemingly simple everyday actions of moving limb and body to accomplish a motor task or interact with the environment are incredibly complex. To reach for a target we first need to sense the target's position with respect to an external coordinate system; we then need to plan a limb trajectory which is executed by issuing an appropriate series of neural commands to the muscles. These, in turn, exert appropriate forces and torques on the joints leading to the desired movement of the arm. Here we review some of the earlier work as well as more recent studies on the control of human movement, focusing on behavioral and modeling studies dealing with task space and joint-space movement planning. At the task level, we describe studies investigating trajectory planning and inverse kinematics problems during point-to-point reaching movements as well as two-dimensional (2D) and three-dimensional (3D) drawing movements. We discuss models dealing with the two-thirds power law, particularly differential geometrical approaches dealing with the relation between path geometry and movement velocity. We also discuss optimization principles such as the minimum-jerk model and the isochrony principle for point-to-point and curved movements. We next deal with joint-space movement planning and generation, discussing the inverse kinematics problem and common solutions to the problems of kinematic redundancy. We address the question of which reference frames are used by the nervous system and review studies examining the employment of kinematic constraints such as Donders' and Listing's laws. We also discuss optimization approaches based on Riemannian geometry. One principle of motor coordination during human locomotion emerging from this body of work is the intersegmental law of coordination. However, the nature of the coordinate systems underlying motion planning remains of interest as they are related to the principles governing the control of human arm movements. There is extensive literature on using convex optimization to derive piece-wise polynomial trajectories for controlling differential flat systems with applications to three-dimensional flight for Micro Aerial Vehicles. In this work, we propose a method to formulate trajectory generation as a quadratic program (QP) using the concept of a Safe Flight Corridor (SFC). The SFC is a collection of convex overlapping polyhedra that models free space and provides a connected path from the robot to the goal position. We derive an efficient convex decomposition method that builds the SFC from a piece-wise linear skeleton obtained using a fast graph search technique. The SFC provides a set of linear inequality constraints in the QP allowing real-time motion planning. Because the range and field of view of the robot's sensors are limited, we develop a framework of Receding Horizon Planning , which plans trajectories within a finite footprint in the local map, continuously updating the trajectory through a re-planning process. The re-planning process takes between 50 to 300 ms for a large and cluttered map. We show the feasibility of our approach, its completeness and performance, with applications to high-speed flight in both simulated and physical experiments using quadrotors. Assuming that a robot trajectory is given from a high-level planning or learning mechanism, it needs to be adapted to react to dynamic environment changes. In this article we propose a novel approach to deform trajectories while keeping their local shape similar, which is based on the discrete Laplace---Beltrami operator. The approach can be readily extended and covers multiple deformation techniques including fixed waypoints that must be passed, positional constraints for collision avoidance or a cooperative manipulation scheme for the coordination of multiple robots. Due to its low computational complexity it allows for real-time trajectory deformation both on local and global scale and online adaptation to changed environmental constraints. Simulations illustrate the straightforward combination of the proposed approach with other established trajectory-related methods like artificial potential fields or prioritized inverse kinematics. Experiments with the HRP-4 humanoid successfully demonstrate the applicability in complex daily-life tasks. This paper addresses the problem of understanding the shape of the locomotor trajectories for a human being walking in an empty space to reach a goal defined both in position and in direction. Among all the possible trajectories reaching a given goal what are the fundamental reasons to chose one trajectory instead of another (see Fig. (1))? The underlying idea to attack this question has been to relate this problem to an optimal control problem: the trajectory is chosen according to some optimization principle. This is our basic starting assumption. The subject being viewed as a controlled system, the question becomes what criteria is optimized? Is it the time to perform the trajectory? the length of the path? the minimum jerk along the path?... In this study we show that the human locomotor trajectories are well approximated by the geodesics of a differential system minimizing the L2 norm of the control. Such geodesics are made of arcs of clothoids. The clothoid or Cornu spiral is a curve, whose curvature grows with the distance from the origin. The study is based on an experimental protocol involving 7 subjects. They had to walk within a motion capture room from a fixed starting point, and to cross over distant porches from which both position in the room and orientation were changing over trials. Direct methods for trajectory optimization are widely used for planning locally optimal trajectories of robotic systems. Many critical tasks, such as locomotion and manipulation, often involve impacting the ground or objects in the environment. Most state-of-the-art techniques treat the discontinuous dynamics that result from impacts as discrete modes and restrict the search for a complete path to a specified sequence through these modes. Here we present a novel method for trajectory planning of rigid-body systems that contact their environment through inelastic impacts and Coulomb friction. This method eliminates the requirement for a priori mode ordering. Motivated by the formulation of multi-contact dynamics as a Linear Complementarity Problem for forward simulation, the proposed algorithm poses the optimization problem as a Mathematical Program with Complementarity Constraints. We leverage Sequential Quadratic Programming to naturally resolve contact constraint forces while simultaneously optimizing a trajectory that satisfies the complementarity constraints. The method scales well to high-dimensional systems with large numbers of possible modes. We demonstrate the approach on four increasingly complex systems: rotating a pinned object with a finger, simple grasping and manipulation, planar walking with the Spring Flamingo robot, and high-speed bipedal running on the FastRunner platform. In this paper we consider the motion planning problem of humanoid with articulated-spine for object manipulation in double support phase. Complexity of motion planning increases due to higher degrees-of-freedom (DOF), redundancy arising due to articulated spine, inherent underactuation and stability constraint. Additionally, loop-closure constraints arise during double-support phase, and self-collision constraints exist independent of the task to be performed. These problems make motion planning of humanoids a challenging task. In this work, we address the above issues by proposing a sampling based approach for planning the motion of the humanoid. Our approach to the problem is based on RRT* which can generate asymptotically optimal paths. The proposed approach deals with the stability constraints by rejective sampling approach which divides random configurations generated into valid and invalid configurations. Additionally, the loop-closure constraints are tackled by separating the humanoid into two open-kinematic sub-chains, and then generating random configurations in one sub-chain whereas the remaining sub-chain uses inverse kinematics for closure. Efficacy of the proposed approach is demonstrated for whole-body motion planning of a 25-DOF humanoid in generating asymptotically optimal end-effector paths. Traditional motion planning approaches for multilegged locomotion divide the problem into several stages, such as contact search and trajectory generation. However, reasoning about contacts and motions simultaneously is crucial for the generation of complex whole-body behaviors. Currently, coupling theses problems has required either the assumption of a fixed gait sequence and flat terrain condition, or nonconvex optimization with intractable computation time. In this letter, we propose a mixed-integer convex formulation to plan simultaneously contact locations, gait transitions, and motion, in a computationally efficient fashion. In contrast to previous works, our approach is not limited to flat terrain nor to a prespecified gait sequence. Instead, we incorporate the friction cone stability margin, approximate the robot's torque limits, and plan the gait using mixed-integer convex constraints. We experimentally validated our approach on the HyQ robot by traversing different challenging terrains, where nonconvexity and flat terrain assumptions might lead to suboptimal or unstable plans. Our method increases the motion robustness while keeping a low computation time. Previous research has shown that inertial cues resulting from passive transport through a large environment do not necessarily facilitate acquiring knowledge about its layout. Here we examine whether the additional body-based cues that result from active movement facilitate the acquisition of spatial knowledge. Three groups of participants learned locations along an 840-m route. One group walked the route during learning, allowing access to body-based cues (i.e., vestibular, proprioceptive, and efferent information). Another group learned by sitting in the laboratory, watching videos made from the first group. A third group watched a specially made video that minimized potentially confusing head-on-trunk rotations of the viewpoint. All groups were tested on their knowledge of directions in the environment as well as on its configural properties. Having access to body-based information reduced pointing error by a small but significant amount. Regardless of the sensory information available during learning, participants exhibited strikingly common biases. To enable safe and efficient human–robot collaboration in shared workspaces, it is important for the robot to predict how a human will move when performing a task. While predicting human motion for tasks not known a priori is very challenging, we argue that single-arm reaching motions for known tasks in collaborative settings (which are especially relevant for manufacturing) are indeed predictable. Two hypotheses underlie our approach for predicting such motions: First, that the trajectory the human performs is optimal with respect to an unknown cost function, and second, that human adaptation to their partner's motion can be captured well through iterative replanning with the above cost function. The key to our approach is thus to learn a cost function that “explains” the motion of the human. To do this, we gather example trajectories from pairs of participants performing a collaborative assembly task using motion capture. We then use inverse optimal control to learn a cost function from these trajectories. Finally, we predict reaching motions from the human's current configuration to a task-space goal region by iteratively replanning a trajectory using the learned cost function. Our planning algorithm is based on the trajectory optimizer: stochastic trajectory optimizer for motion planning [1] ; it plans for a 23-degree-of-freedom human kinematic model and accounts for the presence of a moving collaborator and obstacles in the environment. Our results suggest that in most cases, our method outperforms baseline methods when predicting motions. We also show that our method outperforms baselines for predicting human motion when a human and a robot share the workspace. Trajectory and kinematics of drawing movements are mutually constrained by functional relationships that reduce the degrees of freedom of the hand-arm system. Previous investigations of these relationships are extended here by considering their development in children between 5 and 12 years of age. Performances in a simple motor task—the continuous tracing of elliptic trajectories—demonstrate that both the phenomenon of isochrony (increase of the average movement velocity with the linear extent of the trajectory) and the so-called two-thirds power law (relation between tangential velocity and curvature) are qualitatively present already at the age of 5. The quantitative aspects of these regularities evolve with age, however, and steady-state adult performance is not attained even by the oldest children. The power-law formalism developed in previous reports is generalized to encompass these developmental aspects of the control of movement. Two general frameworks are currently available to conceptualize the motor-control problem. Broadly, the two frameworks differ in the answer that they give to the question "Where do form and structure come from?" According to the motor-program view (e.g., Schmidt, 1988), form and structure in a movement come from a central abstract representation of the intended result. Thus, making a movement entails a detailed mapping of this representation into an appropriate motor plan. The competing task-dynamic view (e.g., Kugler, 1989; Saltzman & Kelso, 1987) emphasizes instead the global morphogenetic power of the nonlinearities of the motor system. These two views need not be mutually exclusive. In fact, different classes of movements may require different conceptual frameworks for their analysis. At any rate (at least in the case of drawing movements considered here), an internal representation of the intended trajectory and kinematics must be available before execution. Consequently, much effort is being devoted to understanding how this internal model is translated into a set of motor commands. In this respect, one basic conceptual difficulty is because the geometry of the arm-forearm-hand system affords a large number of degrees of freedom. Provided that there are no constraints in the workspace, the only trajectory that matters in many manual Location is a key context ingredient and many existing pervasive applications rely on the current locations of their users. However, with the ability to predict the future location and movement behavior of a user, the usability of these applications can be greatly improved. In this paper, we propose an approach to predict both the intended destination and the future route of a person. Rather than predicting the destination and future route separately, we have focused on making prediction in an integrated way by exploiting personal movement data (i.e. trajectories) collected by GPS. Since trajectories contain daily whereabouts information of a person, the proposed approach first detects the significant places where the person may depart from or go to using a clustering-based algorithm called FBM (Forward-Backward Matching), then abstracts the trajectories based on a space partitioning method, and finally extracts movement patterns from the abstracted trajectories using an extended CRPM (Continuous Route Pattern Mining) algorithm. Extracted movement patterns are organized in terms of origin-destination couples. The prediction is made based on a pattern tree built from these movement patterns. With the real personal movement data of 14 participants, we conducted a number of experiments to evaluate the performance of our system. The results show that our approach can achieve approximately 80 and 60 accuracy in destination prediction and 1-step prediction, respectively, and result in an average deviation of approximately 60 m in continuous future route prediction. Finally, based on the proposed approach, we implemented a prototype running on mobile phones, which can extract patterns from a user's historical movement data and predict the destination and future route. We present a new method to classify human activities by leveraging on the cues available from depth images alone. Towards this end, we propose a descriptor which couples depth and spatial information of the segmented body to describe a human pose. Unique poses (i.e. codewords) are then identified by a spatial-based clustering step. Given a video sequence of depth images, we segment humans from the depth images and represent these segmented bodies as a sequence of codewords. We exploit unique poses of an activity and the temporal ordering of these poses to learn subsequences of codewords which are strongly discriminative for the activity. Each discriminative subsequence acts as a classifier and we learn a boosted ensemble of discriminative subsequences to assign a confidence score for the activity label of the test sequence. Unlike existing methods which demand accurate tracking of 3D joint locations or couple depth with color image information as recognition cues, our method requires only the segmentation masks from depth images to recognize an activity. Experimental results on the publicly available Human Activity Dataset (which comprises 12 challenging activities) demonstrate the validity of our method, where we attain a precision recall of 78.1 75.4 when the person was not seen before in the training set, and 94.6 93.1 when the person was seen before. We propose a novel network that learns a part-aligned representation for person re-identification. It handles the body part misalignment problem, that is, body parts are misaligned across human detections due to pose viewpoint change and unreliable detection. Our model consists of a two-stream network (one stream for appearance map extraction and the other one for body part map extraction) and a bilinear-pooling layer that generates and spatially pools a part-aligned map. Each local feature of the part-aligned map is obtained by a bilinear mapping of the corresponding local appearance and body part descriptors. Our new representation leads to a robust image matching similarity, which is equivalent to an aggregation of the local similarities of the corresponding body parts combined with the weighted appearance similarity. This part-aligned representation reduces the part misalignment problem significantly. Our approach is also advantageous over other pose-guided representations (e.g., extracting representations over the bounding box of each body part) by learning part descriptors optimal for person re-identification. For training the network, our approach does not require any part annotation on the person re-identification dataset. Instead, we simply initialize the part sub-stream using a pre-trained sub-network of an existing pose estimation network, and train the whole network to minimize the re-identification loss. We validate the effectiveness of our approach by demonstrating its superiority over the state-of-the-art methods on the standard benchmark datasets, including Market-1501, CUHK03, CUHK01 and DukeMTMC, and standard video dataset MARS. The main objective of this paper is to develop an efficient method for learning and reproduction of complex trajectories for robot programming by demonstration. Encoding of the demonstrated trajectories is performed with hidden Markov model, and generation of a generalized trajectory is achieved by using the concept of key points. Identification of the key points is based on significant changes in position and velocity in the demonstrated trajectories. The resulting sequences of trajectory key points are temporally aligned using the multidimensional dynamic time warping algorithm, and a generalized trajectory is obtained by smoothing spline interpolation of the clustered key points. The principal advantage of our proposed approach is utilization of the trajectory key points from all demonstrations for generation of a generalized trajectory. In addition, variability of the key points' clusters across the demonstrated set is employed for assigning weighting coefficients, resulting in a generalization procedure which accounts for the relevance of reproduction of different parts of the trajectories. The approach is verified experimentally for trajectories with two different levels of complexity. In this paper, we deal with the estimation of body and head poses (i.e orientations) in surveillance videos, and we make three main contributions. First, we address this issue as a joint model adaptation problem in a semi-supervised framework. Second, we propose to leverage the adaptation on multiple information sources (external labeled datasets, weak labels provided by the motion direction, data structure manifold), and in particular, on the coupling at the output level of the head and body classifiers, accounting for the restriction in the configurations that the head and body pose can jointly take. Third, we propose a kernel-formulation of this principle that can be efficiently solved using a global optimization scheme. The method is applied to body and head features computed from automatically extracted body and head location tracks. Thorough experiments on several datasets demonstrate the validity of our approach, the benefit of the coupled adaptation, and that the method performs similarly or better than a state-of-the-art algorithm. In this paper, we address the problem of person re-identification, which refers to associating the persons captured from different cameras. We propose a simple yet effective human part-aligned representation for handling the body part misalignment problem. Our approach decomposes the human body into regions (parts) which are discriminative for person matching, accordingly computes the representations over the regions, and aggregates the similarities computed between the corresponding regions of a pair of probe and gallery images as the overall matching score. Our formulation, inspired by attention models, is a deep neural network modeling the three steps together, which is learnt through minimizing the triplet loss function without requiring body part labeling information. Unlike most existing deep learning algorithms that learn a global or spatial partition-based local representation, our approach performs human body partition, and thus is more robust to pose changes and various human spatial distributions in the person bounding box. Our approach shows state-of-the-art results over standard datasets, Market-1501, CUHK03, CUHK01 and VIPeR. 1 In this paper we introduce a probabilistic approach on multiple person localization using multiple calibrated camera views. People present in the scene are approximated by a population of cylinder objects in the 3-D world coordinate system, which is a realization of a Marked Point Process. The observation model is based on the projection of the pixels of the obtained motion masks in the different camera images to the ground plane and to other parallel planes with different height. The proposed pixel-level feature is based on physical properties of the 2-D image formation process and can accurately localize the leg position on the ground plane and estimate the height of the people, even if the area of interest is only a part of the scene, meanwhile silhouettes from irrelevant outside motions may significantly overlap with the monitored region in some of the camera views. We introduce an energy function, which contains a data term calculated from the extracted features and a geometrical constraint term modeling the distance between two people. The final configuration results (location and height) are obtained by an iterative stochastic energy optimization process, called the Multiple Birth and Death dynamics. The proposed approached is compared to a recent state-of-the-art technique in a publicly available dataset and its advantages are quantitatively demonstrated.