Recently,  neural-network  based  forward  dynamics  models  have  been  proposed that  attempt  to  learn  the  dynamics  of  physical  systems  in  a  deterministic  way. While near-term motion can be predicted accurately, long-term predictions suffer from accumulating input and prediction errors which can lead to plausible but different trajectories that diverge from the ground truth.   A system that predicts distributions of the future physical states for long time horizons based on its uncertainty is thus a promising solution.   In this work, we introduce a novel robust Monte Carlo sampling based graph-convolutional dropout method that allows us to sample multiple plausible trajectories for an initial state given a neural-network based forward dynamics predictor.   By introducing a new shape preservation loss and training our dynamics model recurrently, we stabilize long-term predictions. We show that our model’s long-term forward dynamics prediction errors on complicated physical interactions of rigid and deformable objects of various shapes are significantly lower than existing strong baselines. Lastly, we demonstrate how generating multiple trajectories with our Monte Carlo dropout method can be used to train model-free reinforcement learning agents faster and to better solutions on simple manipulation tasks. Small errors in the input and prediction can lead to significantly different object trajectories. The orange ball could either end up on the left or right side of the wedge. Learning to predict the physical motion of objects from data is an open area of research. Yet, recent (hierarchical) relation network based forward dynamics predictors (Battaglia et al., 2016; Chang et al., 2016; Li et al., 2019) seem to be a promising alternative to conventional physics engines that are key components of robot control, computer vision and reinforcement learning (RL) systems. Physics simulators, both traditional numerical solvers and learned prediction models, still suffer from insufficient accuracy in challenging scenarios. Small errors in the input and model can lead to dramatically different object trajectories. Take the orange ball that is falling on the blue wedge in Figure 1 . Depending on where the orange ball starts or what bias the model has, the ball could either end up on the left or right side. Both are valid outcomes. However, deterministic physics engines will either predict one trajectory or the other. While it is important to reduce errors in each prediction, it is also important to acknowledge that uncertain situations might not have one but multiple possible outcomes. In machine learning, uncertainty-aware neural networks avoid deterministic point estimates by predicting distributions or by randomly sampling in the prediction interval. In the context of dynamics predictions, we propose to use Monte Carlo sampling based dropout on the model weights of a learned forward dynamics predictor to model uncertainty and sample multiple plausible trajectories for an initial state. To stabilize each trajectory and reduce error accumulation over long-time horizons, we use a state-invariant recurrent training mechanism. By feeding back predictions as input over multiple time steps, the model becomes more robust to its own prediction errors without the need for a hidden state. Finally, we introduce a new shape loss on the model predictions that constrains the pairwise distances between objects and object parts and greatly improves shape preservation and the stability of trajectories over long-time horizons. Our final fully differentiable forward dynamics model is able to sample multiple, more accurate and more stable trajectories over long-time horizons compared to existing baselines. An accurate forward dynamics predictor that is able to predict a distribution of future states can be of great importance for robotic control. In model-free reinforcement learning, accomplishing tasks through random exploration is sample inefficient and hardly generalizable. Model-based methods promise greater generalization abilities, but suffer from deterministic world models that are hard to learn and fail in stochastic environments. With our stochastic forward dynamics predictor, we can move part of the sampling process into the environment, physically grounding the random exploration of model-free agents. As the agent is able to observe multiple trajectories at a given state without actually executing multiple actions, the sample efficiency is greatly improved while the stochasticity of each state and action is implicitly learned. We show on several control experiments that a model-free agent trained in our stochastic forward dynamics environment is not only able to better explore and learn faster but often also comes to better solutions than agents trained in deterministic environments. In summary, (1) we propose a stochastic differentiable forward dynamics model that is able to generate multiple plausible trajectories via Monte Carlo (MC) based graph-convolutional dropout. (2) We greatly improve the accuracy and stability of long-term predictions by proposing a new fullyconnected shape loss term and training the model recurrently end-to-end in a state-invariant way. (3) We demonstrate how our stochastic dynamics model can be used to improve the efficiency and performance of model-free reinforcement learning agents on several physical manipulation tasks. Qualitatively our stochastic HRN predicts plausible future trajectories; an experiment in which human subjects were asked to discriminate between ground-truth and predicted trajectories could be used to evaluate its performance quantitatively. Even though this method does not require extensive expert knowledge, a few design decisions have to be made e.g dropout rates for training and inference. During inference, too high of a dropout rate can lead to visually unrealistic dynamics and object interactions. Dropout rate scheduling during training should be investigated to improve convergence of the dynamics model during training, which may improve its performance as an environment for the reinforcement learning tasks. Possible optimizations include more complex, potentially non-linear, annealing schedules during inference, delaying the dropout rate annealing, and finding appropriate starting values. Finding a universal schedule that can be applied to any environment and task has large potential for accelerating reinforcement learning. Further improvements for the physics predictor are key for its use as a physical environment. These can include improvements for: scenarios with multiple materials in one scene, penetrations during collisions that can lead to insufficient position prediction, and generalization to new scenes. Our results show that the proposed sampling method produces physically plausible trajectories in single-and multi-object scenarios as well as across a range of materials. The quality of roll-outs, e.g. shape prediction is not compromised by the introduced noise. Furthermore, our model-free reinforcement learning experiments indicate that agents learning in physically stochastic environments are able to explore better and learn quicker, which confirms the quality of the sampled trajectories. In difficult reinforcement learning scenarios, where a high level of precision is needed to get a reward, we demonstrated that dropout rate annealing is an effective method to avoid too high randomness at the same time not reducing the benefits of stochasticity for exploration in early stages of the training. In this regard, stochastic neural physics engines offer a clear advantage over conventional physics engines.
We propose a stochastic differentiable forward dynamics predictor that is able to sample multiple physically plausible trajectories under the same initial input state and show that it can be used to train model-free policies more efficiently.
{"src": "scitldr-AIC_train", "paper_id": "HkgXteBYPB"}