We present a real-time method for synthesizing highly complex human motions using a novel training regime we call the auto-conditioned Recurrent Neural Network (acRNN). Recently, researchers have attempted to synthesize new motion by using autoregressive techniques, but existing methods tend to freeze or diverge after a couple of seconds due to an accumulation of errors that are fed back into the network. Furthermore, such methods have only been shown to be reliable for relatively simple human motions, such as walking or running. In contrast, our approach can synthesize arbitrary motions with highly complex styles, including dances or martial arts in addition to locomotion. The acRNN is able to accomplish this by explicitly accommodating for autoregressive noise accumulation during training. Our work is the first to our knowledge that demonstrates the ability to generate over 18,000 continuous frames (300 seconds) of new complex human motion w.r.t. different styles. The synthesis of realistic human motion has recently seen increased interest BID12 BID39 BID4 BID14 BID1 BID25 with applications beyond animation and video games. The simulation of human looking virtual agents is likely to become mainstream with the dramatic advancement of Artificial Intelligence and the democratization of Virtual Reality. A challenge for human motion synthesis is to automatically generate new variations of motions while preserving a certain style, e.g., generating large numbers of different Bollywood dances for hundreds of characters in an animated scene of an Indian party. Aided by the availability of large human-motion capture databases, many database-driven frameworks have been employed to this end, including motion graphs BID18 BID33 BID27 , as well as linear BID34 BID2 BID36 and kernel methods BID29 BID31 BID8 BID28 BID42 , which blend key-frame motions from a database. It is hard for these methods, however, to add new variations to existing motions in the database while keeping the style consistent. This is especially true for motions with a complex style such as dancing and martial arts. More recently, with the rapid development in deep learning, people have started to use neural networks to accomplish this task BID13 . These works have shown promising results, demonstrating the ability of using high-level parameters (such as a walking-path) to synthesize locomotion tasks such as jumping, running, walking, balancing, etc. These networks do not generate new variations of complex motion, however, being instead limited to specific use cases.In contrast, our paper provides a robust framework that can synthesize highly complex human motion variations of arbitrary styles, such as dancing and martial arts, without querying a database. We achieve this by using a novel deep auto-conditioned RNN (acRNN) network architecture.Recurrent neural networks are autoregressive deep learning frameworks which seek to predict sequences of data similar to a training distribution. Such a framework is intuitive to apply to human motion, which can be naturally modeled as a time series of skeletal joint positions. We are not the first to leverage RNNs for this task BID4 BID14 BID1 BID25 , and these works produce reasonably realistic output at a number of tasks such as sitting, talking, smoking, etc. However, these existing methods also have a critical drawback: the motion becomes unrealistic within a couple of seconds and is unable to recover.This issue is commonly attributed to error accumulation due to feeding network output back into itself BID13 . This is reasonable, as the network during training is given ground-truth input sequences to condition its subsequent guess, but at run time, must condition this guess on its own output. As the output distribution of the network will not be identical to that of the ground-truth, it is in effect encountering a new situation at test-time. The acRNN structure compensates for this by linking the network's own predicted output into its future input streams during training, a similar approach to the technique proposed in BID0 . Our method is light-weight and can be used in conjunction with any other RNN based learning scheme. Though straightforward, this technique fixes the issue of error accumulation, and allows the network to output incredibly long sequences without failure, on the order of hundreds of seconds (see Figure 5 ). Though we are yet as unable to prove the permanent stability of this structure, it seems empirically that motion can be generated without end. In summary, we present a new RNN training method capable for the first time of synthesizing potentially indefinitely long sequences of realistic and complex human motions with respect to different styles. We have shown the effectiveness of the acLSTM architecture to produce extended sequences of complex human motion. We believe our work demonstrates qualitative state-of-the-art results in motion generation, as all previous work has focused on synthesizing relatively simple human motion for extremely short time periods. These works demonstrate motion generation up to a couple of seconds at most while acLSTM does not fail even after over 300 seconds. Though we are as of yet unable to prove indefinite stability, it seems empirically that acLSTM can generate arbitrarily long sequences. Current problems that exist include choppy motion at times, self-collision of the skeleton, and unrealistic sliding of the feet. Further developement of GAN methods, such as BID19 , could result in increased realism, though these models are notoriously hard to train as they often result in mode collapse. Combining our technique with physically based simulation to ensure realism after synthesis is also a potential next step. Finally, it is important to study the effects of using various condition lengths during training. We begin the exploration of this topic in the appendix, but further analysis is needed. Figure 9 might imply some sort of trade off between motion change over time and short-term motion prediction error when training with different condition lengths. However, it is also possible that limiting motion magnitude on this particular dataset might correspond to lower error. Further experiments of various condition lengths on several motion styles need to be conducted to say anything meaningful about the effect.C VISUAL DIAGRAM OF AUTO-CONDITIONED LSTM
Synthesize complex and extended human motions using an auto-conditioned LSTM network
{"src": "scitldr-AIC_train", "paper_id": "r11Q2SlRW"}