In this work, we propose to integrate physics-based cloth simulation into avatar modeling, so that the clothing on the avatar can be animated photorealistically with the body, while achieving high-quality dynamics, collision handling and capabilities to animate and render avatars with novel clothing. Our work builds upon full-body Codec Avatars [1]}, [2]}, which leverage a Variational Autoencoder (VAE) to model the geometry and appearance of a human. In particular, we follow the multi-layer formulation of [2]}, but redesign the clothing layer to integrate a physically-based simulator. Namely, at training stage, we learn the clothing appearance model using real-world data, by processing raw captures with our dynamic clothing registration pipeline. At test time, we simulate the clothing geometry on top of the underlying body model with appropriate material parameters, and then apply the learned appearance model to synthesize the final output.
