We present a novel method for lip syncing. The method obtains a sizable gap in accuracy over the literature on multiple benchmarks. Applying the method, even in challenging scenarios where the audio and the video do not perfectly match, results in an artifact-free, well-aligned video. As we demonstrate, users find it hard to distinguish, based on either quality or the accuracy of the audio-video alignment, between the modified video and the original unmodified video. While we have demonstrated compelling results, further improvements may be considered.
In order to eliminate the post-processing step,
a monotonic attention mechanism [1]}, [2]}, [3]},
may be utilized, in order to force the decoder to produce a monotonically increasing alignment.
In addition, monotonic constraints could be added to the loss function, as well as using an Euclidean or an ordinal loss function, in order to weight the magnitude of the prediction error. Our code and scripts for training and running the method are attached as supplementary and would be released as open source.
