A related task is that of reconstructing a 3D mesh animation, or generating a video of a talking face, which is aligned to an input audio signal. Karras et al. predict 3D vertex coordinates of a target face model, given a waveform of any source speaker, and aim to control the emotional states of the generated face [1]}. ObamaNet [2]} uses a recurrent neural network (RNN) in order to predict a PCA representation of normalized mouth key-points, given a raw waveform created by a text to speech module. The key-points are then used to condition a generator that synthesizes a video of a target identity. Suwajanakorn et al. trained an RNN on many hours of Obama's weekly address footage to map from the MFCC representation of the input audio stream into mouth shapes [3]}. Then, they synthesized a talking face video of Obama. Son Chung et al. proposed an encoder-decoder model that learns the joint embeddings of a given audio and a target face images, and then generates a lip-synced talking face video [4]}.
