FIELD

The present application relates to digital image processing, and more particularly to systems or methods for matching mouth shape and movement in digital video to alternative audio.

BACKGROUND

Producers of video content often desire to use recorded audio that does not match the images recorded in a motion picture. This can happen for several reasons including, for example, when the dialogue is changed after video capture for creative or legal reasons or when the dialog is recorded in a different language than was spoken by the actor during a performance for distribution to different audiences. Consumers of video content, however, often notice mouth movement that does not match the spoken dialog in timing or the shape of the mouth. Mismatch in either of these qualities can distract the consumer, reduce suspension of disbelief and diminish enjoyment of the video content.

Each frame of recorded video can be altered after the video capture so that the actors' mouth movements better align with accompanying speech and dialog in the audio track. However, methods for altering recorded video are too time-consuming and expensive to be feasible in most releases, diminishing the appeal, reach, and profitability of movies and other video content containing dialog that fails to match actor mouth movement and related facial movement. No prior approach attains a high enough quality at a cost feasible for distribution of cinematic or other commercial video content. Because of the great diversity in actors and image capture parameters used in commercial video, material available for training neural networks and other machine learning algorithms is seldom available in enough quantity to eliminate perceptible mismatch between dialog and mouth movement, and acquiring sufficient training video may be prohibitively expensive or impossible.

It would be desirable, therefore, to develop new methods and other new technologies for matching mouth shape and movement in digital video to alternative audio after initial capture of images and sound that overcomes these and other limitations of the prior art.

SUMMARY

This summary and the following detailed description should be interpreted as complementary parts of an integrated disclosure, which parts may include redundant subject matter and/or supplemental subject matter. An omission in either section does not indicate priority or relative importance of any element described in the integrated application. Differences between the sections may include supplemental disclosures of alternative embodiments, additional details, or alternative descriptions of identical embodiments using different terminology, as should be apparent from the respective disclosures.

In an aspect of the disclosure, a method for matching mouth shape and movement in digital video to alternative audio includes deriving a sequence of facial poses including mouth shapes for an actor from a source digital video in which the actor appears. Each shape in the sequence of shapes corresponds to a middle position of each sample of alternative audio. The method further includes generating an animated face mesh based on the sequence of facial poses and the source digital video. Optionally, the method may include generating a target video by rendering the animated face mesh. The method further includes transferring tracked expressions from at least one of the animated face mesh or the target video to the source video and generating a rough output video that includes transfers of the tracked expressions. The method further includes generating a finished video at least in part by refining the rough video using a parametric autoencoder trained on mouth shapes in animated face mesh or the target video. One or more computers may perform the operations of the method.

The methods described herein provide an automated process for modifying motion picture images or other video so that the mouth shapes and timing of mouth movement of people who appear in the images look more natural when accompanied by an audio recording that does not match speech that was recorded with the original video. Applications for the methods may include, for example, changing the dialog of video content in post-production, providing a translated dialog, or editing content after release to avoid giving offense due to changed cultural norms or other cultural diversity factors, to avoid legal liability, to improve creative impact or for any other reason.

An apparatus for performing the method may include at least one computer processor coupled to a memory and to one or more ports, including at least one input port and at least one output port (e.g., a desktop computer, laptop computer, tablet computer, smartphone, PDA, etc.). A computer processor may include, for example, a microprocessor, microcontroller, system on a chip, or other processing circuit. As used herein, a “processor” means a computer processor.

To the accomplishment of the foregoing and related ends, one or more examples comprise the features hereinafter fully described and particularly pointed out in the claims. The following description and the annexed drawings set forth in detail certain illustrative aspects and are indicative of but a few of the various ways in which the principles of the examples may be employed. Other advantages and novel features will become apparent from the following detailed description when considered in conjunction with the drawings and the disclosed examples, which encompass all such aspects and their equivalents.

