BACKGROUND

Media content creation tasks, such as video editing, audio mixing, and musical composition require a mixture of talents and skills. While some of these tasks require creative thinking, other parts of the media creation workflow involve little or no creativity and are often tedious and time-consuming. Some of these, especially the more repetitive and mechanical tasks, can be automated using various content creation tools, such as by using scripting methods. However, other tasks still need to be performed by humans, even when they involve relatively simple decisions. It is for these tasks, i.e., the tasks that are not purely mechanical in nature, but that do not require significant creative choices, that there exists a need to assist media content creators so that they can be freed from the time-consuming demands of such tasks.

Existing systems that use artificial intelligence (AI) methods to assist with content creation use machine learning components that were trained by the vendor and shipped as a static “black-box,” with no further opportunities for learning. Such systems are inherently restricted by the scope of the training data. As a result, the performance of such systems may be adversely impacted should a user wish to apply them to content not encompassed by the training. There is, therefore, a need for increased flexibility in the tools provided to media content creators so that they can be assisted in a wider range of contexts than initially envisaged by the provider of the tools.

SUMMARY

In general, the methods described herein enable media editing applications to edit media compositions automatically by using a deep-learning neural network model. The model is adapted to users of the media editing application by training the model with a training data set that includes compositions previously edited by the users themselves.

In general, in one aspect, an automatic media editing method includes: providing a computer-implemented media editing application that is able to edit raw media automatically to generate an edited media composition, wherein: the automatic editing comprises automatically generating a value of a parameter that defines an editing function that is applied to the raw media; the automatic editing is based on a trained third-party neural network model for the editing function, wherein the third-party neural network model was trained using a third-party training data set comprising, for each media composition of a plurality of media compositions edited by one or more third-party editors: an input vector comprising representative raw media extracted from raw media that was previously edited by the one or more third-party editors; and an output vector comprising a value specified by the one or more third-party editors of the parameter defining the editing function that was applied to the raw media during the previous editing; training a user-adapted neural network model based on a combination of the third-party training data set and a user-generated training data set, wherein the user-generated training data set comprises, for each media composition of a plurality of media compositions previously edited by a user of the media editing application: an input vector comprising representative raw media extracted from raw media that was previously edited by the user; and an output vector comprising a value specified by the user of the parameter defining the editing function that was applied to the raw media during the editing by the user; and the media editing application: receiving production raw media; extracting production representative raw media from the production raw media; inputting the production representative raw media to the trained user-adaptive neural network model to generate an output value of the parameter defining the editing function to be applied to the production raw media; and applying the editing function as defined by the generated value of the parameter to the production raw media to generate a production edited media composition.

Various embodiments include one or more of the following features. The plurality of media compositions previously edited by the user of the media editing application includes a media composition for which the user specified a value of the parameter defining the editing function by adjusting a value of the parameter that was previously generated automatically by the media editing application. The production edited media composition is a video composition and the production raw media content includes a video clip. The production representative raw media comprises low resolution versions of selected frames of the video clip. The editing function comprises color correction. The raw media comprises a plurality of recordings of multiple on-set takes of spoken dialog corresponding to a script, and the editing function comprises selecting a temporal span within the raw media corresponding to an optimal on-set take. The production edited media composition is an audio composition and the representative production raw media comprises one or more short duration portions of each of a plurality of audio sources. The editing function comprises generating a rough audio mix and the parameter defining the editing function to be applied is a gain level for one of the plurality of audio sources. The editing function comprises noise reduction or equalization. The production edited media composition is a musical score; the representative production raw media comprises spatial coordinates of each of a plurality of score symbols; and the editing function comprises score layout optimization. The production edited media composition is a musical score; the representative production raw media comprises one or more musical parts of the musical score; and the editing function comprises generation of a new musical part that is harmonized to the one or more parts of the musical score. The user-generated training data set further comprises a plurality of training instances obtained from a plurality of media compositions edited by users of the media editing application other than the first-mentioned user. Generating a trained user-adapted neural network model comprises: randomly assigning each instance of the third-party training data set to one of a training data set, a validation data set, and a test data set; randomly assigning each instance of the user-generated training data set to one of the training data set, the validation data set, and the test data set; training the user-adapted neural network model with the training data set; validating the user-adapted neural network model with the validation data set; and testing the user-adapted neural network model with the test data set. Training the user-adapted neural network model comprises: weights of a given number of layers of neurons of the trained third-party neural network model to initialize weights for the given number of layers of the user-adapted neural network; randomly initializing weights of one or more layers of the user-adapted neural network model that are not initialized with weights transferred from the trained third-party neural network model; and training the user-adapted neural network model with the user-generated training data set. The transferred weights are frozen during the training. The transferred weights are adjusted during training. An integrated user interface of the media editing application for performing the editing function on a media composition includes user controls for automatic editing of the function and user controls for manual editing of the function. The representative raw media of the user-generated training data set is extracted by a training data extraction application operating in conjunction with a plug-in software module that is able to read a media composition represented by a compositional data model specific to the media editing application. The user-adapted neural network model is defined by a plurality of hyperparameters, and a user interface of a training application used to train the user-adapted neural network model enables a user of the training application to specify, for each hyperparameter of the plurality of hyperparameters whether the hyperparameter is automatically adjusted by the training application or is manually adjusted. A user interface of a training application used to train the user-adapted neural network model displays an indication of an accuracy of the automatically generated value for the parameter defining the editing function. A hyperparameter of the user-adapted neural network model is a size of the input vector comprising representative raw media extracted from raw media that was previously edited by the user. A value of the hyperparameter is automatically determined during training of the user-adapted neural network model.

In general, in another aspect, a computer program product comprises: a non-transitory computer-readable medium with computer-readable instructions encoded thereon, wherein the computer-readable instructions, when processed by a processing device instruct the processing device to perform an automatic media editing method comprising: providing a computer-implemented media editing application that is able to edit raw media automatically to generate an edited media composition, wherein: the automatic editing comprises automatically generating a value of a parameter that defines an editing function that is applied to the raw media; the automatic editing is based on a trained third-party neural network model for the editing function, wherein the third-party neural network model was trained using a third-party training data set comprising, for each media composition of a plurality of media compositions edited by one or more third-party editors: an input vector comprising representative raw media extracted from raw media that was previously edited by the one or more third-party editors; and an output vector comprising a value specified by the one or more third-party editors of the parameter defining the editing function that was applied to the raw media during the previous editing; training a user-adapted neural network model based on a combination of the third-party training data set and a user-generated training data set, wherein: the user-generated training data set comprises, for each media composition of a plurality of media compositions previously edited by a user of the media editing application: an input vector comprising representative raw media extracted from raw media that was previously edited by the user; and an output vector comprising a value specified by the user of the parameter defining the editing function that was applied to the raw media during the editing by the user; and the media editing application: receiving production raw media; extracting production representative raw media from the production raw media; inputting the production representative raw media to the trained user-adaptive neural network model to generate an output value of the parameter defining the editing function to be applied to the production raw media; and applying the editing function as defined by the generated value of the parameter to the production raw media to generate a production edited media composition.

In general, in a further aspect, a system comprises a memory for storing computer-readable instructions; and a processor connected to the memory, wherein the processor, when executing the computer-readable instructions, causes the system to perform an automatic media editing method comprising: providing a computer-implemented media editing application that is able to edit raw media automatically to generate an edited media composition, wherein: the automatic editing comprises automatically generating a value of a parameter that defines an editing function that is applied to the raw media; the automatic editing is based on a trained third-party neural network model for the editing function, wherein the third-party neural network model was trained using a third-party training data set comprising, for each media composition of a plurality of media compositions edited by one or more third-party editors: an input vector comprising representative raw media extracted from raw media that was previously edited by the one or more third-party editors; and an output vector comprising a value specified by the one or more third-party editors of the parameter defining the editing function that was applied to the raw media during the previous editing; training a user-adapted neural network model based on a combination of the third-party training data set and a user-generated training data set, wherein: user-generated training data set comprises, for each media composition of a plurality of media compositions previously edited by a user of the media editing application: an input vector comprising representative raw media extracted from raw media that was previously edited by the user; and an output vector comprising a value specified by the user of the parameter defining the editing function that was applied to the raw media during the editing by the user; and the media editing application: receiving production raw media; extracting production representative raw media from the production raw media; inputting the production representative raw media to the trained user-adaptive neural network model to generate an output value of the parameter defining the editing function to be applied to the production raw media; and applying the editing function as defined by the generated value of the parameter to the production raw media to generate a production edited media composition.

