NOTICE OF COPYRIGHTS AND TRADE DRESS

A portion of the disclosure of this patent document contains material which is subject to copyright protection. This patent document may show and/or describe matter which is or may become trade dress of the owner. The copyright and trade dress owner has no objection to the facsimile reproduction by anyone of the patent disclosure as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright and trade dress rights whatsoever.

RELATED APPLICATION INFORMATION

This patent claims priority to a counterpart European Patent Application number EP19151222.7 with the same title, filed Jan. 10, 2019 and assigned to the same assignee.

TECHNICAL FIELD

The present disclosure relates to methods, systems and computer programs for classifying object insertion opportunities in a video.

BACKGROUND

With the advent of digital file processing, it is possible to digitally insert visual objects into a video. In order to digitally insert a visual object into a video, first an opportunity for digital object insertion must be identified. That opportunity must then be evaluated to determine its suitability for object insertion, for example based on the value or benefit of digitally inserting the object. If it is decided that the opportunity is suitable, for example there is sufficient benefit or value in inserting the digital object into the video, the process of digital object insertion may then begin.

Digitally inserting objects in a video may have many benefits, for example enhancing the visual effects of the video, or improving the realism of a video, or allowing more flexibility for the video after it is shot meaning that fewer decisions need to be made during the filming of scenes. Consequently, digital object insertion is becoming increasingly common and utilised by video makers for all manner of purposes.

However, not every object insertion opportunity is worth using. For example, in some instances there may be 100s or 1000s of opportunities for inserting objects into a video (particularly for long videos, such as films). Most of those opportunities may not be deemed suitable for object insertion, for example because they are not sufficiently visible to viewers, or lack sufficient impact or benefit. Often, just a handful out of 100s or 1000s of opportunities, may be worth taking forward for visual object insertion. Filtering suitable opportunities (for example, those opportunities that may enhance a scene of the video, or increase viewer impact, etc.) from all of the digital object insert opportunities can be a time consuming and labour intensive process, requiring evaluation of each identified opportunity by human operators. It might also involve a degree of subjective analysis, which may cause inconsistency in evaluation between different operators, or even inconsistencies in evaluation from the same operator at different times. As digital object insertion becomes increasingly common, existing analysis and evaluation techniques have an increasingly detrimental effect on resource efficiency, particularly for videos of significant length, such as films or episodes/shows of television programs. Therefore, a more efficient and consistent approach to the identification of suitable object insertion opportunities is desired.

SUMMARY

In a first aspect of the present disclosure, there is provided a computer implemented method for classifying potential object insertion opportunities in a video, the method comprising: identifying, by analysing a plurality of pixels in each of one or more frames in a scene of the video, at least one potential object insertion opportunity in a scene of the video, wherein each potential object insertion opportunity represents an opportunity to digitally insert one or more visual objects into an image content of the scene; identifying, by analysing the plurality of pixels in each of the one or more frames in the scene of the video, a focus of attention in the scene, wherein the focus of attention represents a part of the scene that is likely to draw a viewer's focus of attention; determining a proximity value for each of the at least one potential object insertion opportunity based at least in part on a region of the scene covered by the potential object insertion opportunity and a region of the scene covered by the focus of attention, wherein the proximity value is indicative of a distance between the respective at least one potential object insertion opportunity and the focus of attention in the scene; and classifying each of the at least one potential object insertion opportunity based at least in part on the proximity value determined for each respective at least one potential object insertion opportunity.

Classifying each of the at least one potential object insertion opportunity may comprise classifying each of the at least one potential object insertion opportunity as a candidate object insertion opportunity or a rejected object insertion opportunity.

Classifying each of the at least one potential object insertion opportunity may further comprise determining an impact value for each candidate object insertion opportunity based at least in part on its associated proximity value.

Determining the proximity value may further comprise: determining a value for a distance parameter based on the region of the scene covered by the respective at least one potential object insertion opportunity and the region of the scene covered by the focus of attention; and determining the proximity value based at least in part on the value for the distance parameter.

The distance parameter may comprise a two dimensional relative distance between the region of the scene covered by the respective at least one potential object insertion opportunity and the region of the scene covered by the focus of attention.

Additionally, or alternatively, the distance parameter may comprise a three dimensional relative distance between the region of the scene covered by the respective at least one potential object insertion opportunity and the region of the scene covered by the focus of attention.

Additionally, or alternatively, the distance parameter may comprise a relative depth between the region of the scene covered by the respective at least one potential object insertion opportunity and the region of the scene covered by the focus of attention.

Additionally, or alternatively, the distance parameter may comprise a scale (such as a scale of 0-4) indicative of a distance between the region of the scene covered by the respective at least one potential object insertion opportunity and the region of the scene covered by the focus of attention. Determining the value of the distance parameter may comprise using an artificial intelligence (AI), for example comprising one or more AI or machine learning algorithms, trained using a corpus of training images, wherein each of the training images comprises a focus of attention, at least one potential object insertion opportunity and, for each potential object insertion opportunity, a value on the scale.

The distance between the respective at least one potential object insertion opportunity and the focus of attention in the scene may comprise one or more of: a) a distance between a centre of gravity of the respective potential object insertion opportunity and a centre of gravity of the focus of attention; b) a distance between a centre of gravity of the respective potential object insertion opportunity and an edge of the focus of attention; c) a distance between an edge of the respective potential object insertion opportunity and a centre of gravity of the focus of attention; d) a distance between an edge of the respective potential object insertion opportunity and an edge of the focus of attention.

The edge of the potential object insertion opportunity may be the edge of the potential object insertion opportunity that is closest to the focus of attention and the edge of the focus of attention may be the edge of the focus of attention that is closest to the respective potential object insertion opportunity.

The analysis of the plurality of pixels in each of the one or more frames in the scene to identify the focus of attention may comprise facial recognition.

At least part of determining a proximity value for each of the one or more potential object insertion opportunities may comprise using an AI algorithm trained using a corpus of training images, wherein each of the training images comprises a focus of attention, at least one potential object insertion opportunity and a corresponding at least one proximity value.

In a second aspect of the present disclosure, there is provided a computer program comprising instructions which, when the program is executed by a computer cause the computer to carry out the method of the first aspect.

In a third aspect of the present disclosure, there is provided a data processing system comprising at least one processor configured to perform the method of the first aspect.

In a fourth aspect of the present disclosure, there is provided a method of training an AI algorithm to determine a value for a distance parameter for at least one potential object insertion opportunity in a video, the method comprising: obtaining a plurality of images, each image comprising data indicative of: a focus of attention in the visual content of the image; at least one potential object insertion opportunity in the visual content of the image, wherein each potential image object insertion opportunity represents an opportunity to digitally insert one or more visual objects into the image; and a corresponding value for the distance parameter for each of the at least one potential object insertion opportunity, wherein the distance parameter is indicative of a distance between the respective at least one potential object insertion opportunity and the focus of attention in the scene; and using the plurality of images as a training corpus to train the AI algorithm to determine values for the distance parameter for potential object insertion opportunities in scenes of videos comprising at least one potential object insertion opportunity and a focus of attention.

In a fifth aspect of the present disclosure, there is provided an AI algorithm trained using the method of the fourth aspect.

In a sixth aspect of the present disclosure, there is provided a computer implemented method for classifying potential object insertion opportunities in a video, the method comprising: obtaining object insertion data comprising: one or more frames of a scene of the video; data indicative of at least one potential object insertion opportunity in the scene of the video, wherein each potential image object insertion opportunity represents an opportunity to digitally insert one or more visual objects into an image content of the scene; and data indicative of a focus of attention in the scene, wherein the focus of attention represents a part of the scene that is likely to draw a viewer's focus of attention; determining a proximity value for each of the at least one potential object insertion opportunity based at least in part on a relative distance between the respective at least one potential object insertion opportunity and the focus of attention in the scene; and classifying each of the at least one potential object insertion opportunity based at least in part on the proximity value determined for each respective at least one potential object insertion opportunity.

In a seventh aspect of the present disclosure, there is provided a classification module configured to perform the steps of the sixth aspect.

