CROSS-REFERENCE TO PRIOR APPLICATIONS

This application is the U.S. National Phase application under 35 U.S.C. § 371 of International Application No. PCT/EP2019/079345, filed on Oct. 28, 2019, which claims the benefit of European Patent Application No. EP18203932.1, filed on Nov. 1, 2018. These applications are hereby incorporated by reference herein.

FIELD OF THE INVENTION

The invention relates to a system for determining one or more light effects based on an analysis of media content, said media content comprising video content and audio content.

The invention further relates to a method of determining one or more light effects based on an analysis of media content, said media content comprising video content and audio content.

The invention also relates to a computer program product enabling a computer system to perform such a method.

BACKGROUND OF THE INVENTION

A dynamic lighting system, such as Philips Hue Entertainment, can dramatically influence the experience and impressiveness of audio-visual material, especially when the audio-visual material is synced with a light script that constitutes an intelligent interpretation of the source material. However, manual creation of a light script can be a daunting task.

For this reason, methods exist to automatically generate a first light script from the source content. The first light script can then be rendered as-is or undergo further manual editing. Prior to, or during the automatic scripting process, there are certain parameters which may be set or changed by a user. These parameters may be set globally for the whole script generation process.

However, there many different ways in which light settings to be specified in a light script can be determined. For example, WO 2017/162469A1 discloses a controller which determines items of metadata associated with an audio stream, performs an image search (e.g. a Google search) based on the metadata, and uses the resulting images to determine light settings for the lighting devices, e.g. hue/brightness/saturation.

US2010/265414A1 discloses a method for controlling an ambient lighting element including determining ambient lighting data to control an ambient lighting element. The method includes processing combined ambient lighting data, wherein the combined ambient lighting data is based on corresponding video content portions and corresponding audio content portions. The processed combined ambient lighting data may then be used to control an ambient lighting element. Video-based ambient lighting data and audio-based ambient lighting data may be combined to produce the combined ambient lighting data. Combining the video-based and audio-based ambient lighting data may include modulating the video-based ambient lighting data by the audio-based ambient lighting data.

Although known automatic scripting methods reduce the amount of user input, the resulting light scripts often do not fit well with the content for/from which the lights script were generated.

SUMMARY OF THE INVENTION

It is a first object of the invention to provide a system, which is able to determine light effects that fit the video content for which they are determined without requiring a significant amount of user input.

It is a second object of the invention to provide a method, which is able to determine light effects that fit the video content for which they are determined without requiring a significant amount of user input.

In a first aspect of the invention, a system for determining one or more light effects based on an analysis of media content, said media content comprising video content and audio content, comprises at least one input interface, at least one output interface, and at least one processor configured to use said at least one input interface to allow a user to influence a video weight and an audio weight used for a determination of one or more light effects, said video weight representing a weight of said video content in said determination of said one or more light effects and said audio weight representing a weight of said audio content in said determination.

Said at least one processor is further configured to use said at least one input interface to obtain information relating to said video content and/or information relating to said audio content, determine said one or more light effects to be rendered on one more light sources while said media content is being rendered, said one or more light effects being determined based on said information relating to said video content in dependence on said video weight and being determined based on said information relating to said audio content in dependence on said audio weight, and use said at least one output interface to control said one or more light sources to render said one or more light effects and/or store a light script specifying said one or more light effects.

By using information derived from the audio, e.g. music, that accompanies visual content (e.g. theme music in a movie or series) to drive the parameters of automatic light script creation for certain pieces or portions of content, the determined light effects may fit better with these certain pieces or portions of content. Meta data related to music (e.g. from existing audio meta data services like Spotify, e.g. parameters such as valence, energy, and danceability, or by local analysis of the raw audio data) may be used to determine, by applying a set of rules, the parameters for automatic light script creation (for example, the type of color extraction method used, transition speed and/or spatial distribution).

By allowing a user to control the balance of influence between audio, e.g. music, and screen parameters toward the resulting light script, a better contextual accuracy of light script extraction may be achieved, thereby reducing the time a user would need to spend on manual adjustment of the light script. In case of real time light effect creation (e.g. Philips HueSync), a similar balancing control might be presented to the end user.

Said system may be part of a lighting system which comprises said one or more light sources or may be used in a lighting system which comprises said one or more light sources, for example. Said information related to audio content may be extracted from said audio content and/or obtained from an Internet server and/or said information related to said video content may be extracted from said video content and/or obtained from an Internet server.

Said information related to said audio content may comprise harmony, valence, energy, mood, and/or danceability information, for example. Said information related to said video content may comprise color information, for example. Said audio weight may be a music weight, i.e. only used to weigh how light effects are determined based on information relating to music content, or said audio weight may also be applied to other types of audio, e.g. speech and/or audio effects.

The user may be able to specify and/or to adapt both the video weight and the audio weight or the user may be able to specify and/or to adapt one of the two weights. In the latter case, after the user has specified or adapted the first weight, the second weight may be determined automatically based on the first weight. The user may specify a single value or a plurality of values, e.g. a range, for a weight. If the user specifies a plurality of values, the system itself may choose one of these values, e.g. depending on the type of audio content, on what channel(s) music is being played and/or on the type of video content. For example, if a certain scene in the movie is very colorful such that the trimean or dominant color does not represent the video content well, the system might decide to go for more audio influenced color, even if the user, for example, selected similar audio and video weight ranges.

Said video weight may comprise a plurality of weight components for different types of video information and/or said audio weight may comprise a plurality of weight components for different types of audio, e.g. music, information. This allows the user to more precisely specify his preferences.

Said at least one processor may be configured to determine whether said video weight and said audio weight have certain values non-zero, determine whether a light script is available for said audio content and use said at least one output interface to control said one or more light sources to render light effects specified in said available light script upon determining that said video weight and said audio weight have said certain non-zero values and said light script is available. For example, if a movie comprises a certain song and a light script is available for this song, a nice entertainment experience may be provided if the user does not prefer the use of only audio, e.g. music, information or only video information.

Said at least one processor may be configured to determine whether said audio content comprises more than a certain amount of dialogue and use said at least said at least one output interface to control said one or more light sources to render said light effects specified in said available light script upon determining that said video weight and said audio weight have said certain non-zero values, said light script is available and said audio content comprises not more than said certain amount of dialogue. For example, if there is more than a certain amount of dialogue in a movie, a light script relating to a song playing in the background may be less desirable.

Said at least one processor may be configured to determine said one or more light effects by determining a color, a start time and a duration of each of said one or more light effects. Said start time and said duration of said one or more light effects may be determined such that said one or more light effects are synchronized with beats, bars, and/or a rhythm of music in said audio content if said video weight has a zero value. This allows the one or more light effects to be based on only audio information.

Said at least one processor may be configured to further determine said one or more light effects by assigning each of said one or more light effects to at least one of said one or more light sources. For example, from which part of the video content and/or from which part of the audio content a light effect is determined for a certain light source may depend on the spatial position of the light source.

Said information related to said video content comprises color information extracted from said video content and said at least one processor may be configured to determine colors from said color information by using a certain color extraction method. Determining light effects based on colors extracted from video is a good way of determining light effects from video.

Said certain color extraction method is selected based on said information related to said audio content if said video weight has a certain non-zero value and said audio weight has a certain non-zero value. Thereby, the one or more light effects are determined based on both audio and video information.

A first color extraction method may be selected if said information related to said audio content comprises a first energy value and a second color extraction method may be selected if said information related to said audio content comprises a second energy value, said second energy value being higher than said first energy value and said second color extraction method involving a lower degree of smoothing than said first color extraction method. This is a beneficial manner of determining light effects based on both audio and video information.

A first color extraction method may be selected if said information related to said audio content comprises a first valence value and a second color extraction method may be selected if said information related to said audio content comprises a second valence value, said second valence value being higher than said first valence value and said second color extraction method determines colors which are more saturated than colors determined by said first color extraction method. This is a beneficial manner of determining light effects based on both audio and video information.

In a second aspect of the invention, a method of determining one or more light effects based on an analysis of media content, said media content comprising video content and audio content, comprises allowing a user to influence a video weight and an audio weight used for a determination of one or more light effects, said video weight representing a weight of said video content in said determination of said one or more light effects and said audio weight representing a weight of said audio content in said determination, and obtaining information relating to said video content and/or information relating to said audio content. Said information related to said video content comprises color information extracted from said video content and said at least one processor is configured to determine colors from said color information by using a certain color extraction method. The color extraction method is selected based on at least said information related to said audio content if said video weight has a certain non-zero value and said audio weight has a certain non-zero value.

Said method further comprises determining said one or more light effects to be rendered on one more light sources while said media content is being rendered, said one or more light effects being determined based on said information relating to said video content in dependence on said video weight and being determined based on said information relating to said audio content in dependence on said audio weight, and storing a light script specifying said one or more light effects and/or controlling said one or more light sources to render said one or more light effects.

Said method may further comprise that a first color extraction method is selected if said information related to said audio content comprises a first energy value and a second color extraction method is selected if said information related to said audio content comprises a second energy value, said second energy value being higher than said first energy value and said second color extraction method involving a lower degree of smoothing than said first color extraction method,

Said method may be performed by software running on a programmable device. This software may be provided as a computer program product.

Moreover, a computer program for carrying out the methods described herein, as well as a non-transitory computer readable storage-medium storing the computer program are provided. A computer program may, for example, be downloaded by or uploaded to an existing device or be stored upon manufacturing of these systems.

A non-transitory computer-readable storage medium stores a software code portion, the software code portion, when executed or processed by a computer, being configured to perform executable operations for determining one or more light effects based on an analysis of media content, said media content comprising video content and audio content.

The executable operations comprise allowing a user to influence a video weight and an audio weight used for a determination of one or more light effects, said video weight representing a weight of said video content in said determination of said one or more light effects and said audio weight representing a weight of said audio content in said determination, obtaining information relating to said video content and/or information relating to said audio content, determining said one or more light effects to be rendered on one more light sources while said media content is being rendered, said one or more light effects being determined based on said information relating to said video content in dependence on said video weight and being determined based on said information relating to said audio content in dependence on said audio weight, and storing a light script specifying said one or more light effects and/or controlling said one or more light sources to render said one or more light effects.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a device, a method or a computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit”, “module” or “system.” Functions described in this disclosure may be implemented as an algorithm executed by a processor/microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer readable storage medium may include, but are not limited to, the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store, a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java™, Smalltalk, C++ or the like, conventional procedural programming languages, such as the “C” programming language or similar programming languages, and functional programming languages such as Scala, Haskel or the like. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or a central processing unit (CPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

