FIELD OF THE INVENTION

The present disclosure describes systems and methods for the generation of haptic signals, in particular for the generation of haptic vibrational signals based on received audio.

BACKGROUND

In video-game playing or video watching it is often desirable to generate haptic signals. An amplified haptic signal is sent to a transducer, such as a linear resonant actuator (LRA), mounted inside a device such as a smartphone, tablet, video game controller etc. in order to cause the device to vibrate. These physical vibrations provide feedback to the user to enrich the game playing or viewing experience. Haptic vibrations are typically generated when gunshots, explosions, or other high intensity events occur during a video-game or video.

High intensity events often correspond to loud sounds that are generated by a video or video-game. Haptic signals can be generated directly from a video or video-game which may output a haptic signal track in the same way as it outputs an audio signal track. Alternatively, an Audio-to-Haptics system can be used to automatically generate haptics signals based on audio output—e.g. audio frames—output by the video-game. So, one source of haptic signals is the audio signal itself. The audio signal output from a video-game or video is often a mix of different types of sound, such as music, sound-effects and speech. It is often desired to generate haptic signals corresponding to intense sound effects, e.g. gunshots, explosions, but without generating any haptic signals from an audio signal which comprises music, speech, or less intense sound effects. The present examples are generally concerned with generating haptic effects from an audio signal having certain characteristics or properties, without generating haptic effects from other audio signals having different characteristics or properties. For a particular video-game or video, or a particular scene from a video-game or video, it is often possible to specify a haptic audio power threshold. An audio signal whose power envelope rises above this threshold may be considered to be a good candidate for generating a haptic event. An audio signal whose power envelope is below this threshold may be more likely to correspond to e.g. speech, music, or less intense sound effects where the generation of a corresponding haptic event is not desired. Thus, the power threshold can be considered to be the power of audio signal at or above which the system is configured to generate a haptic effect.

However, the haptic audio power envelope event threshold may change from one game to another, or from one scene of a game to another scene of that same game. This problem will be explained with reference toFIG.1.

FIGS.1atocshow plots of the dB power envelope of lowpass filtered audio output of three example video games. For Video Game1—shown inFIG.1a—there are essentially two sounds—first-person gunshots at about −1 dB and distant gunfire reaching a max of −7 dB. There is also some background sound at much lower levels. It is desirable to generate haptics signals for the first-person gunshots but not for the distant gunfire. Setting a threshold halfway between at −4 dB works well for Video Game1.

For Video Game2—shown inFIG.1b—the peaks at around −8 dB beginning at 5 seconds are all candidates for haptics generation. There are also some larger peaks e.g. −5 dB at 50 seconds but it is desirable for all the peaks >=−8 dB to generate haptics.

For Video Game3—shown inFIG.1c—the beginning is speech. The first legitimate clash of weapons that should generate a haptics event occurs at −6 dB 43 seconds. The peak at −6 dB 33 seconds is a loud shout which should not generate haptics, but if a haptics event does occur it would be acceptable. The main goal is to avoid generating haptics for the preceding speech segment.

As a result, three different Video Game applications have the following characteristics:

Video GameHaptic Event Min.Non-Haptic Event Max.Threshold1 (FIG. 1a)−1 dB−7dB−4dB2 (FIG. 1b)−8 dB−14dB−11dB3 (FIG. 1c)−7 dB−10dB−8.5dB

It is desirable to set the threshold for each game or scene of a game so that it is above the non-haptics events maximum and below the haptics events minimum—e.g. half-way between the two as indicated above. As can be seen, there is not a single threshold that works for all of the games. If the threshold is set high enough to avoid the distant gunshots in Video Game1at −7 dB, then it will also exclude many of the desired haptic events of Video Game2at −8 dB. A need exists for a system that can automatically determine the haptics threshold for a particular game or scene from a game, while the game is being played.

The techniques described herein generally seek to improve on the previously proposed systems and/or seek to mitigate the problems associated with previously proposed systems.

SUMMARY

According to an example of a first aspect there is provided a processing system/module for a haptic signal generation system, the processing system comprising:

an input for receiving an input signal derived from an audio signal;

a threshold setting mechanism configured, based on a first representation or feature derived from the input signal, to adaptively adjust an existing threshold to derive a first threshold if a correspondence between the existing threshold and the first representation or feature is not met; and

a gain determining mechanism configured to determine an output gain to be applied to a haptic signal based on a comparison between the first threshold and a second representation or feature derived from the input signal.

According to at least one example the threshold setting mechanism may be configured to:

i) adjust the first threshold upwards if the first representation exceeds the first threshold by a predetermined amount;

ii) adjust the first threshold downwards if the first representation is below the first threshold by a predetermined amount.

According to at least one example the first representation or feature comprises information about the peak power levels of the audio signal. Furthermore, the second representation or feature may comprise information about the power envelope of the audio signal. The information about the power envelope may comprise a smoothed power envelope that rises more quickly when a power level of the input signal rises and falls relatively slowly when the power level of the input signal falls.

According to at least one example the gain determining mechanism is configured to determine a ratio between the first threshold and the second representation. The ratio may be processed such that when the first threshold and the second representation are equal, the output gain is zero. A saturation mechanism may be provided which is configured to saturate the range of the output gain between zero and 1.

According to an example of a second aspect there is provided a haptic signal generation system comprising a processing system according to an example of the first aspect and further comprising:

an audio signal input for receiving an audio signal; and

a haptic signal generator for generating a haptic signal,

wherein the output gain determined by the gain determining mechanism, or a value derived therefrom, is applied to the haptic signal, and wherein a haptic output signal is generated if the second representation exceeds the first threshold or comes within range of the first threshold.

The haptic system may further comprise a filtering mechanism provided in advance of the processing system and configured to exclude selected frequencies. In one example the filtering mechanism comprises first and second filters each having a different cutoff frequency that is selected to remove first and second components of the audio signal. For example the first filter may comprises a low pass filter and the second filter comprises a high pass filter. According to at least one example, the haptic system may further comprising a second threshold evaluation mechanism which is configured to receive first and second input signals derived from the output of the first and second filters. The second threshold evaluation mechanism may be configured to determine a product of the first and second input signals and to issue a haptic trigger signal if the magnitude of the product is sufficiently high to indicate that both the magnitude of both the first and second input signals is high.

The haptic system may further comprise a re-set mechanism, the re-set mechanism being configured to restore the first threshold to an initial value if a condition of silence is met for a pre-set time period. For example, a condition of silence is met if a metric derived from the at least one input signal to the processing system is the less than or equal to a third threshold. In particular, a condition of silence may be met if a product of the first and second input signals is less than or equal to a third threshold. According to one example the re-set mechanism is configured to advance a counter value if the condition of silence is met, wherein if the counter value exceeds a pre-set time period, the processing system is configured to restore the first threshold to the initial value.

According to an example of a third aspect there is provided a method of generating an output gain to be applied to a haptic signal, the method comprising:

receiving an input signal derived from an audio signal;

deriving a first feature from the input signal;

adjusting an existing threshold to derive a first threshold if a correspondence between the existing threshold and the first feature is not met; and

comparing the first threshold and a second feature derived from the input signal in order to generate the output gain.

According to one example the method may further comprise applying the output gain, or a value derived therefrom, to a haptic signal; and generating a haptic output signal if the second representation exceeds the first threshold or comes within range of the first threshold.

Associated methods are provided for each of the examples, aspects and embodiments described herewith.

