TECHNICAL FIELD

This disclosure relates to processing audio data. In particular, this disclosure relates to processing media input audio data corresponding to a media stream and microphone input audio data from at least one microphone.

BACKGROUND

The use of audio devices such as headphones and earbuds has become extremely common. Such audio devices can at least partially occlude sounds from the outside world. Some headphones are capable of creating a substantially closed system between headphone speakers and the eardrum, in which sounds from the outside world are greatly attenuated. There are various potential advantages of attenuating sounds from the outside world via headphones or other such audio devices, such as eliminating distortion, providing a flat equalization, etc. However, when wearing such audio devices a user may not be able to hear sounds from the outside world that it would be advantageous to hear, such as the sound of an approaching car, the sound of a friend's voice, etc.

SUMMARY

Some methods disclosed herein may involve receiving media input audio data corresponding to a media stream and receiving microphone input audio data from at least one microphone. As used herein, the terms “media stream,” “media signal” and “media input audio data” may be used to refer to audio data corresponding to music, a podcast, a movie soundtrack, etc. However the terms are not limited to such examples. Instead, the terms “media stream,” “media signal” and “media input audio data” may be used to refer to the audio data corresponding to other sounds received for playback, e.g., as part of a telephone conversation. Some methods may involve determining a first level of at least one of a plurality of frequency bands of the media input audio data and determining a second level of at least one of a plurality of frequency bands of the microphone input audio data. Some such methods may involve producing media output audio data and microphone output audio data by adjusting levels of one or more of the first and second plurality of frequency bands. For example, some methods may involve adjusting levels such that a first difference between a perceived loudness of the microphone input audio data and a perceived loudness of the microphone output audio data in the presence of the media output audio data is less than a second difference between the perceived loudness of the microphone input audio data and a perceived loudness of the microphone input audio data in the presence of the media input audio data. Some such methods may involve mixing the media output audio data and the microphone output audio data to produce mixed audio data. Some such examples may involve providing the mixed audio data to speakers of an audio device, such as a headset or earbuds.

In some implementations, the adjusting may involve only boosting the levels of one or more of the plurality of frequency bands of the microphone input audio data. However, in some examples the adjusting may involve both boosting the levels of one or more of the plurality of frequency bands of the microphone input audio data and attenuating the levels of one or more of the plurality of frequency bands of the media input audio data. The perceived loudness of the microphone output audio data in the presence of the media output audio data may, in some examples, be substantially equal to the perceived loudness of the microphone input audio data. According to some examples, the total loudness of the media and microphone output audio data may be in a range between the total loudness of the media and microphone input audio data and the total loudness of the media and microphone output audio data. However, in some instances, the total loudness of the media and microphone output audio data may be substantially equal to the total loudness of the media and microphone input audio data, or may be substantially equal to the total loudness of the media and microphone output audio data.

Some implementations may involve receiving (or determining) a mode-switching indication and modifying one or more process based, at least in part, on the mode-switching indication. For example, some implementations may involve modifying at least one of the receiving, determining, producing or mixing process based, at least in part, on the mode-switching indication. In some instances, the modifying may involve increasing a relative loudness of the microphone output audio data, relative to a loudness of the media output audio data. According to some such examples, increasing the relative loudness of the microphone output audio data may involve suppressing the media input audio data or pausing the media stream.

According to some implementations, the mode-switching indication may be based, at least in part, on an indication of a head movement and/or an indication of an eye movement. In some such implementations, the mode-switching indication may be based, at least in part, on inertial sensor data. The inertial sensor data may, for example, correspond to movement of a headset. In some instances, an indication of an eye movement may include camera data and/or electroencephalogram data.

Some examples may involve determining a direction of a sound source based, at least in part, on microphone data from two or more microphones. Some such examples may involve determining whether the direction of the sound source corresponds with a head movement and/or an eye movement. Alternatively, or additionally, some examples may involve receiving an indication from a user of a selected sound source direction. Some such examples may involve determining a direction of a sound source based, at least in part, on microphone data from two or more microphones. Some such examples may involve determining that the location of the sound source is a mode-switching indication if the location of the sound source corresponds with the selected sound source direction.

Some other examples may involve determining a direction of a sound source based, at least in part, on microphone data from two or more microphones. Some such examples may involve determining whether there is a mode-switching indication based, at least in part, on the movement direction of the sound source. Some such examples may involve determining a mode-switching indication based, at least in part, on a determination that the movement direction of the sound source may be towards at least one of the microphones.

Alternatively, or additionally, some examples may involve determining a velocity of the sound source Some such examples may involve determining a mode-switching indication based, at least in part, on a determination that the velocity of the sound source exceeds a threshold.

According to some implementations, the mode-switching indication may be based, at least in part, on recognition of speech in the microphone input audio data. Some such examples may involve a classification of the microphone input audio data. For example, the classification may involve determining whether the microphone input audio data includes a car horn, an approaching vehicle sound, a scream, a yell, a pre-selected person's voice, a pre-selected key word and/or a public announcement. The mode-switching indication may be based, at least in part, on the classification.

The methods disclosed herein may be implemented via hardware, firmware, software stored in one or more non-transitory media, and/or combinations thereof. For example, at least some aspects of this disclosure may be implemented in an apparatus that includes an interface system and a control system. The interface system may include a user interface and/or a network interface. In some implementations, the apparatus may include a memory system. The interface system may include at least one interface between the control system and the memory system.

The control system may include at least one processor, such as a general purpose single- or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and/or combinations thereof.

According to some examples, the control system may be capable of receiving media input audio data corresponding to a media stream and receiving microphone input audio data from at least one microphone. In some implementations, the apparatus may include a microphone system having one or more microphones. In some examples, the microphone system may include two or more microphones. In some implementations, the apparatus may include a speaker system that includes one or more speakers. According to some such implementations, the apparatus may be a headset, or a component of a headset. However, in other implementations the apparatus may be configured for receiving microphone input audio data and/or media input audio data corresponding to a media stream from another device.

In some examples, the control system may be capable of determining a first level of at least one of a plurality of frequency bands of the media input audio data and determining a second level of at least one of a plurality of frequency bands of the microphone input audio data. The control system may, for example, be capable of producing media output audio data and microphone output audio data by adjusting levels of one or more of the first and second plurality of frequency bands. For example, the control system may be capable of adjusting levels such that a first difference between a perceived loudness of the microphone input audio data and a perceived loudness of the microphone output audio data in the presence of the media output audio data is less than a second difference between the perceived loudness of the microphone input audio data and a perceived loudness of the microphone input audio data in the presence of the media input audio data. In some examples, the control system may be capable of mixing the media output audio data and the microphone output audio data to produce mixed audio data. According to some examples, the control system may be capable of providing the mixed audio data to speakers of an audio device, such as a headset or earbuds.

In some implementations, the adjusting may involve only boosting the levels of one or more of the plurality of frequency bands of the microphone input audio data. However, in some examples the adjusting may involve both boosting the levels of one or more of the plurality of frequency bands of the microphone input audio data and attenuating the levels of one or more of the plurality of frequency bands of the media input audio data. The perceived loudness of the microphone output audio data in the presence of the media output audio data may, in some examples, be substantially equal to the perceived loudness of the microphone input audio data. According to some examples, the total loudness of the media and microphone output audio data may be in a range between the total loudness of the media and microphone input audio data and the total loudness of the media and microphone output audio data. However, in some instances, the total loudness of the media and microphone output audio data may be substantially equal to the total loudness of the media and microphone input audio data, or may be substantially equal to the total loudness of the media and microphone output audio data.

According to some examples, the control system may be capable of receiving (or determining) a mode-switching indication and of modifying one or more process based, at least in part, on the mode-switching indication. For example, the control system may be capable of modifying at least one of the receiving, determining, producing or mixing process based, at least in part, on the mode-switching indication. In some instances, the modifying may involve increasing a relative loudness of the microphone output audio data, relative to a loudness of the media output audio data. According to some such examples, increasing the relative loudness of the microphone output audio data may involve suppressing the media input audio data or pausing the media stream.

According to some implementations, the control system may be capable of determining the mode-switching indication based, at least in part, on an indication of a head movement and/or an indication of an eye movement. In some such implementations, the apparatus may include an inertial sensor system. According to some such implementations, the control system may be capable of determining the mode-switching indication based, at least in part, on inertial sensor data received from the inertial sensor system. The inertial sensor data may, for example, correspond to movement of a headset.

In some instances, the apparatus may include an eye movement detection system. According to some such implementations, the control system may be capable of determining the mode-switching indication based, at least in part, on data received from the eye movement detection system. The eye movement detection system may, in some examples, include one or more cameras. The eye movement detection system may, in some examples, include an electroencephalogram (EEG) system, which may include one or more EEG electrodes. According to some implementations, the EEG electrodes may be configured to be positioned in a user's ear canal and/or on the user's scalp. According to some such examples, the control system may be capable of detecting a user's eye movements via analysis of an EEG signal received from one or more EEG electrodes of the EEG system. In some such examples, the control system may be capable of determining a mode-switching indication based, at least in part, on an indication of an eye movement. The indication of the eye movement may be based on camera data and/or EEG data from the eye movement detection system.

According to some examples, the control system may be capable of determining a direction of a sound source based, at least in part, on microphone data from two or more microphones. According to some such examples, the control system may be capable of determining whether the direction of the sound source corresponds with a head movement and/or an eye movement. Alternatively, or additionally, the control system may be capable of receiving an indication from a user of a selected sound source direction. In some such examples, the control system may be capable of determining a direction of a sound source based, at least in part, on microphone data from two or more microphones. The control system may, for example, be capable of determining that the location of the sound source is a mode-switching indication if the location of the sound source corresponds with the selected sound source direction.

Some other examples may involve determining a direction of a sound source based, at least in part, on microphone data from two or more microphones. In some such examples, the control system may be capable of determining whether there is a mode-switching indication based, at least in part, on the movement direction of the sound source. In some such examples, the control system may be capable of may involve determining a mode-switching indication based, at least in part, on a determination that the movement direction of the sound source may be towards at least one of the microphones.

Alternatively, or additionally, the control system may be capable of determining a velocity of the sound source. In some such examples, the control system may be capable of determining a mode-switching indication based, at least in part, on a determination that the velocity of the sound source exceeds a threshold.

According to some implementations, the mode-switching indication may be based, at least in part, on recognition of speech in the microphone input audio data. In some such examples, the control system may be capable of making a classification of the microphone input audio data. For example, the classification may involve determining whether the microphone input audio data includes a car horn, an approaching vehicle sound, a scream, a yell, a pre-selected person's voice, a pre-selected key word and/or a public announcement. The mode-switching indication may be based, at least in part, on the classification.

Some implementations may include one or more non-transitory media having software stored thereon. In some examples, the non-transitory media may include flash memory, a hard drive and/or other memory devices. The software may include instructions for controlling at least one apparatus for receiving media input audio data corresponding to a media stream and receiving microphone input audio data from at least one microphone. The software may include instructions for determining a first level of at least one of a plurality of frequency bands of the media input audio data and determining a second level of at least one of a plurality of frequency bands of the microphone input audio data. The software may include instructions for producing media output audio data and microphone output audio data by adjusting levels of one or more of the first and second plurality of frequency bands. For example, the software may include instructions for adjusting levels such that a first difference between a perceived loudness of the microphone input audio data and a perceived loudness of the microphone output audio data in the presence of the media output audio data is less than a second difference between the perceived loudness of the microphone input audio data and a perceived loudness of the microphone input audio data in the presence of the media input audio data. In some examples, the software may include instructions for mixing the media output audio data and the microphone output audio data to produce mixed audio data. Some such examples may involve providing the mixed audio data to speakers of an audio device, such as a headset or earbuds.

In some implementations, the adjusting may involve only boosting the levels of one or more of the plurality of frequency bands of the microphone input audio data. However, in some examples the adjusting may involve both boosting the levels of one or more of the plurality of frequency bands of the microphone input audio data and attenuating the levels of one or more of the plurality of frequency bands of the media input audio data. The perceived loudness of the microphone output audio data in the presence of the media output audio data may, in some examples, be substantially equal to the perceived loudness of the microphone input audio data. According to some examples, the total loudness of the media and microphone output audio data may be in a range between the total loudness of the media and microphone input audio data and the total loudness of the media and microphone output audio data. However, in some instances, the total loudness of the media and microphone output audio data may be substantially equal to the total loudness of the media and microphone input audio data, or may be substantially equal to the total loudness of the media and microphone output audio data.

In some examples, the software may include instructions for receiving (or determining) a mode-switching indication and for modifying one or more process based, at least in part, on the mode-switching indication. For example, in some implementations the software may include instructions for modifying at least one of the receiving, determining, producing or mixing process based, at least in part, on the mode-switching indication. In some instances, the modifying may involve increasing a relative loudness of the microphone output audio data, relative to a loudness of the media output audio data. According to some such examples, the software may include instructions for increasing the relative loudness of the microphone output audio data by suppressing the media input audio data or pausing the media stream.

According to some implementations, the software may include instructions for determining the mode-switching indication based, at least in part, on an indication of a head movement and/or an indication of an eye movement. In some such implementations, the software may include instructions for determining the mode-switching indication based, at least in part, on inertial sensor data. The inertial sensor data may, for example, correspond to movement of a headset. In some instances, an indication of an eye movement may include camera data and/or electroencephalogram data.

In some examples, the software may include instructions for determining a direction of a sound source based, at least in part, on microphone data from two or more microphones. In some such examples, the software may include instructions for determining whether the direction of the sound source corresponds with a head movement and/or an eye movement. Alternatively, or additionally, in some examples the software may include instructions for receiving an indication from a user of a selected sound source direction. In some such examples, the software may include instructions for determining a direction of a sound source based, at least in part, on microphone data from two or more microphones. In some such examples, the software may include instructions for determining that the location of the sound source is a mode-switching indication if the location of the sound source corresponds with the selected sound source direction.

Some other examples may involve determining a direction of a sound source based, at least in part, on microphone data from two or more microphones. According to some implementations, the software may include instructions for determining whether there is a mode-switching indication based, at least in part, on the movement direction of the sound source. Some such examples may involve determining a mode-switching indication based, at least in part, on a determination that the movement direction of the sound source may be towards at least one of the microphones.

Alternatively, or additionally, in some examples the software may include instructions for determining a velocity of the sound source. In some such examples, the software may include instructions for determining a mode-switching indication based, at least in part, on a determination that the velocity of the sound source exceeds a threshold.

According to some implementations, the mode-switching indication may be based, at least in part, on recognition of speech in the microphone input audio data. In some such examples, the software may include instructions for making a classification of the microphone input audio data. For example, the classification may involve determining whether the microphone input audio data includes a car horn, an approaching vehicle sound, a scream, a yell, a pre-selected person's voice, a pre-selected key word and/or a public announcement. In some such examples, the software may include instructions for determining a mode-switching indication based, at least in part, on the classification.

Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.

