Depth perception of audio sources in stereo 3D environments
CC Attribution - NoDerivatives 2.0 UK: England & Wales:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Corrigan, David
Gorzel, Marcin
Squires, John
Boland, Frank
In this paper we undertook perceptual experiments to determine the allowed differences in depth between audio and visual stimuli in stereoscopic-3D environments while being perceived as congruent. We also investigated whether the nature of the environment and stimuli affects the perception of congruence. This was achieved by creating an audio-visual environment consisting of a photorealistic visual environment captured by a camera under orthostereoscopic conditions and a virtual audio environment generated by measuring the acoustic properties of the real environment. The visual environment consisted of a room with a loudspeaker or person forming the visual stimulus and was presented to the viewer using a passive stereoscopic display. Pink noise samples and female speech were used as audio stimuli which were presented over headphones using binaural renderings. The stimuli were generated at different depths from the viewer and the viewer was asked to determine whether the audio stimulus was nearer, further away or at the same depth as the visual stimulus. From our experiments it is shown that there is a significant range of depth differences for which audio and visual stimuli are perceived as congruent. Furthermore, this range increases as the depth of the visual stimulus increases. 
 Thank you very much. So rather like quite similar to last talk, what we're interested in studying is the influence in 3D presentations when you have 3D, the Client Stereoscopic 3D with 3D audio technologies. So what we're really interested in is the implications for how you create 3D content when you have to consider both audio and 3D video. So like the previous presenters stated that there is this known modal dominance of visual cues over audio cues. So things like ventriloquism whereby vision can alter the perception of where sound source comes from. And what this indicates is this potential for a zone of congruence whereby you could have some small mismatches between audio and visual cues or audio visual stimuli positions. So what we want to do is to quantify that congruence for Stereo23D video in a home environment or in a cinema environment. So what we did in these experiments was just basically to estimate the zone of congruence for audio and visual stimuli positions along the depth axis. So how far apart could those sources be separated while still being perceived as coming from the same position? Okay, so basically we conducted two tests and so we tried to just basically overarching point of view to see how that affected the zone of audio and visual congruence. So in the first experiment we were looking at determining the zone of congruence with respect to visual depth. So you have a loudspeaker which is representing your visual source of your sound and you have a very position of loudspeaker and you want to measure the zone of congruence in each case. The second point or second factor we considered was the familiarity of the audio source. So we have a pick noise source which represents an unfamiliar sound source and then speech is representing a familiar sound source which may have additional depth cues in it that would affect a perception of depth. And in the second experiment we considered other factors including the visual stimulus type. So if you consider you use a static visual stimulus like a loudspeaker or whether you have a human voicing the sounds, does that make a difference? And finally the test environment, really testing there is reverberation, the amount of reverberation in your room and how does that affect those kind of judgments? Okay, so for each of the two tests, so before I describe the experiments themselves, I just want to talk about the capture and display methodologies we use for both audio and video. So for the two experiments that we did we actually adopted two different capture methodologies and the reason why is because that test two uses moving stimuli and that meant that we had to adjust. So for the static stimuli which are only in test one, we would just simply capture the stereo 3D by scanning a camera using a linear positioner at a controlled speed, sort of along an axis perpendicular to the depth axis and that would allow us then to just generate the stereo 3D pair by just extracting appropriate frames in the video. The advantage of that methodology was that it allowed us basically to vary the IPD so we could change or sorry vary the interaction distance to match the inter-privileged distance of the test subject. Whereby for the second experiment we just used a conventional stereo rig again just because we couldn't capture moving stimuli with the first methodology. But unfortunately we lose the ability to vary the IPD but we set the camera separation to be just the average eye separation for humans. So when getting the depth use, basically what we want to do is we want to make sure that we have some calibration between the position of the audio source and the position of video sources. So from a visual point of view what we try to do is generate as close as possible to orthostereoscopic conditions. So the first condition is to match the camera baseline with the IPD which is what I described in the previous slide. Also you have to adjust the field of view of the camera to match the extent of the display or the expended display would be in the viewer's field of vision. So we typically shot with a wider field of view than we needed and we just cropped the image by the factor you can see there and basically all those kind of values are known. So you can work out the correct ratio, the image resolution and to work out the correct image size. And then finally then we have to adjust the convergence point of the screen depth and that's just by applying a simple horizontal image translation to each image. So now onto the audio next. So basically what we want to do is to simulate sort of a home listening environment. But we kind of adopted a kind of a novel approach doing that. So what we were trying to do basically is to render, instead of rendering over a loudspeaker array like 5.1 array like the previous speaker was talking about, we used a virtual loudspeaker array developed over headphones. Now there's a number of aspects to this procedure. So it's going to describe each one in term. The first one is the actual format we use to store the 3D sound fields. So we used a first order ambisonics approach represented 3D sound fields. So the basis of ambisonics is that you represent the 3D sound field as a pressure field centered on the point surrounding the listener. And basically you decompose that sound field into weighted sum of spherical harmonics basis functions. So the first order ambisonics we're using, zero and first order spherical harmonics. And the main benefit of using ambisonics is it allows you to decouple the choice of loudspeaker array from the way you capture the data. So you can decide your loudspeaker array as opposed, you know, after you've captured all data which is kind of nice. And secondly, it allows you to easily rotate the sound field by just simply rotating spherical harmonic spaces functions. So that will come in useful later on when I go through the whole system. So the second aspect is how actually the data is captured itself. So instead of actually recording the audio stimuli live in the room, the approach that we adopted was to record the audio stimuli in a studio and then basically to insert all relevant audio debt cues by convolving the audio signal with a spatial room impulse response which essentially modifies all the debt cues that are pertained to the particular test environment. Again, the main reason for adopting this approach is it allows greater efficiency in terms of creating the audio stimuli for all the different test environments that we have and the different sort of audio source distances so we can just generate the correct audio files for different audio stimuli much quicker. And it's also less prone to environmental noise when you're actually recording the data. And then finally then is the virtual loudspeaker rendering. So at this stage, you know, you have two choices. You could just adopt my kind of conventional loudspeaker array approach. And because we've used Ambisonics, we can choose any loudspeaker configuration basically we want. In this case, we're just going for an octagonal loudspeaker array centered on the listener, on the user. But instead of using a real array, we're using a virtual array. So it's essentially trying to simulate the same sound field but over headphones rather than over a real array. And the challenge is to simulate the effect of the head and your shoulders and your pinnacle on the audio signal. And that varies according to the speaker position. So that's captured basically by what's known as a head related impulse response or HR IR. And essentially for each loudspeaker position to get the virtual signal what you have to do is to involve what would be the loudspeaker output or the mono source with these two HR IRs for the left and right ear to generate signal for the left and right ear. Then you basically repeat the process for every speaker in the array and that allows you then to get the virtual 3D sound field displayed overhead once. Now you might think that this, well, let's just say that this is a maybe a strange approach but we've done a lot of experiments previously to show that this type of approach to rendering sound fields is perceptually equivalent to real sound sources. So here is an experiment which is a kind of a typical distance judgment experiment that you would see in the audio domain where you have to judge the distance of a sound source and we're just comparing the various different methodologies. So the two lines in the graph that are relevant are the black and blue line which the black line representing having an actual loudspeaker specific distance and you have to judge how far away that is and the set of blue line is the approach we were adopting in this paper, the first order amysoics approach and we can show that within statistical significance bounds that they are perceptually equivalent. So the key thing that makes this work is head tracking. So head tracking is required essentially to create stable 3D sound field. So if I'm listening to 3D audio overhead once I rotate my head the sound field shouldn't rotate with me. This sound field should stay in the same position. So head tracking really is the thing that makes this work in practice. So the experiments themselves. So the first experiment like I said before we're comparing visual distance and stimuli and we're comparing the zone of congruence. So we have three visual distances and we have 14 audio distances. So the same audio distances are used for each of the three visual distances. We have two audio stimuli and basically all combinations of audio and visual stimuli are presented to the listener once. So we had 20 test subjects and they were basically placed in a darkened room. They were told to sit two meters away from the screen. It was a polarized display in a darkened room and they were asked basically they were allowed to present it with the UI you see at the bottom where they were able to play a sound repeatedly and they were asked to judge whether the sound came from in front of the loudspeaker at the loudspeaker or behind the loudspeaker. So here are the results of the first experiment. So this first slide now is just comparing the familiarity of the sound source and the zone of congruence. So let me just explain the graphs and it's the same convention for all the graphs to see later on. So we have the green line just basically represents a portion of people who think that the audio and visual source come from the same position. The red line is the percentage of people who think that the audio is in front of the video, of the visual stimulus and the blue line is the proportion of people who think it's behind. So you can see that the graphs follow what you would expect the pattern to be. But if you're talking about the zone of congruence, we've kind of defined it here as the range of depths for which the green value is highest, including the error bounds as well. So if there's a large crossover between the two error bounds, we don't count it as part of the zone of congruence. So for this slide where we're just comparing for one visual distance, the difference between the pink noise, audio stimulus and the female speech stimulus, you can see that we don't really detect any significantly difference in the zone of congruence. So it's pretty much the same. So from somewhere closer than one meter, we could reasonably hypothesize that would be greater than zero is the closest that the zone of congruence could be started at. And it goes back to something like two and a half meters, which is half a meter behind the position of the stimulus. Now, we repeat this experiment again for the other visual distances and we just conclude the same results at each distance that there is no statistical difference between the two sound sources. So moving on from that down, we're now comparing visual distance. And I suppose we have two, four and eight meters and like you would expect, the zone of congruence increases as the visual distance increases. So again, you might have a range of say two and a half meters maximum, say for the two meter visual distance that increases three and a half meters at four meters and at eight meters is somewhere over five meters, which is probably significant, be higher. So we were kind of limited in that scenario by the size of the room. We actually couldn't go back much further than 11 meters. That was just the back wall of the room. So in the second test of our experiment then, we are comparing the zone of congruence with respect to the, all right, okay, with respect to the test environment and whether we use the speaker, the speaker, female. And really hear what we're talking about. The test environment one is just has more reverberance in it. So reverberance is shown to be quite a strong depth view and it could effectively alter perception. So I'm just going to quickly summarize over the results. Basically, although we saw some slight variations in the results, I don't think we were convinced that there was actually any statistical differences in the zone of congruence across all the different combination of factors. In fact, we have known, I mean some known inaccuracies in the way we captured the data, which could explain some of the difference that are apparent in that graph. So our conclusion there basically is that there is no significant difference. So finally then, just include, there's definitely a clear link between the zone of congruence and the visual depth, the size of the zone of congruence and the visual depth. But from all the other factors that we test, we couldn't really detect significant differences. So for future work then, we would like to refine our capture procedures just to get more robustness in them and to get more data by testing on more subjects. And also then I guess finally we might like to determine how do large errors or perceivable errors in audio and video position affect people's perception of quality and immersiveness. So thank you very much for your attention. I'd be glad to answer any questions if you have time. Okay. Thank you very much. Questions for David? All right. Well, thank you very much. All right. Thank you.
 Thank you very much.  So rather like quite similar to last talk, what we're interested in studying is the influence  in 3D presentations when you have 3D, the Client Stereoscopic 3D with 3D audio technologies.  So what we're really interested in is the implications for how you create 3D content  when you have to consider both audio and 3D video.  So like the previous presenters stated that there is this known modal dominance of visual  cues over audio cues.  So things like ventriloquism whereby vision can alter the perception of where sound source  comes from.  And what this indicates is this potential for a zone of congruence whereby you could  have some small mismatches between audio and visual cues or audio visual stimuli positions.  So what we want to do is to quantify that congruence for Stereo23D video in a home environment  or in a cinema environment.  So what we did in these experiments was just basically to estimate the zone of congruence  for audio and visual stimuli positions along the depth axis.  So how far apart could those sources be separated while still being perceived as coming from  the same position?  Okay, so basically we conducted two tests and so we tried to just basically overarching  point of view to see how that affected the zone of audio and visual congruence.  So in the first experiment we were looking at determining the zone of congruence with  respect to visual depth.  So you have a loudspeaker which is representing your visual source of your sound and you have  a very position of loudspeaker and you want to measure the zone of congruence in each case.  The second point or second factor we considered was the familiarity of the audio source.  So we have a pick noise source which represents an unfamiliar sound source and then speech  is representing a familiar sound source which may have additional depth cues in it that  would affect a perception of depth.  And in the second experiment we considered other factors including the visual stimulus  type.  So if you consider you use a static visual stimulus like a loudspeaker or whether you  have a human voicing the sounds, does that make a difference?  And finally the test environment, really testing there is reverberation, the amount of reverberation  in your room and how does that affect those kind of judgments?  Okay, so for each of the two tests, so before I describe the experiments themselves, I just  want to talk about the capture and display methodologies we use for both audio and video.  So for the two experiments that we did we actually adopted two different capture methodologies  and the reason why is because that test two uses moving stimuli and that meant that we  had to adjust.  So for the static stimuli which are only in test one, we would just simply capture the  stereo 3D by scanning a camera using a linear positioner at a controlled speed, sort of  along an axis perpendicular to the depth axis and that would allow us then to just generate  the stereo 3D pair by just extracting appropriate frames in the video.  The advantage of that methodology was that it allowed us basically to vary the IPD so  we could change or sorry vary the interaction distance to match the inter-privileged distance  of the test subject.  Whereby for the second experiment we just used a conventional stereo rig again just because  we couldn't capture moving stimuli with the first methodology.  But unfortunately we lose the ability to vary the IPD but we set the camera separation  to be just the average eye separation for humans.  So when getting the depth use, basically what we want to do is we want to make sure that  we have some calibration between the position of the audio source and the position of video  sources.  So from a visual point of view what we try to do is generate as close as possible to  orthostereoscopic conditions.  So the first condition is to match the camera baseline with the IPD which is what I described  in the previous slide.  Also you have to adjust the field of view of the camera to match the extent of the display  or the expended display would be in the viewer's field of vision.  So we typically shot with a wider field of view than we needed and we just cropped the  image by the factor you can see there and basically all those kind of values are known.  So you can work out the correct ratio, the image resolution and to work out the correct  image size.  And then finally then we have to adjust the convergence point of the screen depth and  that's just by applying a simple horizontal image translation to each image.  So now onto the audio next.  So basically what we want to do is to simulate sort of a home listening environment.  But we kind of adopted a kind of a novel approach doing that.  So what we were trying to do basically is to render, instead of rendering over a loudspeaker  array like 5.1 array like the previous speaker was talking about, we used a virtual loudspeaker  array developed over headphones.  Now there's a number of aspects to this procedure.  So it's going to describe each one in term.  The first one is the actual format we use to store the 3D sound fields.  So we used a first order ambisonics approach represented 3D sound fields.  So the basis of ambisonics is that you represent the 3D sound field as a pressure field centered  on the point surrounding the listener.  And basically you decompose that sound field into weighted sum of spherical harmonics basis  functions.  So the first order ambisonics we're using, zero and first order spherical harmonics.  And the main benefit of using ambisonics is it allows you to decouple the choice of loudspeaker  array from the way you capture the data.  So you can decide your loudspeaker array as opposed, you know, after you've captured  all data which is kind of nice.  And secondly, it allows you to easily rotate the sound field by just simply rotating spherical  harmonic spaces functions.  So that will come in useful later on when I go through the whole system.  So the second aspect is how actually the data is captured itself.  So instead of actually recording the audio stimuli live in the room, the approach that  we adopted was to record the audio stimuli in a studio and then basically to insert all  relevant audio debt cues by convolving the audio signal with a spatial room impulse response  which essentially modifies all the debt cues that are pertained to the particular test  environment.  Again, the main reason for adopting this approach is it allows greater efficiency in terms of  creating the audio stimuli for all the different test environments that we have and the different  sort of audio source distances so we can just generate the correct audio files for different  audio stimuli much quicker.  And it's also less prone to environmental noise when you're actually recording the data.  And then finally then is the virtual loudspeaker rendering.  So at this stage, you know, you have two choices.  You could just adopt my kind of conventional loudspeaker array approach.  And because we've used Ambisonics, we can choose any loudspeaker configuration basically  we want.  In this case, we're just going for an octagonal loudspeaker array centered on the listener,  on the user.  But instead of using a real array, we're using a virtual array.  So it's essentially trying to simulate the same sound field but over headphones rather  than over a real array.  And the challenge is to simulate the effect of the head and your shoulders and your pinnacle  on the audio signal.  And that varies according to the speaker position.  So that's captured basically by what's known as a head related impulse response or HR IR.  And essentially for each loudspeaker position to get the virtual signal what you have to  do is to involve what would be the loudspeaker output or the mono source with these two HR  IRs for the left and right ear to generate signal for the left and right ear.  Then you basically repeat the process for every speaker in the array and that allows  you then to get the virtual 3D sound field displayed overhead once.  Now you might think that this, well, let's just say that this is a maybe a strange approach  but we've done a lot of experiments previously to show that this type of approach to rendering  sound fields is perceptually equivalent to real sound sources.  So here is an experiment which is a kind of a typical distance judgment experiment that  you would see in the audio domain where you have to judge the distance of a sound source  and we're just comparing the various different methodologies.  So the two lines in the graph that are relevant are the black and blue line which the black  line representing having an actual loudspeaker specific distance and you have to judge how  far away that is and the set of blue line is the approach we were adopting in this paper,  the first order amysoics approach and we can show that within statistical significance  bounds that they are perceptually equivalent.  So the key thing that makes this work is head tracking.  So head tracking is required essentially to create stable 3D sound field.  So if I'm listening to 3D audio overhead once I rotate my head the sound field shouldn't  rotate with me.  This sound field should stay in the same position.  So head tracking really is the thing that makes this work in practice.  So the experiments themselves.  So the first experiment like I said before we're comparing visual distance and stimuli  and we're comparing the zone of congruence.  So we have three visual distances and we have 14 audio distances.  So the same audio distances are used for each of the three visual distances.  We have two audio stimuli and basically all combinations of audio and visual stimuli are  presented to the listener once.  So we had 20 test subjects and they were basically placed in a darkened room.  They were told to sit two meters away from the screen.  It was a polarized display in a darkened room and they were asked basically they were allowed  to present it with the UI you see at the bottom where they were able to play a sound repeatedly  and they were asked to judge whether the sound came from in front of the loudspeaker at the  loudspeaker or behind the loudspeaker.  So here are the results of the first experiment.  So this first slide now is just comparing the familiarity of the sound source and the  zone of congruence.  So let me just explain the graphs and it's the same convention for all the graphs to  see later on.  So we have the green line just basically represents a portion of people who think that the audio  and visual source come from the same position.  The red line is the percentage of people who think that the audio is in front of the video,  of the visual stimulus and the blue line is the proportion of people who think it's behind.  So you can see that the graphs follow what you would expect the pattern to be.  But if you're talking about the zone of congruence, we've kind of defined it here as the range  of depths for which the green value is highest, including the error bounds as well.  So if there's a large crossover between the two error bounds, we don't count it as part  of the zone of congruence.  So for this slide where we're just comparing for one visual distance, the difference between  the pink noise, audio stimulus and the female speech stimulus, you can see that we don't  really detect any significantly difference in the zone of congruence.  So it's pretty much the same.  So from somewhere closer than one meter, we could reasonably hypothesize that would be  greater than zero is the closest that the zone of congruence could be started at.  And it goes back to something like two and a half meters, which is half a meter behind  the position of the stimulus.  Now, we repeat this experiment again for the other visual distances and we just conclude  the same results at each distance that there is no statistical difference between the two  sound sources.  So moving on from that down, we're now comparing visual distance.  And I suppose we have two, four and eight meters and like you would expect, the zone  of congruence increases as the visual distance increases.  So again, you might have a range of say two and a half meters maximum, say for the two  meter visual distance that increases three and a half meters at four meters and at eight  meters is somewhere over five meters, which is probably significant, be higher.  So we were kind of limited in that scenario by the size of the room.  We actually couldn't go back much further than 11 meters.  That was just the back wall of the room.  So in the second test of our experiment then, we are comparing the zone of congruence with  respect to the, all right, okay, with respect to the test environment and whether we use  the speaker, the speaker, female.  And really hear what we're talking about.  The test environment one is just has more reverberance in it.  So reverberance is shown to be quite a strong depth view and it could effectively alter  perception.  So I'm just going to quickly summarize over the results.  Basically, although we saw some slight variations in the results, I don't think we were convinced  that there was actually any statistical differences in the zone of congruence across all the different  combination of factors.  In fact, we have known, I mean some known inaccuracies in the way we captured the data,  which could explain some of the difference that are apparent in that graph.  So our conclusion there basically is that there is no significant difference.  So finally then, just include, there's definitely a clear link between the zone of congruence  and the visual depth, the size of the zone of congruence and the visual depth.  But from all the other factors that we test, we couldn't really detect significant differences.  So for future work then, we would like to refine our capture procedures just to get  more robustness in them and to get more data by testing on more subjects.  And also then I guess finally we might like to determine how do large errors or perceivable  errors in audio and video position affect people's perception of quality and immersiveness.  So thank you very much for your attention.  I'd be glad to answer any questions if you have time.  Okay.  Thank you very much.  Questions for David?  All right.  Well, thank you very much.  All right.  Thank you.
Depth_perception_of_audio_sources_in_stereo_3D_environments_0_83.png Depth_perception_of_audio_sources_in_stereo_3D_environments_84_155.png Depth_perception_of_audio_sources_in_stereo_3D_environments_156_251.png Depth_perception_of_audio_sources_in_stereo_3D_environments_252_2519.png Depth_perception_of_audio_sources_in_stereo_3D_environments_2520_4547.png Depth_perception_of_audio_sources_in_stereo_3D_environments_4548_6779.png Depth_perception_of_audio_sources_in_stereo_3D_environments_6780_8423.png Depth_perception_of_audio_sources_in_stereo_3D_environments_8424_11243.png Depth_perception_of_audio_sources_in_stereo_3D_environments_11244_12995.png Depth_perception_of_audio_sources_in_stereo_3D_environments_12996_15263.png Depth_perception_of_audio_sources_in_stereo_3D_environments_15264_17471.png Depth_perception_of_audio_sources_in_stereo_3D_environments_17472_19031.png Depth_perception_of_audio_sources_in_stereo_3D_environments_19032_21671.png Depth_perception_of_audio_sources_in_stereo_3D_environments_21672_22811.png Depth_perception_of_audio_sources_in_stereo_3D_environments_22812_23627.png Depth_perception_of_audio_sources_in_stereo_3D_environments_23628_24335.png Depth_perception_of_audio_sources_in_stereo_3D_environments_24336_25967.png Depth_perception_of_audio_sources_in_stereo_3D_environments_25968_26030.png