A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> GPU-BASED AURALIZATION <s> Auralization is a term introduced to be used in analogy with visualization to describe rendering audible (imaginary) sound fields. Several modeling methods are available in architectural acoustics for this purpose. If auralization is done by computer modeling, it can be thought of as "true" acoustical computer-aided design. Together with new hardware implementations of signal processing routines, auralization forms the basis of a powerful new technology for room simulation and aural event generation. The history, trends, problems, and possibilities of auralization are described. The discussion primarily deals with auralization of auditorium acoustics andloudspeaker installations. The advantages and disadvantages of various approaches are discussed, as are possible testing and verification techniques. The possibility of using acoustic scale models for auralization is also discussed. \ Demonstrations of auralization have been made, but still the technology's ability to reproduce the subjective impression of the audible characteristics of a hall accurately remains to be verified. This limits the credibility of auralization as a design tool, and verification of auralization the foremost problem to be attacked at this time. The verification problem also applies to the basic room impulse response prediction programs. The combination of auralization with transaural reproduction, room equalization, and active noise control could make it possible to expand the applications of the technology beyond the laboratory and beyond simple headphone reproduction. A large number of interesting applications outside room and psychoacoustics research are conceivable, the most interesting of which are probably its use in information, education, and entertainment. <s> BIB001 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> GPU-BASED AURALIZATION <s> This paper describes a public-domain database of high-spatial-resolution head-related transfer functions measured at the UC Davis CIPIC Interface Laboratory and the methods used to collect the data.. Release 1.0 (see http://interface.cipic.ucdavis.edu) includes head-related impulse responses for 45 subjects at 25 different azimuths and 50 different elevations (1250 directions) at approximately 5/spl deg/ angular increments. In addition, the database contains anthropometric measurements for each subject. Statistics of anthropometric parameters and correlations between anthropometry and some temporal and spectral features of the HRTFs are reported. <s> BIB002 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> GPU-BASED AURALIZATION <s> An experimental procedure and signal processing method to measure Head Related Transfer Functions (HRTFs) is reviewed. The technique based on Fourier analysis system identification has an advantage over the commonly used maximum-length sequence and Golay methods if nonlinear distortions are present in the loudspeakers and their power amplification circuits. The method has been used to produce a new public HRTF database. Compared to existent public domain databases, these transfer functions have been measured at points spaced more densely and uniformly around the subject, which poses an advantage for fitting and interpolating methods. The measured HRTFs (seven subjects to date) are available by request from the authors. <s> BIB003 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> GPU-BASED AURALIZATION <s> The Fourier transform is a well known and widely used tool in many scientific and engineering fields. The Fourier transform is essential for many image processing techniques, including filtering, manipulation, correction, and compression. As such, the computer graphics community could benefit greatly from such a tool if it were part of the graphics pipeline. As of late, computer graphics hardware has become amazingly cheap, powerful, and flexible. This paper describes how to utilize the current generation of cards to perform the fast Fourier transform (FFT) directly on the cards. We demonstrate a system that can synthesize an image by conventional means, perform the FFT, filter the image, and finally apply the inverse FFT in well under 1 second for a 512 by 512 image. This work paves the way for performing complicated, real-time image processing as part of the rendering pipeline. <s> BIB004 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> GPU-BASED AURALIZATION <s> Many contemporary visualization tools comprise some image filtering approach. Since image filtering approaches are very computationally demanding, the acceleration using graphics-hardware (GPU) is very desirable to preserve interactivity of the main visualization tool itself. In this article we take a close look on GPU implementation of two basic approaches to image filtering - Fast Fourier Transform (frequency domain) and convolution (spatial domain). We evaluate these methods in terms of the performance in real time applications and suitability for GPU implementation. Convolution yields better performance than Fast Fourier Transform (FFT) in many cases; however, this observation cannot be generalized. In this article we identify conditions under which the FFT gives better performance than the corresponding convolution and we assess the different kernel sizes and issues of application of multiple filters on one image. <s> BIB005 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> GPU-BASED AURALIZATION <s> To be immersed in a virtual environment, the user must be presented with plausible sensory input including auditory cues. A virtual (three-dimensional) audio display aims to allow the user to perceive the position of a sound source at an arbitrary position in three-dimensional space despite the fact that the generated sound may be emanating from a fixed number of loudspeakers at fixed positions in space or a pair of headphones. The foundation of virtual audio rests on the development of technology to present auditory signals to the listener's ears so that these signals are perceptually equivalent to those the listener would receive in the environment being simulated. This paper reviews the human perceptual and technical literature relevant to the modeling and generation of accurate audio displays for virtual environments. Approaches to acoustical environment simulation are summarized and the advantages and disadvantages of the various approaches are presented. <s> BIB006 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> GPU-BASED AURALIZATION <s> The generation of spatial audio is computationally very demanding and therefore, accurate spatial audio is typically overlooked in games and virtual environment applications thus leading to a decrease in both performance and the user's sense of presence or immersion. Previous work has examined the application of the graphics processing unit to the generation of real-time spatial audio. In particular, a GPU-based convolution method was developed that allowed for real-time convolution between an arbitrarily sized auditory signal and a filter. Despite the large computational savings, that GPU-based method introduced noise/artifacts to the lower-order bytes of the resulting output signal which may have resulted in a number of perceptual consequences. This work builds upon the previous GPU-based convolution method and describe a GPU-based convolution method that employs a superior GPU that eliminates the noise/artifacts of the previous method and provides further computational savings. <s> BIB007
Kleiner, Dalenbäck, & Svensson BIB001 define auralization as "the process of rendering audible, by physical or mathematical modeling, the sound field of a source in space in such a way as to simulate the binaural listening experience at a given position in the modeled space". The goal of auralization is to recreate a particular listening environment taking into account the environmental acoustics (i.e., the "room acoustics") and the listener's characteristics. Auralization is typically defined in terms of the binaural room impulse response (BRIR). The BRIR represents the response of a particular acoustical environment and human listener to sound energy and captures the room acoustics for a particular sound source and listener configuration. The recorded response then forms the basis of a filter that is used to process source sound material (anechoic or synthesized sound) via a convolution operation before presenting it to the listener. When the listener is presented with this filtered sound, the direct and reflected sounds of the environment are reproduced in addition to directional filtering e ects introduced by the original listener  . Although interlinked, for simplicity and reasons of practicality, the room response and the response of the human receiver are commonly determined separately and combined via a post-processing operation to provide an approximation to the actual BRIR BIB001 . The response of the room is known as the room impulse response (RIR) and captures the reflection properties (reverberation), di raction, refraction, sound attenuation, and absorption properties of a particular room configuration (i.e., the "room acoustics"). The response of the human receiver captures the direction dependent e ects introduced by the listener due to the listener's physical make-up (e.g., pinna, head, shoulders, neck, and torso) and is known as the head related transfer function (HRTF). HRTFs encompass various sound localization cues including interaural time di erences (ITDs), interaural level di erences (ILDs), and the changes in the spectral shape of the sound reaching a listener. The HRTF modifies the spectrum and timing of sound signals reaching each ear in a location-dependent manner  . The process of collecting a set of individualized HRTFs is an extremely di cult, time consuming, tedious, and delicate process requiring the use of special equipment and environments such as an anechoic chamber. Although the HRTF of individuals can vary greatly, it is impractical to use individualized HRTFs and as a result, generalized (or generic) non-individualized HRTFs are often used instead. Non-individualized HRTFs can be obtained using a variety of methods such as measuring the HRTFs of an anthropomorphic "dummy" head, or of an above average human localizer or averaging the HRTFs measured from several di erent individuals (and/or "dummy heads"). Several non-individualized HRTF datasets are freely available BIB002  BIB003  (see BIB006 for greater details regarding the problems associated with non-individualized HRTFs). The output of the methods used to determine the HRTF and the RIR is typically a transfer function which forms the basis of a filter that can be used to modulate source sound material (i.e., anechoic or synthesized sound) via a convolution operation which is still primarily performed in software in the time domain. When the filtered sounds are presented to the listener, in the case of HRTFs, they create the impression of a sound source located at the corresponding HRTF measurement position while when considering the RIR, the filtered sounds recreate a particular acoustic environment. However, convolution is a computationally expensive operation especially when considering long filters associated with HRTFs and RIRs (filters with 512 coe cients are not uncommon) thus limiting their use to non-real-time applications. Performance improvements can be made by performing the convolution operation in the frequency domain  . In order to accomplish this, the input and filters must be converted to their frequency domain representation using the fast Fourier transform; a time consuming process when performed in software making it impractical for real-time, interactive use. Recent work in image processing has established a GPU-based convolution method capable of performing a two-dimensional convolution operation in real-time BIB005 BIB004 . In addition to software-based convolution methods, programmable DSP cards are available which allow for hardware-based convolution thus greatly improving performance. However, these boards are very specialized and typically only available to product developers and not the general consumer  . Cowan and Kapralos presented a GPU-based convolution method using the OpenGL shading language (GLSL)  BIB007 . A comparison of the computational running time requirements for both the conventional (software-based) and GPU-based convolution method was made by measuring the computational time requirements when convolving a particular input signal with an HRTF using two video cards (GPUs) for further comparisons: the NVIDIA GTX 8800 and the NVIDIA GTX 280 which supports double precision floating point operations. Both video cards supported realtime convolution for an input signal whose size ranged from 5,000 to 60,000 and a filter containing 200 samples; approximately four and two milliseconds (including any required CPU processing time) for the GTX 8800 and the GTX 280 respectively, in contrast to the software-based method whose computational requirements increased linearly with increasing input size (ranging from approximately 4 to 25 ms). With a constant running time of 2 ms for the convolution operation (the NVIDIA GTX 280 video card), realistic spatial auditory cues can be incorporated into video games and virtual environments in real-time. Rather than using measured HRTFs (individualized or non-individualized), Röber  et al. describe an alternative approach whereby the HRTF is modeled using GPU-based ray tracing techniques using a 3D mesh model of the upper torso including the pinna. The HRTFs are approximated by simulating an impulse response that is recorded by a semispherical surface placed inside the ear canal of the model. The 3D mesh can be changed easily which makes the method suitable for measuring individual HRTFs. The sound source is approximated by a point light and the microphone is represented by a hemispherical camera surface. To simplify computation, the algorithm is applied to high frequencies, since these hold important spatialization cues. The lower frequencies usually bend around the head and are not as important for spatialization and therefore are approximated by amplitude and time shifts. Verification of the method with human participants is required to compare the resulting HRTFs with existing non-individualized HRTFs given the assumptions made (e.g., ignoring the lower frequencies).