A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> ). <s> Abstract A concept on integration of 3D visual and auditory rendering is proposed in this paper. By the auditory rendering we mean the computer acoustical simulation with direct sound outputs that is based on the physical model simulation. The so-called visual rendering means the computer generation of 3D images of real world and even virtual world, but without sounds. By integration of 3D visual and auditory rendering we mean while computer generates a 3D image of physical model of real world, it generates the sound based on the physical model displayed if there is a reverberation-free sound source. Based on some kinds of similarity between the propagation nature of light and sound, a modified radiosity algorithm has been proposed for integrating implementation. The algorithm is suitable for both visual and room auditory rendering. It means for a given enclosure, once the form factors are calculated, they can be applied for calculating both the light radiosities and sound radiosities simultaneously. The features of the algorithm are indicated. The implementation techniques are given in detail. The integration of visual and auditory rendering is a big step toward real world simulation, furthermore it will extend the research direction of visualization in scientific computing (ViSC) to visualization and audiobilization in scientific computing (VAiSC). The combination of VAiSC with virtual reality technique will improve human-computer interface greatly. <s> BIB001 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> ). <s> We capitalize on recent advances in modern programmable graphics hardware, originally designed to support advanced local illumination models for shading, to instead perform two different kinds of global illumination models for light transport. We first use the new floating-point texture map formats to find matrix radiosity solutions for light transport in a diffuse environment, and use this example to investigate the differences between GPU and CPU performance on matrix operations. We then examine multiple-scattering subsurface light transport, which can be modeled to resemble a single radiosity gathering step. We use a multiresolution meshed atlas to organize a hierarchy of precomputed subsurface links, and devise a three-pass GPU algorithm to render in real time the subsurface-scattered illumination of an object, with dynamic lighting and viewing. <s> BIB002 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> ). <s> This paper explores acoustical (or time-dependent) radiosity--a geometrical-acoustics sound-field prediction method that assumes diffuse surface reflection. The literature of acoustical radiosity is briefly reviewed and the advantages and disadvantages of the method are discussed. A discrete form of the integral equation that results from meshing the enclosure boundaries into patches is presented and used in a discrete-time algorithm. Furthermore, an averaging technique is used to reduce computational requirements. To generalize to nonrectangular rooms, a spherical-triangle method is proposed as a means of evaluating the integrals over solid angles that appear in the discrete form of the integral equation. The evaluation of form factors, which also appear in the numerical solution, is discussed for rectangular and nonrectangular rooms. This algorithm and associated methods are validated by comparison of the steady-state predictions for a spherical enclosure to analytical solutions. <s> BIB003 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> ). <s> The modelling of room acoustics remains a difficult and computationally expensive task. Two main techniques exist, one focusing on the real physical wave-oriented sound propagation, while the other approximates sound waves using rays and ray-tracing techniques. One of the most popular wave-based technique are waveguide meshes, which simulate the propagation of sound using time domain difference models. In this paper we discuss a realtime implementation of the waveguide technique using fragment shaders and computer graphics hardware. Graphics hardware is well suited for parallel implementations and thus able to speedup the computation. Additionally, we discuss optimal sampling on hexagonal grids to further increase the computational efficiency. We compare both implementations and discuss the results achieved. <s> BIB004 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> ). <s> Real-time sound rendering engines often render occlusion and early sound reflection effects using geometrical techniques such as ray or beam tracing. They can only achieve interactive rendering for environments of low local complexity resulting in crude effects which can degrade the sense of immersion. However, surface detail or complex dynamic geometry has a strong influence on sound propagation and the resulting auditory perception. This paper focuses on high-quality modeling of first-order sound scattering. Based on a surface-integral formulation and the Kirchhoff approximation, we propose an efficient evaluation of scattering effects, including both diffraction and reflection, that leverages programmable graphics hardware for dense sampling of complex surfaces. We evaluate possible surface simplification techniques and show that combined normal and displacement maps can be successfully used for audio scattering calculations. We present an auralization framework that can render scattering effects interactively thus providing a more compelling experience. We demonstrate that, while only considering first order phenomena, our approach can provide realistic results for a number of practical interactive applications. It can also process highly detailed models containing millions of unorganized triangles in minutes, generating high-quality scattering filters. Resulting simulations compare well with on-site recordings showing that the Kirchhoff approximation can be used for complex scattering problems. <s> BIB005 </s> A Review of Spatial Sound for Virtual Environments and Games with Graphics Processing Units <s> ). <s> The modeling of room acoustics and simulation of sound wave propagation remain a difficult and computationally expensive task. Two main techniques have evolved, with one focusing on a real physical - wave-oriented - sound propagation, while the other approximates sound waves as rays using raytracing techniques. Due to many advances in computer science, and especially computer graphics over the last decade, interactive 3D sound simulations for complex and dynamic environments are within reach. Inthispaperweanalyzesoundpropagationintermsofacoustic energy and explore the possibilities to map these concepts to radiometry and graphics rendering equations. Although we concentrate on ray-based techniques, we also partially consider wavebased sound propagation effects. The implemented system exploits modern graphics hardware and rendering techniques and is able to efficiently simulate 3D room acoustics, as well as to measure simplified personal HRTFs through acoustic raytracing. <s> BIB006
By assuming that the pressure at each of these elements is a linear combination of a finite number of basis functions, the boundary integral form of the wave equation can be solved  . The acoustical radiosity method, a modified version of the image synthesis radiosity technique, is an example of such an approach BIB003 BIB001 . The numerical approximations associated with wave-based methods are computationally prohibitive making the use of traditional softwarebased methods impractical except for the simplest static environments. Aside from basic or simple environments, such techniques are currently beyond our computational ability for real-time, interactive virtual environment and video game applications. That being said, the processing power inherent in GPUs has been exploited in a number of wave-based methods. Röber et al. present a (low-frequency) wave-based acoustical modeling method that made use of the GPU and in particular, fragment shaders, 3D textures, and the OpenGL framebu er objects extension, in order to take advantage of the inherent parallelism of wave-based solutions to acoustical modeling BIB004 . The one-dimensional mesh is extended by constructing a digital mesh from bi-linear delay lines that connect junctions that act as temporal and spatial sampling points. The programmable vertex shader is used to implement computations on a per vertex basis on a 3D space and the fragment shader is used to compute the final pixel color. Waveguide node data is stored in three bu ers that are combined into one RGB texture with the data stored in the red and blue components and the geometry and boundary coe cients in the green channel. During each time frame, the fragment shader computes the di erence equations for each node in the mesh and stores the result in the bu er. They have used a body centered cubic grid (BCC) which is an hexagonal lattice that requires only 70% of the sampling points compared to the usual rectilinear grid which is a cubic cartesian lattice. This data structure reduces the computation load by 2 and can be decomposed into cubic grids that make the GPU implementation straightforward. The limitations of this approach are a direction dependent dispersion error and the finite mesh resolution to model boundary behavior. Also, the approach implements 2D meshes only. The system was tested on a PC with an AMD64 4000+ dual-core CPU and an NVIDIA GeForce 7900 GT video card and showed speed-ups of factors of from 4.5 to 69 when compared to a software-based implementation. However, the CPU implementation was not optimized. Tsingos et al. BIB005 present a new approach for highquality modeling of first-order sound propagation. The method employs a surface-integral formulation and Fig. (2) . Wave-based acoustical modeling. Reprinted from BIB006 . Kirchho approximation, which can be viewed as a hybrid between ray tracing and wave acoustics. In contrast to other sound propagation techniques, it is capable of rendering very complex scenes and implements both di raction and reflection in a unified manner. The method maps well to graphical hardware since it computes the scattering from detailed, dynamic geometry. It combines the HelmotKirchho theorem with the Kirchho approximation to derive an expression for first order scattering e ects. A GPU implementation in this case is suitable because the above formulation is similar to the reflective shadow map that is introduced to compute interactive first order global illumination e ects. It also allows the implementation of a level-ofdetail approach that reduces the geometry processing for audio rendering while preserving the scattering behavior of complex surfaces by allowing bump or displacement mapping. There are two steps in the computation: i) during the first step, all scattering surfaces visible from the source are determined and sampled, and ii) in the second step, the evaluation and summation of the di erential contribution of clocked plus reflected wavefronts for all surface samples is made. The first task is implemented using a computer graphics shadow mapping source-view technique that renders the scene from the location of the sound source. For the second task, a hierarchal integration method known as "mipmapping" is used that requires log(rez)/log(k) render passes where rez is the rendering resolution and k is the reduction factor. At each pass a k k block of values is summed to give a single value which is recursively integrated in the next pass until the value of the integral is reached. Both visual rendering and the calculation of audio scattering coe cients are done on the GPU. The auralization is achieved by reequalizing performed asynchronously on the CPU. The method was tested on a Pentium IV 3.4GHz CPU and an NVIDIA GeForce 8800 GTX graphics card and was compared to a C++ implementation on the CPU. For an interactive scenario the GPU-based method was found to be 40 times faster. The limitations of this method are that it is prone to aliasing due to insu cient sampling at high frequencies and is also limited to first order scattering and therefore cannot be used for some audio e ects such as reverberation and occlusion. Also, by using a frequency domain approach, essential for an e cient implementation, this method introduces an approximation because of the limited number of frequency bands. Despite the progress to-date, plenty of work remains to allow for real-time, accurate, wave-based acoustical modeling on the GPU. Of course, depending on the application, completely and faithfully recreating the acoustics of a particular environment may not be necessary; hearing is a perceptual process and there is no one-to-one mapping between physical acoustical phenomena and our perception of these phenomena. Therefore, accuracy may not always be necessary. Greater work needs to be conducted to examine this issue more carefully. Finally, although the number of e orts investigating wave-based acoustical modeling using the GPU are limited, extensive work has been carried out using such techniques for the computation of global illumination. A number of wave-based techniques utilizing the GPU are available including radiosity BIB002 , etc. If suitably modified, these techniques could be applied to acoustical modeling applications.