A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> The geometric structure inherent in the definition of the shapes of three-dimensional objects and environments is used not just to define their relative motion and placement, but also to assist in solving many other problems of systems for producing pictures by computer. By using an extension of traditional structure information, or a geometric hierarchy, five significant improvements to current techniques are possible. First, the range of complexity of an environment is greatly increased while the visible complexity of any given scene is kept within a fixed upper limit. Second, a meaningful way is provided to vary the amount of detail presented in a scene. Third, “clipping” becomes a very fast logarithmic search for the resolvable parts of the environment within the field of view. Fourth, frame to frame coherence and clipping define a graphical “working set,” or fraction of the total structure that should be present in primary store for immediate access by the visible surface algorithm. Finally, the geometric structure suggests a recursive descent, visible surface algorithm in which the computation time potentially grows linearly with the visible complexity of the scene. <s> BIB001 </s> A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> We describe an adaptive display algorithm for interactive frame rates during visualization of very complex virtual environments. The algorithm relies upon a hierarchical model representation in which objects are described at multiple levels of detail and can be drawn with various rendering algorithms. The idea behind the algorithm is to adjust image quality adaptively to maintain a uniform, user-specified target frame rate. We perform a constrained optimization to choose a level of detail and rendering algorithm for each potentially visible object in order to generate the “best” image possible within the target frame time. Tests show that the algorithm generates more uniform frame rates than other previously described detail elision algorithms with little noticeable difference in image quality during visualization of complex models. CR <s> BIB002 </s> A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> *Virtual reality and virtual environments *The historical development of VR *3D computer graphics *Geometric modelling *Geometric transformations *A generic VR system *Animating the virtual environment *Physical simulation *Human factors *Virtual reality hardware *Virtual reality software *Virtual reality applications *The future <s> BIB003 </s> A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> This paper presents a new method of rendering for interaction with 3D virtual space with the use of gaze detection devices. In this method, hierarchical geometric models of graphic objects are constructed prior to the rendering process. The rendering process first calculates the visual acuity, which represents the importance of a graphic object for a human operator, from the gaze position of the operator. Second, the process selects a level from the set of hierachical geometric models depending on the value of visual acuity. That is, a simpler level of detail is selected where the visual acuity is lower, and a more complicated level is used where it is higher. Then, the selected graphic models are rendered on the display. This paper examines three visual characteristics to calculate the visual acuity: the central/peripheral vision, the kinetic vision, and the fusional vision. The actual implementation and our testbed system are described, as well as the details of the visual acuity model. <s> BIB004 </s> A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> Two user studies were performed to evaluate the effect of level-of-detail (LOD) degradation in the periphery of head-mounted displays on visual search performance. In the first study, spatial detail was degraded by reducing resolution. In the second study, detail was degraded in the color domain by using grayscale in the periphery. In each study, 10 subjects were given a complex search task that required users to indicate whether or not a target object was present among distracters. Subjects used several different displays varying in the amount of detail presented. Frame rate, object location, subject input method, and order of display use were all controlled. The primary dependent measures were search time on correctly performed trials and the percentage of all trials correctly performed. Results indicated that peripheral LOD degradation can be used to reduce color or spatial visual complexity by almost half in some search tasks with out significantly reducing performance. <s> BIB005 </s> A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> Gaze-contingent variable resolution display techniques allocate computational resources for image generation preferentially to the area around the center of gaze where visual sensitivity to detail is the greatest. Although these techniques are computationally efficient, their behavioral consequences with realistic tasks and materials are not well understood. The behavior of human observers performing visual search of natural scenes using gaze-contingent variable resolution displays is examined. A two-region display was used where a high-resolution region was centered on the instantaneous center of gaze, and the surrounding region was presented in a lower resolution. The radius of the central high-resolution region was varied from 1 to 15 degrees while the total amount of computational resources required to generate the visual display was kept constant. Measures of reaction time, accuracy, and fixation duration suggest that task performance is comparable to that seen for uniform resolution displays when the central region size is approximately 5 degrees. <s> BIB006 </s> A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> This paper describes hardware and software requirements for the development of a gaze-contingent virtual reality system which incorporates several cues to presence. The user’s gaze direction, as well as head position and orientation, are tracked to allow dynamic level-of-detail changes for rendering. Users can see themselves, rather than representations thereof, within blue-screened virtual environments, and limited vestibular feedback is provided through a motion simulator. The aesthetic appearance of environments is driven by advanced graphical techniques (i.e., radiosity) motivated by the goal of photorealistic representation of natural scenes. Taken together, the components identified in this paper describe a platform suitable for development of a variety of “smart” virtual environments. <s> BIB007 </s> A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> Level of Detail (LOD) techniques for real-time rendering and related perceptual issues have received a lot of attention in recent years. Researchers have also begun to look at the issue of perceptually adaptive techniques for plausible physical simulations. In this article, we are particularly interested in the problem of realistic collision simulation in scenes where large numbers of objects are colliding and processing must occur in real-time. An interruptible and therefore degradable collision-handling mechanism is used and the perceptual impact of this degradation is explored. We look for ways in which we can optimize the realism of such simulations and describe a series of psychophysical experiments that investigate different factors affecting collision perception, including eccentricity, separation, distractors, causality, and accuracy of physical response. Finally, strategies for incorporating these factors into a perceptually adaptive real-time simulation of large numbers of visually similar objects are presented. <s> BIB008 </s> A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> Gaze-contingent multiresolutional displays (GCMRDs) center high-resolution information on the user's gaze position, matching the user's area of interest (AOI). Image resolution and details outside the A01 are reduced, lowering the requirements for processing resources and transmission bandwidth in demanding display and imaging applications. This review provides a general framework within which GCMRD research can be integrated, evaluated, and guided. GCMRDs (or "moving windows") are analyzed in terms of (a) the nature of their images (i.e., "multiresolution," "variable resolution," "space variant," or "level of detail"), and (b) the movement of the A01 (i.e., "gaze contingent," "foveated," or "eye slaved"). We also synthesize the known human factors research on GCMRDs and point out important questions for future research and development. Actual or potential applications of this research include flight, medical, and driving simulators; virtual reality; remote piloting and teleoperation; infrared and indirect vision; image transmission and retrieval; telemedicine; video teleconferencing; and artificial vision systems. <s> BIB009 </s> A Breadth-First Survey of Eye Tracking Applications <s> Gaze-Contingent Displays <s> Visual Information Processing and Saccadic Eye Movements. Eye Movements in Reading and Language Processing. Computational Models of Eye Movement Control in Reading. Eye Movements in Human-Computer Interaction. Eye Movements in Media Applications and Communication. <s> BIB010
In general, eye-based interactive applications can be thought of as selective, since gaze is used to select or point to some aspect of the display, whether it is two-dimensional (e.g., desktop), collaborative, or immersive (such as a virtual environment). Mixing both directly interactive and indirectly "passive" usage styles of gaze are gaze-contingent displays. Here, gaze is used not so much as a pointing device, but rather as a passive indicator of gaze. Given the user's point of regard, a system can tailor the display so that the most informative details of the display are generated at the point of gaze but are degraded in some way in the periphery. The purpose of these displays is usually to minimize bandwidth requirements, as in video telephony applications, or in graphical applications where complex data sets cannot be fully displayed in real-time, by matching the display's spatial frequency to functional visual acuity across the visual field. When evaluating Gaze-Contingent Displays (GCDs), it is often necessary to distinguish between two main types of influences of fixation-dependent changes: those affecting perception and those affecting performance. As a general rule, perception is more sensitive than performance. That is, it may be possible to degrade a display to a quite noticeable effect without necessarily degrading performance. In either case, one of the main difficulties that must be addressed is the latency of the system. Without predictive capabilities, most gaze-contingent displays will lag behind the user's gaze somewhat, usually by a constant amount of time proportional to both measurement of gaze (which may take up to the time it takes to display a single frame of video, typically 16 ms for a 60 Hz video-based tracker), and the subsequent time it takes to refresh the gaze-contingent display (this may take another 33 ms for a system with update rate of 30 frames per second). Two main types of gaze-contingent applications are discussed: screen-based and model-based. The former deals with image (pixel) manipulation, while the latter is concerned with the manipulation of graphical objects or models prior to rendering. Screen-Based Displays. Loschky and McConkie (2000) conducted an experiment on a gaze-contingent display investigating spatial, resolutional, and temporal parameters affecting perception and performance. Two key issues addressed are the timing of GCDs and the detectability of the peripherally degraded component of the GCD. In all experiments, monochromatic photographic scenes were used as stimuli with a circular, high-resolution window surrounded by a degraded peripheral region. In one facet of the experiment, it was found that for an image change to go undetected, it must be started within 5 ms after the end of the eye movement. Detection likelihood rose quickly beyond that point. In another facet of the study concerning detection of peripheral degradation, results showed that the least peripheral degradation went undetected even at the smallest window size (2 • ), where the opposite was true with the highest level of degradationit was quite detectable at even the largest window size (5 • ). It was found that the generation of an imperceptible GCD was quite difficult in comparison to the generation of a GCD which does not deteriorate performance. While greater delays (e.g., 15 ms) and greater degradation produce detectable visual artifacts, they appear to have minimal impact on performance of visual tasks when there is a 4.1 • high-resolution area centered at the point of gaze. BIB006 investigated behavioral effects of a two-region gaze-contingent display. A central high-resolution region, varying from 1 to 15 degrees, was presented at the instantaneous center of gaze during a visual search task. Parkhurst et al. found that reaction time and accuracy co-varied as a function of the central region size and note this as a clear indicator of a strategic speed/accuracy tradeoff where participants favor speed in some conditions and accuracy in others. For small central region sizes, slow reaction times are accompanied by high accuracy. Conversely, for large central regions sizes, fast reaction times are accompanied by low accuracy. A secondary finding indicated that fixation duration varies as a function of central region size. For small central region sizes, participants tend to spend more time examining each fixation than under normal viewing conditions. For large central regions, fixation durations tend to be closer to normal. In agreement with reaction time and accuracy, fixation duration is approximately normal (comparable to that seen for uniform resolution displays) with a central region size of 5 • . By distinguishing the effects of gaze-contingent windowing and peripheral filtering, (in this issue) have recently shown that peripheral target detection with GCDs may be negatively impacted by not only the loss of perceptual detail due to filtering, but also by the mere presence of the gaze-contingent window (the window effect). An important consideration in Reingold and Loschky's study is the saliency of the gaze-contingent window boundary: whether the gaze-contingent window is sharply defined or smoothly gradated appears to have no effect on peripheral target acquisition. For screen-based VR rendering the work of BIB005 is particularly relevant. Watson et al. studied the effects of Level Of Detail (LOD) peripheral degradation on visual search performance, evaluating both spatial and chrominance detail degradation effects in Head Mounted Displays (HMDs). To sustain acceptable frame rates, two polygons were texture mapped in real-time to generate a high resolution inset within a low resolution display field. Watson et al. suggest that visual spatial and chrominance complexity can be reduced by almost half without degrading performance. In an approach similar to Watson et al.'s, Reddy (1998) used a view-dependent LOD technique to evaluate both perceptual effects and system performance gains. Reddy reported a perceptually modulated LOD system which affords a factor 4.5 improvement in frame rate. For excellent reviews of Gaze-Contingent Displays, see BIB009 as well as  . Model-Based Graphical Displays. As an alternative to the screen-based peripheral degradation approach, model-based methods aim at reducing resolution by directly manipulating the model geometry prior to rendering. The technique of simplifying the resolution of geometric objects as they recede from the viewer, as originally proposed by BIB001 , is now standard practice, particularly in real-time applications such as VR BIB003 ). Clarke's original criteria of using the projected area covered by the object for descending the object's LOD hierarchy is still widely used today. However, as Clarke suggested, the LOD management typically employed by these polygonal simplification schemes relies on pre-computed fine-to-coarse hierarchies of an object. This leads to uniform, or isotropic, object resolution degradation. A gaze-contingent model-based adaptive rendering scheme was proposed by BIB004 , where three visual characteristics were considered: central/peripheral vision, kinetic vision, and fusional vision. The LOD algorithm generated isotropically degraded objects at different visual angles. Although the use of a binocular eye tracker was proposed, the system as discussed used only head tracking as a substitute for gaze tracking. Isotropic object degradation is not always desirable, especially when viewing large objects at close distances. In this case, traditional LOD schemes will display an LOD mesh at its full resolution even though the mesh may cover the entire field of view. Since acute resolvability of human vision is limited to the central 5 • , object resolution need not be uniform. This is the central tenet of gaze-contingent systems. Numerous multiresolution mesh modeling techniques suitable for gaze-contingent viewing have recently been developed  . Due to the advancements of multiresolution modeling techniques and to the increased affordability of eye trackers, it is now becoming feasible to extend the LOD approach to gaze-contingent displays, where models are rendered nonisotropically. An early example of a nonisotropic model-based gazecontingent system, where gaze direction is directly applied to the rendering algorithm, was presented by Levoy and Whitaker (1990) , where a spatially adaptive near real-time ray tracer for volume data displayed an eye-slaved ROI by modulating both the number of rays cast per unit area on the image plane and the number of samples drawn per unit length along each ray as a function of local retinal acuity. The raytraced image was sampled by a nonisotropic convolution filter to generate a 12 • foveal ROI within a 20 • mid-resolution transitional region. Based on preliminary estimates, Levoy and Whitaker suggested a reduction in image generation time by a factor of up to 5. For environments containing significant topological detail, such as virtual terrains, rendering with multiple levels of detail, where the level is based on user position and gaze direction, is essential to provide an acceptable combination of surface detail and frame rate BIB007 . BIB007 used an eye tracker as an indicator of gaze in a gaze-contingent multiresolution terrain navigation environment. A surface, represented as a quadrilateral mesh, was divided into fixed-size (number of vertices) sub-blocks, allowing rendering for variable LOD on a per-sub-block basis. Resolution level was chosen per sub-block, based on viewer distance and direction of gaze. To exaggerate the gaze-contingent effect, fractal mountains disappear when not in view. More recent work on gaze-contingent LOD modeling has been carried out by Luebke and Erikson (1997) . Luebke and Erikson present a view-dependent LOD technique suitable for gaze-contingent rendering. While simplification of individual geometric objects is discussed in their work, it appears the strategy is ultimately directed toward solving the interactive "walkthrough" problem BIB002 . In this application, the view-dependent LOD technique seems more suitable to the (possibly) gaze-contingent rendering of an entire scene or environment. Recently, Luebke, Hallen, Newfield, and Watson (2000) have developed a gaze-directed LOD technique to facilitate the gazecontingent display of geometric objects. To test their rendering approach Luebke et al. employed a table-mounted monocular eye tracker to measure the viewer's real-time location of gaze over a desktop display. A new object-based LOD method has been developed by  . The technique is similar to Luebke and Erikson's and to Ohshima et al.' s, where objects are modeled for gaze-contingent viewing. Unlike the approach of Ohshima et al., resolution degradation is applied nonisotropically. The spatial degradation function for LOD selection differs significantly from the area-based criteria originally proposed by Clarke. Instead of evaluating the screen coverage of the projected object, the degradation function is based on the evaluation of visual angle in world coordinates. A three-dimensional spatial degradation function was obtained from human subject experiments in an attempt to imperceptibly display spatially degraded geometric objects. System performance measurements indicate an approximate overall 10-fold average frame rate improvement during gaze-contingent viewing. Another interesting approach to gaze-contingent modeling for real-time graphics rendering was taken by BIB008 and BIB010 . Instead of degrading the spatial resolution of peripherally located geometric objects, O'Sullivan and Dingliana (2001) consider a degradable collision handling mechanism to effectively limit object collision resolution outside a foveal ROI. When the viewer is looking directly at a collision, it is given higher (computational) priority than collisions occurring in the periphery. Object collisions given higher priority are allocated more processing time so that the contact model and resulting response is more believable. O'Sullivan and Dingliana tested viewers' sensitivity to collision resolution (size of gap between colliding objects) at eccentricity and noted a significant fall-off in detection accuracy at about 4 • visual angle. Based on their previous psychophysical findings, BIB010 developed a gaze-contingent collision handling system. Two variants of the gaze-contingent system were compared, each containing a high-priority ROI wherein collisions were processed at higher resolution than outside the ROI. In the tracked case, the high-priority ROI was synchronized to the viewer's tracked gaze position, while in the random case, the ROI position was determined randomly every 5 frames. O'Sullivan et al. report an overall improvement in the perception of the tracked simulation.