We propose a framework called inverse augmented reality (IAR) which describes the scenario that a virtual agent living in the virtual world can observe both virtual objects and real objects. This is different from the traditional augmented reality. The traditional virtual reality, mixed reality and augmented reality are all generated for humans, i.e., they are human-centered frameworks. On the contrary, the proposed inverse augmented reality is a virtual agent-centered framework, which represents and analyzes the reality from a virtual agent's perspective. In this paper, we elaborate the framework of inverse augmented reality to argue the equivalence of the virtual world and the physical world regarding the whole physical structure.
In the past, a lot of related researches about augmented reality have been presented. Before IAR, some novel styles of reality have been proposed. For example, Lifton al @cite_4 proposed the dual reality'' system to make the virtual world and the physical world be corresponding to each other. Roo al @cite_0 proposed the one reality'' system, which contained a 6-level mixture of virtual and real contents ranging from purely physical to purely virtual world. But they all describe the mixed reality from the perspective of humans, ignoring the view from the virtual world.
@cite_0 @cite_4
2744963411 2043470575 1482795035 2584852980 2160971146 1544132830 2490487394 2767058122 2126514505 2532511219 2072903758 2344474200 2800595980 2963826402 1953967342 2119300480 2517064749 2286501983 2110365529 2890415768
Most of our daily activities take place in the physical world, which inherently imposes physical constraints. In contrast, the digital world is very flexible, but usually isolated from its physical counterpart. To combine these two realms, many Mixed Reality (MR) techniques have been explored, at different levels in the continuum. In this work we present an integrated Mixed Reality ecosystem that allows users to incrementally transition from pure physical to pure virtual experiences in a unique reality. This system stands on a conceptual framework composed of 6 levels. This paper presents these levels as well as the related interaction techniques. We present a method for estimating the real-world lighting conditions within a scene in real-time. The estimation is based on the visual appearance of a human face in the real scene captured in a single image of a monocular camera. In hardware setups featuring a user-facing camera, an image of the user’s face can be acquired at any time. The limited range in variations between different human faces makes it possible to analyze their appearance offline, and to apply the results to new faces. Our approach uses radiance transfer functions – learned offline from a dataset of images of faces under different known illuminations – for particular points on the human face. Based on these functions, we recover the most plausible real-world lighting conditions for measured reflections in a face, represented by a function depending on incident light angle using Spherical Harmonics. The pose of the camera relative to the face is determined by means of optical tracking, and virtual 3D content is rendered and overlaid onto the real scene with a fixed spatial relationship to the face. By applying the estimated lighting conditions to the rendering of the virtual content, the augmented scene is shaded coherently with regard to the real and virtual parts of the scene. We show with different examples under a variety of lighting conditions, that our approach provides plausible results, which considerably enhance the visual realism in real-time Augmented Reality applications. Like virtual reality, augmented reality is becoming an emerging platform in new application areas for museums, edutainment, home entertainment, research, industry, and the art communities using novel approaches which have taken augmented reality beyond traditional eye-worn or hand-held displays. In this book, the authors discuss spatial augmented reality approaches that exploit optical elements, video projectors, holograms, radio frequency tags, and tracking technology, as well as interactive rendering algorithms and calibration techniques in order to embed synthetic supplements into the real environment or into a live video of the real environment. Special Features: - Comprehensive overview - Detailed mathematical equations - Code fragments - Implementation instructions - Examples of Spatial AR displays The authors have put together a preliminary collection of Errata. Updates will be posted to this site as necessary. Augmented reality combines real footage taken of a scene with virtual elements. However, most current methods rely on camera localisation and 3D reconstruction or point cloud generation in order to integrate augmented reality to the footage. In contrast, in this work we present a novel method to augment virtual reality to the scene based on the recognition of dominant planes in interior scenes. Our method uses a rule system to select predefined decisions on a set of variables in order to infer dominant planes in the scene. For this, we propose to combine information from texture features, a measure of blurring in the dominant planes, and a segmentation of regions in a scene based on superpixels. The rule system infers the regions corresponding to the dominant planes, whilst light intensity in the scene is inferred from segmented regions. We also propose an approach to remove regions misclassified as dominant planes. Finally, the floor in the scene is recognised as the most dominant plane and then replaced with an augmented texture. We demonstrate our approach in a video sequence where our method is applied in a frame-to-frame basis, thus, for each single image in the video sequence, the floor is automatically recognised as the most dominant plane and then replaced with a virtual texture and furthermore, whose appearance is modified according to our coarse light model inferred from our approach. This paper proposes the convergence of sensor networks and virtual worlds not only as a possible solution to their respective limitations, but also as the beginning of a new creative medium. In such a “dual reality,” both real and virtual worlds are complete unto themselves, but also enhanced by the ability to mutually reflect, influence, and merge by means of sensor actuator networks deeply embedded in everyday environments. This paper describes a full implementation of a dual reality system using a popular online virtual world and a human-centric sensor network designed around a common electrical power strip. Example applications (e.g., browsing sensor networks in online virtual worlds), interaction techniques, and design strategies for the dual reality domain are demonstrated and discussed. This work presents a system for creating augmented virtuality. We present a tool and methodology that can be used to create virtual worlds that are augmented by video textures taken of real world objects. These textures are images automatically extracted from images of real world scenes. The idea is to construct and update, in real time, a representation of the salient and relevant features of the real world. This idea has the advantage of constructing a virtual world that has the relevant data of the real world, but maintaining the flexibility of a virtual world. One advantage of the virtual-real world representation is that it is not dependent on physical location and can be manipulated in a way not subject to the temporal, spatial, and physical constraints found in the real world. The problem of development a sustainable efficient and accurate augmented reality system possessing the interactivity property and basically implementing markerless approach for supplementing the real scene is addressed in this paper. Custom implementation is based on applying of feature-based SLAM algorithm, which is resistant to change light conditions, able to track both textured and non-textured objects in real-time mode achieved by introducing a once-performed offline system teaching step. The proposed solution is universal because of the system’s configurability in accordance with the desirable result. Augmented Reality (AR) scenarios aim to provide realistic blending between real world and virtual objects. A key factor for realistic AR is thus a correct illumination simulation. This consists in estimating the characteristics of real light sources and use them to model virtual lighting. In this paper, we briefly introduce a novel method for recovering both 3D position and intensity of multiple light sources using detected cast shadows. Our algorithm has been successfully tested on a set of real scenes where virtual objects have visually coherent shadows. In this paper we discuss augmented reality (AR) displays in a general sense, within the context of a reality-virtuality (RV) continuum, encompassing a large class of mixed reality' (MR) displays, which also includes augmented virtuality (AV). MR displays are defined by means of seven examples of existing display concepts in which real objects and virtual objects are juxtaposed. Essential factors which distinguish different MR display systems from each other are presented, first by means of a table in which the nature of the underlying scene, how it is viewed, and the observer's reference to it are compared, and then by means of a three dimensional taxonomic framework comprising: extent of world knowledge, reproduction fidelity, and extent of presence metaphor. A principal objective of the taxonomy is to clarify terminology issues and to provide a framework for classifying research across different disciplines. We present an end-to-end system for augmented and virtual reality telepresence, called Holoportation. Our system demonstrates high-quality, real-time 3D reconstructions of an entire space, including people, furniture and objects, using a set of new depth cameras. These 3D models can also be transmitted in real-time to remote users. This allows users wearing virtual or augmented reality displays to see, hear and interact with remote participants in 3D, almost as if they were present in the same physical space. From an audio-visual perspective, communicating and interacting with remote users edges closer to face-to-face communication. This paper describes the Holoportation technical system in full, its key interactive capabilities, the application scenarios it enables, and an initial qualitative study of using this new communication medium. We describe a design approach, Tangible Augmented Reality, for developing face-to-face collaborative Augmented Reality (AR) interfaces. Tangible Augmented Reality combines Augmented Reality techniques with Tangible User Interface elements to create interfaces in which users can interact with spatial data as easily as real objects. Tangible AR interfaces remove the separation between the real and virtual worlds, and so enhance natural face-to-face communication We present several examples of Tangible AR interfaces and results from a user study that compares communication in a collaborative AR interface to more traditional approaches. We find that in a collaborative AR interface people use behaviours that are more similar to unmediated face-to-face collaboration than in a projection screen interface. Augmented reality (AR) allows to seamlessly insert virtual objects in an image sequence. In order to accomplish this goal, it is important that synthetic elements are rendered and aligned in the scene in an accurate and visually acceptable way. The solution of this problem can be related to a pose estimation or, equivalently, a camera localization process. This paper aims at presenting a brief but almost self-contented introduction to the most important approaches dedicated to vision-based camera localization along with a survey of several extension proposed in the recent years. For most of the presented approaches, we also provide links to code of short examples. This should allow readers to easily bridge the gap between theoretical aspects and practical implementations. In this paper, we propose a novel robocentric formulation of the visual-inertial navigation system (VINS) within a sliding-window filtering framework and design an efficient, lightweight, robocentric visual-inertial odometry (R-VIO) algorithm for consistent motion tracking even in challenging environments using only a monocular camera and a 6-axis IMU. The key idea is to deliberately reformulate the VINS with respect to a moving local frame, rather than a fixed global frame of reference as in the standard world-centric VINS, in order to obtain relative motion estimates of higher accuracy for updating global poses. As an immediate advantage of this robocentric formulation, the proposed R-VIO can start from an arbitrary pose, without the need to align the initial orientation with the global gravitational direction. More importantly, we analytically show that the linearized robocentric VINS does not undergo the observability mismatch issue as in the standard world-centric counterpart which was identified in the literature as the main cause of estimation inconsistency. Additionally, we investigate in-depth the special motions that degrade the performance in the world-centric formulation and show that such degenerate cases can be easily compensated in the proposed robocentric formulation, without resorting to additional sensors as in the world-centric formulation, thus leading to better robustness. The proposed R-VIO algorithm has been extensively tested through both Monte Carlo simulations and real-world experiments with different sensor platforms navigating in different environments, and shown to achieve better (or competitive at least) performance than the state-of-the-art VINS, in terms of consistency, accuracy and efficiency. Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI algorithms, (ii) properties of the worlds can be modified (e.g., material and reflectance), (iii) physical simulations can be performed, and (iv) algorithms can be learnt and evaluated. But creating realistic virtual worlds is not easy. The game industry, however, has spent a lot of effort creating 3D worlds, which a player can interact with. So researchers can build on these resources to create virtual worlds, provided we can access and modify the internal data structures of the games. To enable this we created an open-source plugin UnrealCV (Project website: http: unrealcv.github.io) for a popular game engine Unreal Engine 4 (UE4). We show two applications: (i) a proof of concept image dataset, and (ii) linking Caffe with the virtual world to test deep network algorithms. Videos and 3D models have traditionally existed in separate worlds and as distinct representations. Although texture maps for 3D models have been traditionally derived from multiple still images, real-time mapping of live videos as textures on 3D models has not been attempted. This paper presents a system for rendering multiple live videos in real-time over a 3D model as a novel and demonstrative application of the power of commodity graphics hardware. The system, metaphorically called the Video Flashlight system, "illuminates" a static 3D model with live video textures from static and moving cameras in the same way as a flashlight (torch) illuminates an environment. The Video Flashlight system is also an augmented reality solution for security and monitoring systems that deploy numerous cameras to monitor a large scale campus or an urban site. Current video monitoring systems are highly limited in providing global awareness since they typically display numerous camera videos on a grid of 2D displays. In contrast, the Video Flashlight system exploits the real-time rendering capabilities of current graphics hardware and renders live videos from various parts of an environment co-registered with the model. The user gets a global view of the model and is also able to visualize the dynamic videos simultaneously in the context of the model. In particular, the location of pixels and objects seen in the videos are precisely overlaid on the model while the user navigates through the model. The paper presents an overview of the system, details of the real-time rendering and demonstrates the efficacy of the augmented reality application. While the knowledge economy has reshaped the world, schools lag behind in producing appropriate learning for this social change. Science education needs to prepare students for a future world in which multiple representations are the norm and adults are required to “think like scientists.” Location-based augmented reality games offer an opportunity to create a “post-progressive” pedagogy in which students are not only immersed in authentic scientific inquiry, but also required to perform in adult scientific discourses. This cross-case comparison as a component of a design-based research study investigates three cases (roughly 28 students total) where an Augmented Reality curriculum, Mad City Mystery, was used to support learning in environmental science. We investigate whether augmented reality games on handhelds can be used to engage students in scientific thinking (particularly argumentation), how game structures affect students’ thinking, the impact of role playing on learning, and the role of the physical environment in shaping learning. We argue that such games hold potential for engaging students in meaningful scientific argumentation. Through game play, players are required to develop narrative accounts of scientific phenomena, a process that requires them to develop and argue scientific explanations. We argue that specific game features scaffold this thinking process, creating supports for student thinking non-existent in most inquiry-based learning environments. Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI algorithms, (ii) properties of the worlds can be modified (e.g., material and reflectance), (iii) physical simulations can be performed, and (iv) algorithms can be learnt and evaluated. But creating realistic virtual worlds is not easy. The game industry, however, has spent a lot of effort creating 3D worlds, which a player can interact with. So researchers can build on these resources to create virtual worlds, provided we can access and modify the internal data structures of the games. To enable this we created an open-source plugin UnrealCV (this http URL) for a popular game engine Unreal Engine 4 (UE4). We show two applications: (i) a proof of concept image dataset, and (ii) linking Caffe with the virtual world to test deep network algorithms. Augmented and virtual reality have the potential of being indistinguishable from the real world. Holographic displays, including head mounted units, support this vision by creating rich stereoscopic scenes, with objects that appear to float in thin air - often within arm's reach. However, one has but to reach out and grasp nothing but air to destroy the suspension of disbelief. Snake-charmer is an attempt to provide physical form to virtual objects by revisiting the concept of Robotic Graphics or Encountered-type Haptic interfaces with current commodity hardware. By means of a robotic arm, Snake-charmer brings physicality to a virtual scene and explores what it means to truly interact with an object. We go beyond texture and position simulation and explore what it means to have a physical presence inside a virtual scene. We demonstrate how to render surface characteristics beyond texture and position, including temperature; how to physically move objects; and how objects can physically interact with the user's hand. We analyze our implementation, present the performance characteristics, and provide guidance for the construction of future physical renderers. Novel approaches are needed to reduce the high rates of childhood obesity in the developed world. While multifactorial in cause, a major factor is an increasingly sedentary lifestyle of children. Our research shows that a mixed reality system that is of interest to children can be a powerful motivator of healthy activity. We designed and constructed a mixed reality system that allowed children to exercise, play with, and train a virtual pet using their own physical activity as input. The health, happiness, and intelligence of each virtual pet grew as its associated child owner exercised more, reached goals, and interacted with their pet. We report results of a research study involving 61 children from a local summer camp that shows a large increase in recorded and observed activity, alongside observational evidence that the virtual pet was responsible for that change. These results, and the ease at which the system integrated into the camp environment, demonstrate the practical potential to impact the exercise behaviors of children with mixed reality. We propose a new approach for 3D reconstruction of dynamic indoor and outdoor scenes in everyday environments, leveraging only cameras worn by a user. This approach allows 3D reconstruction of experiences at any location and virtual tours from anywhere. The key innovation of the proposed ego-centric reconstruction system is to capture the wearer's body pose and facial expression from near-body views, e.g. cameras on the user's glasses, and to capture the surrounding environment using outward-facing views. The main challenge of the ego-centric reconstruction, however, is the poor coverage of the near-body views – that is, the user's body and face are observed from vantage points that are convenient for wear but inconvenient for capture. To overcome these challenges, we propose a parametric-model-based approach to user motion estimation. This approach utilizes convolutional neural networks (CNNs) for near-view body pose estimation, and we introduce a CNN-based approach for facial expression estimation that combines audio and video. For each time-point during capture, the intermediate model-based reconstructions from these systems are used to re-target a high-fidelity pre-scanned model of the user. We demonstrate that the proposed self-sufficient, head-worn capture system is capable of reconstructing the wearer's movements and their surrounding environment in both indoor and outdoor situations without any additional views. As a proof of concept, we show how the resulting 3D-plus-time reconstruction can be immersively experienced within a virtual reality system (e.g., the HTC Vive). We expect that the size of the proposed egocentric capture-and-reconstruction system will eventually be reduced to fit within future AR glasses, and will be widely useful for immersive 3D telepresence, virtual tours, and general use-anywhere 3D content creation.