Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> The problem we are addressing in Alvey Project MMI149 is that of using computer vision to understand the unconstrained 3D world, in which the viewed scenes will in general contain too wide a diversity of objects for topdown recognition techniques to work. For example, we desire to obtain an understanding of natural scenes, containing roads, buildings, trees, bushes, etc., as typified by the two frames from a sequence illustrated in Figure 1. The solution to this problem that we are pursuing is to use a computer vision system based upon motion analysis of a monocular image sequence from a mobile camera. By extraction and tracking of image features, representations of the 3D analogues of these features can be constructed. <s> BIB001 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> This paper presents cognitive studies and analyses relating to how augmented reality (AR) interacts with human abilities in order to benefit manufacturing and maintenance tasks. A specific set of applications is described in detail, as well as a prototype system and the software library that it is built upon. An integrated view of information flow to support AR is also presented, along with a proposal for an AR media language (ARML) that could provide interoperability between various AR implementations. <s> BIB002 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> This paper describes a new approach to low level image processing; in particular, edge and corner detection and structure preserving noise reduction. ::: ::: Non-linear filtering is used to define which parts of the image are closely related to each individual pixel; each pixel has associated with it a local image region which is of similar brightness to that pixel. The new feature detectors are based on the minimization of this local image region, and the noise reduction method uses this region as the smoothing neighbourhood. The resulting methods are accurate, noise resistant and fast. ::: ::: Details of the new feature detectors and of the new noise reduction method are described, along with test results. <s> BIB003 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> In this paper, we present a novel scale- and rotation-invariant interest point detector and descriptor, coined SURF (Speeded Up Robust Features). It approximates or even outperforms previously proposed schemes with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster. ::: ::: This is achieved by relying on integral images for image convolutions; by building on the strengths of the leading existing detectors and descriptors (in casu, using a Hessian matrix-based measure for the detector, and a distribution-based descriptor); and by simplifying these methods to the essential. This leads to a combination of novel detection, description, and matching steps. The paper presents experimental results on a standard evaluation set, as well as on imagery obtained in the context of a real-life object recognition application. Both show SURF's strong performance. <s> BIB004 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> While feature point recognition is a key component of modern approaches to object detection, existing approaches require computationally expensive patch preprocessing to handle perspective distortion. In this paper, we show that formulating the problem in a Naive Bayesian classification framework makes such preprocessing unnecessary and produces an algorithm that is simple, efficient, and robust. Furthermore, it scales well to handle large number of classes. To recognize the patches surrounding keypoints, our classifier uses hundreds of simple binary features and models class posterior probabilities. We make the problem computationally tractable by assuming independence between arbitrary sets of features. Even though this is not strictly true, we demonstrate that our classifier nevertheless performs remarkably well on image datasets containing very significant perspective changes. <s> BIB005 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> Augmented Reality (AR) is a technique for overlaying virtual objects onto the real world. AR has recently been applied to many kinds of entertainment applications by using visionbased tracking technique, such as [Klein & Drummond, 2004, Henrysson et al., 2005, Haller et al. 2005, Schmalstieg & Wagner, 2007, Looser et al. 2007]. AR can provide users with immersive feeling by allowing the interaction between the real and virtual world. In the AR entertainment applications, virtual objects (world) generated with Computer Graphics are overlaid onto the real world. This means that the real 3D world is captured by a camera, and then the virtual objects are superimposed onto the captured images. By seeing the real world through some sort of displays, the users find that the virtual world is mixed with the real world. In such AR applications, the users carry a camera and move around the real world in order to change their view points. Therefore the pose and the position of the moving user’s camera should be obtained so that the virtual objects can be overlaid at correct position in the real world according to the camera motion. Such camera tracking should also be performed in real-time for interactive operations of the AR applications. Vision-based camera tracking for AR is one of the popular research areas because the visionbased method does not require any special device except cameras, in contrast with sensorbased approaches. And also, marker-based approach is a quite easy solution to make the vision-based tracking robust and running in real-time. This chapter focuses on marker-based approach. Especially, “AR-Toolkit” [H. Kato & M. Billinghurst, 1999] is a very popular tool for implementing simple on-line AR applications. ARToolkit is a kind of planar square marker for the camera tracking and estimates the camera position and pose with respect to the marker. By using the camera position and pose, virtual objects are overlaid onto the images as if the objects exist in the real world where the marker is placed. Since the user only has to place the marker, this kind of markerbase registration is very easy to implement AR systems. If only one marker is utilized, however, the camera's movable area is limited so that the camera (user) can see the marker. Moreover, when the marker cannot be recognized properly because of a change in its visibility, the registration of the virtual objects is getting unstable. In order to solve such problems, using multiple markers is a popular way. When multiple markers are utilized in a vision-based method, it is necessary to know the geometrical arrangement information of the marker such as their position and pose in advance. For example, the method in [Umlauf et al., 2002] requires the position and pose of a square marker. The method in [Kato et al., 2000] also needs the position of a point marker in advance. In [Genc et al., 2002], they proposed two-step approach; learning process and O pe n A cc es s D at ab as e w w w .ite ch on lin e. co m <s> BIB006 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> Although Augmented Reality technology was first developed over forty years ago, there has been little survey work giving an overview of recent research in the field. This paper reviews the ten-year development of the work presented at the ISMAR conference and its predecessors with a particular focus on tracking, interaction and display research. It provides a roadmap for future augmented reality research which will be of great value to this relatively young field, and also for helping researchers decide which topics should be explored when they are beginning their own studies in the area. <s> BIB007 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> Research studies in the application of Augmented Reality (AR) in the Architecture, Engineering, and Construction (AEC) industry have suggested its feasibility. However, realization of the use of AR in AEC requires not only demonstration of feasibility but also validation of its suitability. This paper comprehensively identifies AR application areas in industrial construction based on suitability of AR technologies. In order to successfully explore suitability of AR, this paper assesses work tasks from the viewpoint of human factors regarding visual information requirements to find rationale for the benefits of AR in work tasks. Based on the assessment of work tasks, this paper presents a comprehensive map that identifies AR application areas in industrial construction. The comprehensive map reveals that eight work tasks (layout, excavation, positioning, inspection, coordination, supervision, commenting, and strategizing) out of 17 classified work tasks may potentially benefit from AR support. <s> BIB008 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> We present a robust elastic and partial matching metric for face recognition. To handle challenges such as pose, facial expression and partial occlusion, we enable both elastic and partial matching by computing a part based face representation. In which N local image descriptors are extracted from densely sampled overlapping image patches. We then define a distance metric where each descriptor in one face is matched against its spatial neighborhood in the other face and the minimal distance is recorded. For implicit partial matching, the list of all minimal distances are sorted in ascending order and the distance at the αN-th position is picked up as the final distance. The parameter 0 ≤ α ≤ 1 controls how much occlusion, facial expression changes, or pixel degradations we would allow. The optimal parameter values of this new distance metric are extensively studied and identified with real-life photo collections. We also reveal that filtering the face image by a simple difference of Gaussian brings significant robustness to lighting variations and beats the more utilized self-quotient image. Extensive evaluations on face recognition benchmarks show that our method is leading or is competitive in performance when compared to state-of-the-art. <s> BIB009 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> We propose to use binary strings as an efficient feature point descriptor, which we call BRIEF. We show that it is highly discriminative even when using relatively few bits and can be computed using simple intensity difference tests. Furthermore, the descriptor similarity can be evaluated using the Hamming distance, which is very efficient to compute, instead of the L2 norm as is usually done. ::: ::: As a result, BRIEF is very fast both to build and to match. We compare it against SURF and U-SURF on standard benchmarks and show that it yields a similar or better recognition performance, while running in a fraction of the time required by either. <s> BIB010 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> In this paper, we present a vision-based framework to manipulate the augmented reality (AR) objects robustly in a marker-less AR system. It is known that one of the promising ways to develop a marker-less AR system is using human’s body such as hand or face for replacing fiducial markers. This paper presents a real-time method to manipulate the AR objects dynamically in a developed 3D hand coordinate system. To build the marker-less system, we utilize YCbCr skin-color model for hand shape detection and curvature-based fingertip detection from an input video images. Using the detected fingertips the camera pose are estimated to register virtual objects on the hand coordinate system. In order to manipulate the movements of the virtual objects rendered on the marker-less AR system dynamically, a vision-based hand control interface, which utilizes pattern recognition for the hand command initiation, is developed. From the experiments we can prove that the proposed hand mouse can manipulate the objects in a feasible fashion. <s> BIB011 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> This paper surveys the current state-of-the-art of technology, systems and applications in Augmented Reality. It describes work performed by many different research groups, the purpose behind each new Augmented Reality system, and the difficulties and problems encountered when building some Augmented Reality applications. It surveys mobile augmented reality systems challenges and requirements for successful mobile systems. This paper summarizes the current applications of Augmented Reality and speculates on future applications and where current research will lead Augmented Reality's development. Challenges augmented reality is facing in each of these applications to go from the laboratories to the industry, as well as the future challenges we can forecast are also discussed in this paper. Section 1 gives an introduction to what Augmented Reality is and the motivations for developing this technology. Section 2 discusses Augmented Reality Technologies with computer vision methods, AR devices, interfaces and systems, and visualization tools. The mobile and wireless systems for Augmented Reality are discussed in Section 3. Four classes of current applications that have been explored are described in Section 4. These applications were chosen as they are the most famous type of applications encountered when researching AR apps. The future of augmented reality and the challenges they will be facing are discussed in Section 5. <s> BIB012 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> The Scale-Invariant Feature Transform (or SIFT) algorithm is a highly robust method to extract and consequently match distinctive invariant features from images. These features can then be used to reliably match objects in diering images. The algorithm was rst proposed by Lowe [12] and further developed to increase performance resulting in the classic paper [13] that served as foundation for SIFT which has played an important role in robotic and machine vision in the past decade. <s> BIB013 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> Feature matching is at the base of many computer vision problems, such as object recognition or structure from motion. Current methods rely on costly descriptors for detection and matching. In this paper, we propose a very fast binary descriptor based on BRIEF, called ORB, which is rotation invariant and resistant to noise. We demonstrate through experiments how ORB is at two orders of magnitude faster than SIFT, while performing as well in many situations. The efficiency is tested on several real-world applications, including object detection and patch-tracking on a smart phone. <s> BIB014 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> We present a novel real-time monocular SLAM system which can robustly work in dynamic environments. Different to the traditional methods, our system allows parts of the scene to be dynamic or the whole scene to gradually change. The key contribution is that we propose a novel online keyframe representation and updating method to adaptively model the dynamic environments, where the appearance or structure changes can be effectively detected and handled. We reliably detect the changed features by projecting them from the keyframes to current frame for appearance and structure comparison. The appearance change due to occlusions also can be reliably detected and handled. The keyframes with large changed areas will be replaced by newly selected frames. In addition, we propose a novel prior-based adaptive RANSAC algorithm (PARSAC) to efficiently remove outliers even when the inlier ratio is rather low, so that the camera pose can be reliably estimated even in very challenging situations. Experimental results demonstrate that the proposed system can robustly work in dynamic environments and outperforms the state-of-the-art SLAM systems (e.g. PTAM). <s> BIB015 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> This paper presents a method to address the issue of augmenting a markerless 3D object with a complex shape. It relies on a model-based tracker which takes advantage of GPU acceleration and 3D rendering in order to handle the complete 3D model, whose sharp edges are efficiently extracted. In the pose estimation step, we propose to robustly combine geometrical and color edge-based features in the nonlinear minimization process, and to integrate multiple-hypotheses in the geometrical edge-based registration phase. Our tracking method shows promising results for augmented reality applications, with a Kinect-based reconstructed 3D model. <s> BIB016 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> To maintain a reliable registration of the virtual world with the real world, augmented reality (AR) applications require highly accurate, low-latency tracking of the device. In this paper, we propose a novel method for performing this fast 6-DOF head pose tracking using a cluster of rolling shutter cameras. The key idea is that a rolling shutter camera works by capturing the rows of an image in rapid succession, essentially acting as a high-frequency 1D image sensor. By integrating multiple rolling shutter cameras on the AR device, our tracker is able to perform 6-DOF markerless tracking in a static indoor environment with minimal latency. Compared to state-of-the-art tracking systems, this tracking approach performs at significantly higher frequency, and it works in generalized environments. To demonstrate the feasibility of our system, we present thorough evaluations on synthetically generated data with tracking frequencies reaching 56.7 kHz. We further validate the method's accuracy on real-world images collected from a prototype of our tracking system against ground truth data using standard commodity GoPro cameras capturing at 120 Hz frame rate. <s> BIB017 </s> Augmented-Reality Survey: from Concept to Application <s> Vision-based approaches <s> Augmented reality (AR) allows to seamlessly insert virtual objects in an image sequence. In order to accomplish this goal, it is important that synthetic elements are rendered and aligned in the scene in an accurate and visually acceptable way. The solution of this problem can be related to a pose estimation or, equivalently, a camera localization process. This paper aims at presenting a brief but almost self-contented introduction to the most important approaches dedicated to vision-based camera localization along with a survey of several extension proposed in the recent years. For most of the presented approaches, we also provide links to code of short examples. This should allow readers to easily bridge the gap between theoretical aspects and practical implementations. <s> BIB018
This type of approach determines the camera position through the use of optical-sensor data. These optical sensors can be divided into the following three categories: infrared sensor, visible-light sensor, and 3D-structure sensor. Also, vision-based tracking is a popular method because a camera is the only requisite special device BIB006 . For the vision-based tracking techniques, image-analysis methods are used to estimate the camera pose relative to the real-world objects BIB007 BIB002 . Fiducial markers were employed for the early vision-based approaches BIB008 , but most of the recent approaches are based on the markerless approach BIB017 . In the first processing step of the markerless tracking, the feature points and/or edges are detected to interpret the camera images. The popular feature-detection methods include Harris BIB001 , SUSAN BIB003 , FAST  , and DoG (Difference of Gaussian) BIB009 . The extracted features or edges must be represented as a multi-dimensional vector value; that is, in terms of the use of these vectors, the feature descriptors are required to represent the extracted features and for the estimation of the camera pose. The representative feature descriptors include SIFT BIB013 , SURF BIB004 , BRIEF BIB010 , ORB BIB014 , and FERNs BIB005 . The inter-frame features are matched by tree-based searching, hashing, or the full-search method. Likewise, for the feature-based methods, the matching relation between the 2D-image features is first discovered, followed by an estimation of the features' 3D world-frame coordinates BIB011 . For the estimation of the camera pose, the model-based methods can be applied. The model-based methods use models of the tracked-object features or templates BIB007 . If a matching relation is defined between the 3D world frame and the 2D image, it is possible to estimate the camera pose by projecting the 3D features into the observed 2D image plane, and by minimizing the distance between these 3D features and the corresponding 2D-feature coordinates BIB012 . SLAM is a principal technique that is used to progressively construct a map within an unprepared environment, while its current locaton is tracked based on the extracted scene features BIB015 . This approach has been applied for the robust tracking of the AR in a small space. The updating of the 3D structure in correspondence with the dynamic objects, however, is not considered for the SLAM system, so the target working scene must be kept stationary. Petit et al. BIB016 presented a tracking method for which the classical geometrical and color-edge-based features are integrated in the pose-estimation phase. See BIB018 for a comprehensive reference regarding the AR camera-pose estimation.