3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Abstract Three studies were conducted comparing speed of performance, error rates and user preference ratings for three selection devices. The devices tested were a touchscreen, a touchscreen with stabilization (stabilization software filters and smooths raw data from hardware), and a mouse. The task was the selection of rectangular targets 1, 4, 16 and 32 pixels per side (0·4 × 0·6, 1·7 × 2·2, 6·9 × 9·0, 13·8 × 17·9 mm respectively). Touchscreen users were able to point at single pixel targets, thereby countering widespread expectations of poor touchscreen resolution. The results show no difference in performance between the mouse and touchscreen for targets ranging from 32 to 4 pixels per side. In addition, stabilization significantly reduced the error rates for the touchscreen when selecting small targets. These results imply that touchscreens, when properly used, have attractive advantages in selecting targets as small as 4 pixels per size (approximately one-quarter of the size of a single character). A variant of Fitts' Law is proposed to predict touchscreen pointing times. Ideas for future research are also presented. <s> BIB001 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> This paper surveys the field of augmented reality AR, in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality. <s> BIB002 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Optical tracking systems allow three-dimensional input for virtual environment applications with high precision and without annoying cables. Spontaneous and intuitive interaction is possible through gestures. The authors present a finger tracker that allows gestural interaction and is simple, cheap, fast, robust against occlusion and accurate. It is based on a marked glove, a stereoscopic tracking system and a kinematic 3D model of the human finger. Within our augmented reality application scenario, the user is able to grab, translate, rotate, and release objects in an intuitive way. We demonstrate our tracking system in an augmented reality chess game, allowing a user to interact with virtual objects. <s> BIB003 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> This paper presents a technique for natural, fingertip-based interaction with virtual objects in Augmented Reality (AR) environments. We use image processing software and finger- and hand-based fiducial markers to track gestures from the user, stencil buffering to enable the user to see their fingers at all times, and fingertip-based haptic feedback devices to enable the user to feel virtual objects. Unlike previous AR interfaces, this approach allows users to interact with virtual content using natural hand gestures. The paper describes how these techniques were applied in an urban planning interface, and also presents preliminary informal usability results. <s> BIB004 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> We present a set of mobile augmented reality interaction techniques for authoring situated media: multimedia and hypermedia that are embedded within the physical environment. Our techniques are designed for use with a tracked hand-held tablet display with an attached camera, and rely on "freezing" the frame for later editing. <s> BIB005 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Retrieving the stylus of a pen-based device takes time and requires a second hand. Especially for short intermittent interactions many users therefore choose to use their bare fingers. Although convenient, this increases targeting times and error rates. We argue that the main reasons are the occlusion of the target by the user's finger and ambiguity about which part of the finger defines the selection point. We propose a pointing technique we call Shift that is designed to address these issues. When the user touches the screen, Shift creates a callout showing a copy of the occluded screen area and places it in a non-occluded location. The callout also shows a pointer representing the selection point of the finger. Using this visual feedback, users guide the pointer into the target by moving their finger on the screen surface and commit the target acquisition by lifting the finger. Unlike existing techniques, Shift is only invoked when necessary--over large targets no callout is created and users enjoy the full performance of an unaltered touch screen. We report the results of a user study showing that with Shift participants can select small targets with much lower error rates than an unaided touch screen and that Shift is faster than Offset Cursor for larger targets. <s> BIB006 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> In this paper we present an evaluation of several different techniques for virtual object positioning and rotation on a mobile phone. We compare gesture input captured by the phone's front camera, to tangible input, keypad interaction and phone tilting in increasingly complex positioning and rotation tasks in an AR context. Usability experiments found that tangible input techniques are best for translation tasks, while keypad input is best for rotation tasks. Implications for the design of mobile phone 3D interfaces are presented as well as directions for future research. <s> BIB007 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> This paper describes, Napkin Sketch, a 3D sketching interface which attempts to support sketch-based artistic expression in 3D, mimicking some of the qualities of conventional sketching media and tools both in terms of physical properties and interaction experience. A portable tablet PC is used as the sketching platform, and handheld mixed reality techniques are employed to allow 3D sketches to be created on top of a physical napkin. Intuitive manipulation and navigation within the 3D design space is achieved by visually tracking the tablet PC with a camera and mixed reality markers. For artistic expression using sketch input, we improve upon the projective 3D sketching approach with a one stroke sketch plane definition technique. This coupled with the hardware setup produces a natural and fluid sketching experience. <s> BIB008 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> We present a one-handed approach for augmented reality and interaction on mobile devices. The proposed application considers common situations with mobile devices such as when a user's hand holds a mobile device while the other hand is free. It also supports natural augmented reality environment such as when a user interacts with augmented reality contents anytime and anywhere without special equipment such as visual markers or tags. In our approach, a virtual object is augmented on the palm of a user's free hand, as if the virtual object is just sitting on the palm, using a palm pose estimation method. The augmented virtual object reacts (e.g. moving or animation) to motions of the hand such as opening or closing the hand based on fingertip tracking. Moreover, it provides tactile interactions with the virtual object by wearing a tactile glove with vibration sensors. This paper describes how to implement the augmented reality application, and preliminary results show its potential as a new approach to mobile augmented reality interaction. <s> BIB009 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Mobile computing devices are getting popular as a platform for augmented reality (AR) application, and efficient interaction methods for mobile AR environments are considered necessary. Recently, touch interfaces are getting popular and drawing attention as a future standard interface on mobile computing devices. However, accurate touch interactions are not that easy in mobile AR environments where users tend to move and viewpoints easily get shaky. In this paper, the authors suggest a new interaction method for handheld mobile AR environments, named 'Freeze-Set-Go'. The proposed interaction method lets users to 'freeze' the real world view tentatively, and continue to manipulate virtual entities within the AR scene. According to the user experiment, the proposed method turns out to be helping users to interact with mobile AR environments using touch interfaces in a more accurate and comfortable way. <s> BIB010 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> In this paper, we explore how to add pointing input capabilities to very small screen devices. On first sight, touchscreens seem to allow for particular compactness, because they integrate input and screen into the same physical space. The opposite is true, however, because the user's fingers occlude contents and prevent precision. We argue that the key to touch-enabling very small devices is to use touch on the device backside. In order to study this, we have created a 2.4" prototype device; we simulate screens smaller than that by masking the screen. We present a user study in which participants completed a pointing task successfully across display sizes when using a back-of device interface. The touchscreen-based control condition (enhanced with the shift technique), in contrast, failed for screen diagonals below 1 inch. We present four form factor concepts based on back-of-device interaction and provide design guidelines extracted from a second user study. <s> BIB011 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> In this paper we present an Augmented Reality (AR) authoring tool for smartphones which facilitates intuitive interactions that manipulate the augmented virtual objects in real-time. A novel 3D interaction method using multi-touch interface and camera pose is proposed for intuitive authoring. With the gestures of two fingers on the touch screen, the user can adjust 3 DOF translation and 3 DOF rotation to a selected virtual object. The capabilities of the authoring tool are demonstrated on a smartphone. <s> BIB012 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Interaction techniques for handheld mobile Augmented Reality (AR) often focus on device-centric methods based around touch input. However, users may not be able to easily interact with virtual objects in mobile AR scenes if they are holding the handheld device with one hand and touching the screen with the other, while at the same time trying to maintain visual tracking of an AR marker. In this paper we explore novel interaction methods for handheld mobile AR that overcomes this problem. We investigate two different approaches; (1) freeze view touch and (2) finger gesture based interaction. We describe how each method is implemented and present findings from a user experiment comparing virtual object manipulation with these techniques to more traditional touch methods. <s> BIB013 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> B. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Nowadays, handheld devices are capable of displaying augmented environments in which virtual content overlaps reality. To interact with these environments it is necessary to use a manipulation technique. The objective of a manipulation technique is to define how the input data modify the properties of the virtual objects. Current devices have multi-touch screens that can serve as input. Additionally, the position and rotation of the device can also be used as input creating both an opportunity and a design challenge. In this paper we compared three manipulation techniques which namely employ multi-touch, device position and a combination of both. A user evaluation on a docking task revealed that combining multi-touch and device movement yields the best task completion time and efficiency. Nevertheless, using only the device movement and orientation is more intuitive and performs worse only in large rotations. <s> BIB014
Touch-based interaction is one of the popular interaction categories based on the suitability to be implemented in AR. In the early stage, the potential of using touch-based interaction was studied by Xin et al. BIB008 , Lee BIB010 , and Guven et al. BIB005 but not focusing on 3D object manipulation. Existing touch-based interactions using single fingers and two fingers were then implemented in handheld mobile AR environments studied earlier by Jung et al. BIB012 . The hybrid technique was then introduced afterward by Mossel et al.  and Marzo et al. BIB014 . They conducted similar works by introducing two types of 3D object manipulation techniques in AR environments. Both researches suggested to use touch-based interaction for 3D object rotation instead of translation to reduce the number of fingers used in 3D object manipulation, and mode switching method is required to separate 3D object translation and rotation (shown in Fig. 4 ), while the wellknown Arcball concept had been studied and improved to perform 3D object rotation with one finger. However, if the user wants to perform 3D object manipulation integrally, at least two fingers are required. The existing issues include the difficulty to control the movements of each contact point mapped with each finger beside the occlusion problem. After the wide discussion about the touch-based interaction, the important researches had been compressed in a systematic summarized table (Table 1) by keywords either it becomes the basement to AR research works, about 3D object manipulation, or some relevant issues based on touch-based techniques. As stated in VR field that also is the issue in AR, the occlusion problem, which may cause the fat finger effect, the main issue faced when using the touch-based interaction technique, which is already widely discussed above. Several suggestions and solutions have been introduced based on this basis. For example, the Shift technique proposed by Vogel and Baudisch BIB006 improved the well-known offset pointer technique proposed at the early stage BIB001 focusing on small screen touch-based interfaces. This technique reveals the occluded screen content in a callout displayed near the action touch by the user's finger. It provides a potential solution to make sure the user can still fine-tune and see the target content on the screen although it still consists of many shortcomings such as the portion of on-screen content overlaid by the callout cannot be seen by the user, including the part of the content that is occluded by the finger below the touch point. Furthermore, Paudisch and Chu BIB011 suggested to use the back-side touch to solve the occlusion problem; however, this technique still needs more investigation since the new design of the current handheld mobile device is required to enable such technique. Due to the difficulties in interacting with virtual objects in handheld mobile AR scenes when the user needs to hold the handheld mobile device with one hand and touch the screen with the other, freeze view BIB010 , BIB013 becomes a potential solution. According to BIB005 , the freeze-frame technique can be used to allow the user to snapshot the environment to work on it and map the results back to the physical world. Their works had been extended by Bai et al. BIB013 to ease 3D object manipulation in AR interfaces while users can freeze the view to stop the AR marker tracking process so that users can focus on the 3D object manipulation task. Although the freeze view touch performed fastest and most accurately, it mitigated the user's AR experiences since the static view reduces real-time engagement and so was not attractive enough for them BIB013 . Furthermore, these kinds of interactions without real-time engagement have already violated the concept of AR universally defined and accepted by researchers in this field BIB002 -  . Due to the naturality and the ability to provide intuitive and seamless interaction experience, mid-air gesturesbased interaction then had been extended from the previous works that mostly done in stereoscopic interactive tabletop or wall towards the AR environment. The earlier research related to mid-air gestures-based interaction in AR environment done by Dorfmüller-Ulhaas and Schmalstieg BIB003 , Buchmann et al. BIB004 , and Radkowski and Stritzke  . In BIB003 , the user's own hand had been used to play AR chess. To allow relatively unrestrained environment, a marked glove was used for real-time separation of the finger from the background to provide robust interaction technique to select, translate and rotate the chess piece. While in BIB004 , FingARtips had been introduced by marking the user's fingertips as fiducial markers to track the user's finger gestures to manipulate a virtual object in the augmented environment. The prototype had been developed to enable FingARtips for urban planning workplace showed in Fig. 6a . Four different types of gesture interaction were implemented: grabbing, pointing, navigating and command gestures. Followed by  , hand pose had been used to perform 3D object manipulation, the user moved his/her hand after grasping a moving direction to translate the 3D object and open his/her hand to release the object (showed in Fig. 6b) . The mobility and popularity of handheld mobile devices especially smartphones lead the research trend from tabletops (large displays) to handheld mobile AR (small displays). Thus, in the last five years, researches relating to large display-based AR had reduced and switched to handheld FIGURE 6. Using mid-air gestures-based interaction in an augmented environment BIB004 ,  . (a) Using FingARtips for urban planning workplace BIB004 . (b) 3D translation  . mobile AR. At the early stage, Henrysson et al. BIB007 did a comparison between several interaction techniques for 3D object manipulation in handheld mobile AR (cell phone). One of the techniques compared was mid-air gestures-based interaction that uses the user's own fingers. In their study, cell phones with built-in camera were used as primary handheld mobile devices that are limited to low resolution video frame, whereas a simple frame-differencing tracking method was used to enable 2D manipulation while 3D object manipulation could be activated through switching modality to access the third rotation axis but was not available for cell phones at that time due to the slow interaction process. Furthermore, Seo et al. BIB009 proposed to use the user's palm for registering 3D objects while the palm was treated as a marker, and through the palm pose estimation process, the 3D object could be augmented precisely on the user's palm. 3D object manipulation could be performed when users move or rotate his/her palm while additional vibration sensors were added to enhance the tactile experience. The increasing number of researches relating to 3D object manipulation in handheld mobile AR then induced Bai et al. BIB013 to do the comparison between touch-based interaction and mid-air gestures-based interaction. In their study, the most prominent fingertip would be identified and marked by a small white circle in the ''Translation'' and ''Rotation'' modes, while a second fingertip would be recognized and marked as a small red circle in the ''Scaling'' mode. Results showed that the midair technique implemented in their study appeared to have less usability compared with touch-based technique due to the immatureness of their gesture recognition software.