3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> FIGURE 7. Processes involved in <s> Balloon selection is a 3D interaction technique that is modeled after the real world metaphor of manipulating a helium balloon attached to a string. Balloon selection allows for precise 3D selection in the volume above a tabletop surface by using multiple fingers on a multi-touch-sensitive surface. The 3DOF selection tasks is decomposed in part into a 2DOF positioning task performed by one finger on the tabletop in an absolute 2D Cartesian coordinate system and a 1DOF positioning task performed by another finger on the tabletop in a relative 2D polar coordinate system. We have evaluated balloon selection in a formal user study that compared it to two well-known interaction techniques for selecting a static 3D target: a 3DOF tracked wand and keyboard cursor keys. We found that balloon selection was significantly faster than using cursor keys and had a significantly lower error rate than the wand. The lower error rate appeared to result from the user's hands being supported by the tabletop surface, resulting in significantly reduced hand tremor and arm fatigue. <s> BIB001 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> FIGURE 7. Processes involved in <s> Over the past few years, Augmented Reality has become widely popular in the form of smart phone applications, however most smart phone-based AR applications are limited in user interaction and do not support gesture-based direct manipulation of the augmented scene. In this paper, we introduce a new AR interaction methodology, employing users' hands and fingers to interact with the virtual (and possibly physical) objects that appear on the mobile phone screen. The goal of this project was to support different types of interaction (selection, transformation, and fine-grain control of an input value) while keeping the methodology for hand detection as simple as possible to maintain good performance on smart phones. We evaluated our methods in user studies, collecting task performance data and user impressions about this direct way of interacting with augmented scenes through mobile phones. <s> BIB002 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> FIGURE 7. Processes involved in <s> In this paper, we present a novel gesture-based interaction method for handheld Augmented Reality (AR) implemented on a tablet with an RGB-Depth camera attached. Compared with conventional device-centric interaction methods like keypad, stylus, or touchscreen input, natural gesture-based interfaces offer a more intuitive experience for AR applications. Combining with depth information, gesture interfaces can extend handheld AR interaction into full 3D space. In our system we retrieve the 3D hand skeleton from color and depth frames, mapping the results to corresponding manipulations of virtual objects in the AR scene. Our method allows users to control virtual objects in 3D space using their bare hands and perform operations such as translation, rotation, and zooming. <s> BIB003 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> FIGURE 7. Processes involved in <s> Stereoscopic tabletops offer unique visualization capabilities, enabling users to perceive virtual objects as if they were lying above the surface. While allowing virtual objects to coexist with user actions in the physical world, interaction with these virtual objects above the surface presents interesting challenges. In this paper, we aim to understand which approaches to 3D virtual object manipulations are suited to this scenario. To this end, we implemented five different techniques based on the literature. Four are mid-air techniques, while the remainder relies on multi-touch gestures, which act as a baseline. Our setup combines affordable non-intrusive tracking technologies with a multi-touch stereo tabletop, providing head and hands tracking, to improve both depth perception and seamless interactions above the table. We conducted a user evaluation to find out which technique appealed most to participants. Results suggest that mid-air interactions, combining direct manipulation with six degrees of freedom for the dominant hand, are both more satisfying and efficient than the alternatives tested. <s> BIB004 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> FIGURE 7. Processes involved in <s> In this paper, we present a prototype for exploring natural gesture interaction with Handheld Augmented Reality (HAR) applications, using visual tracking based AR and freehand gesture based interaction detected by a depth camera. We evaluated this prototype in a user study comparing 3D gesture input methods with traditional touch-based techniques, using canonical manipulation tasks that are common in AR scenarios. We collected task performance data and user feedback via a usability questionnaire. The 3D gesture input methods were found to be slower, but the majority of the participants preferred them and gave them higher usability ratings. Being intuitive and natural was the most common feedback about the 3D freehand interface. We discuss implications of this research and directions for further work. <s> BIB005 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> FIGURE 7. Processes involved in <s> Interaction for Handheld Augmented Reality (HAR) is a challenging research topic because of the small screen display and limited input options. Although 2D touch screen input is widely used, 3D gesture interaction is a suggested alternative input method. Recent 3D gesture interaction research mainly focuses on using RGB-Depth cameras to detect the spatial position and pose of fingers, using this data for virtual object manipulations in the AR scene. In this paper we review previous 3D gesture research on handheld interaction metaphors for HAR. We present their novelties as well as limitations, and discuss future research directions of 3D gesture interaction for HAR. Our results indicate that 3D gesture input on HAR is a potential interaction method for assisting a user in many tasks such as in education, urban simulation and 3D games. <s> BIB006 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> FIGURE 7. Processes involved in <s> In this paper we present a novel framework for simultaneous detection of click action and estimation of occluded fingertip positions from egocentric viewed single-depth image sequences. For the detection and estimation, a novel probabilistic inference based on knowledge priors of clicking motion and clicked position is presented. Based on the detection and estimation results, we were able to achieve a fine resolution level of a bare hand-based interaction with virtual objects in egocentric viewpoint. Our contributions include: (i) a rotation and translation invariant finger clicking action and position estimation using the combination of 2D image-based fingertip detection with 3D hand posture estimation in egocentric viewpoint. (ii) a novel spatio-temporal random forest, which performs the detection and estimation efficiently in a single framework. We also present (iii) a selection process utilizing the proposed clicking action detection and position estimation in an arm reachable AR/VR space, which does not require any additional device. Experimental results show that the proposed method delivers promising performance under frequent self-occlusions in the process of selecting objects in AR/VR space whilst wearing an egocentric-depth camera-attached HMD. <s> BIB007 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> FIGURE 7. Processes involved in <s> Mobile augmented reality (AR) has been widely used in smart and mobile device-based applications such as entertainment, games, visual experience, and information visualization. However, most of the mobile AR applications have limitations in natural user interaction and do not fully support the direct manipulation of 3D AR objects. This paper proposes a new method for naturally and directly manipulating 3D AR objects through touch and hand gesture-based interactions in handheld devices. The touch gesture is used for the AR object selection and the natural hand gesture is used for the direct and interactive manipulation of the selected objects. Thus, the hybrid interaction makes the user more accurately interact with and manipulate AR objects in the real 3D space, not in the 2D space. In particular, natural hand gestures are detected by the Leap Motion sensor attached to the front or back of mobile devices. Thus the user can easily interacts with 3D AR objects for 3D transformation to enhance usability and usefulness. In this research, comprehensive comparative analyses were performed among the proposed approach and the widely used screen touch-based approach and vision-based approach in terms of quantitative and qualitative aspects. Quantitative analysis was conducted by measuring task completion time and failure rate to perform given tasks such as 3D object matching and grasp-hang-release operation. Both tasks require simultaneous 3D translation and 3D rotation. In addition, we have compared the gesture performance depending on whether the gesture sensor is located in the front or the back of the mobile device. Furthermore, to support other complex operations, an assembly task has also been evaluated. The assembly task consists of a sequence of combining parts into a sub-assembly. Qualitative analysis was performed through enquiring questionnaire after the experiment that examines factors such as ease-of-use, ease-of-natural interaction, etc. Both analyses showed that the proposed approach can provide more natural and intuitive interaction and manipulation of mobile AR objects. Several implementation results will also be given to show the advantage and effectiveness of the proposed approach. <s> BIB008
Their works and then were followed by Chun and Höllerer BIB002 that did a user study to measure their interaction technique towards 3D translation and scaling tasks (Fig. 7a) . Their technique can successfully address the problem of potential camera motion during gesture detection by using projective texture mapping and background subtraction in maintaining a good frame rate. Moreover, their gesture recognition can avoid self-occlusion and can detect finger gestures accurately. Extended from BIB002 , Bai et al. BIB003 applied free-hand interaction using an RGB-depth camera for 3D object translation, rotation, and zooming. By mapping the fingertips to project points and polygons (Fig. 7b-i) , users can perform 6DOF 3D object translation and rotation (Fig. 7b-ii) although their technique still suffered self-occlusion and accuracy limitations. Furthermore, pinch gestures used in BIB001 and BIB004 were studied and evaluated in handheld mobile AR environments by Bai et al. BIB005 to investigate the usefulness of mid-air gestures-based interaction in 3D object manipulation tasks. Convincing results gained in their study encouraged midair gestures-based interaction to be studied in more details to become an intuitive interaction solution in AR environments. Afterward, Kim and Lee BIB008  proposed to align the user view in allowing the user to manipulate 3D object more accurate using the mid-air gestures-based technique. However, additional depth camera and lens to recognize the user's face were required in order to adjust the view. The prototype proposed still immature and additional evaluation was required to measure its performance. Since AR had been claimed can provides intuitive interaction experiences, mid-air gestures-based interaction technique which is the intuitive interaction way had been widely explored in AR field including handheld mobile AR interface discussed above and so Yusof et al. BIB006 produced a short review about mid-air gestures-based interaction technique in handheld mobile AR due to the active responses of many researchers towards this interaction. They summarized some of the related studies and discussed the existing limitations of the mid-air gestures-based interaction such as the quality of depth sensors nowadays directly affect the recognition of the hand or finger postures, self-occlusion and lack accuracy still the main issues to be cured although there have some researches that provide potential solutions towards these issues such as 3D finger CAPE that proposed by Jang et al. BIB007 against occlusion estimated the 3D finger joint and 3D fingertip locations to form fingertip detector based on depth images. Their framework had been evaluated focusing on occlusion-invariant fingertip position estimation and the results showed that 3D finger CAPE can enable 3D object manipulation intuitively within the AR environment. However, the utilized hand tracking method sometimes fails to track when most of the fingers are completely occluded and it is expensive due to the calculation of different postures based on the occluded data and still limited towards the different combinations of the finger actions. Due to the complicated computational tracking information (in-depth data required) and the essential of additional depth devices (such as in-depth camera) may exhaust the battery capacity of a handheld mobile device  , BIB005 also the inaccuracy of the finger tracking (caused by occlusion) prolong the interaction period and limit the mid-air gesturesbased 3D object manipulation techniques to be used within handheld mobile AR. After the wide discussion about the mid-air gestures-based interaction, the important researches had been compressed in a systematic summarized table (Table 2 ) by keywords either it becomes the basement to AR research works, about 3D object manipulation, or some relevant issues.