3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VOLUME 7, 2019 <s> The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device’s video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device’s touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks. <s> BIB001 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VOLUME 7, 2019 <s> Compared with traditional screen-touch input, natural gesture-based interaction approaches could offer a more intuitive user experience in handheld Augmented Reality (AR) applications. However, most gesture interaction techniques for handheld AR only use two degrees of freedom without the third depth dimension, while AR virtual objects are overlaid on a view of a three dimensional space. In this paper, we investigate a markerless fingertip-based 3D interaction method within a client-server framework in a small workspace. Our solution includes seven major components: (1) fingertip detection (2) fingertip depth acquisition (3) marker tracking (4) coordinate transformation (5) data communication (6) gesture interaction (7) graphic rendering. We describe the process of each step in details and present performance results of our prototype. <s> BIB002 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VOLUME 7, 2019 <s> Over the past few years, Augmented Reality has become widely popular in the form of smart phone applications, however most smart phone-based AR applications are limited in user interaction and do not support gesture-based direct manipulation of the augmented scene. In this paper, we introduce a new AR interaction methodology, employing users' hands and fingers to interact with the virtual (and possibly physical) objects that appear on the mobile phone screen. The goal of this project was to support different types of interaction (selection, transformation, and fine-grain control of an input value) while keeping the methodology for hand detection as simple as possible to maintain good performance on smart phones. We evaluated our methods in user studies, collecting task performance data and user impressions about this direct way of interacting with augmented scenes through mobile phones. <s> BIB003 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VOLUME 7, 2019 <s> In this paper, we present a novel gesture-based interaction method for handheld Augmented Reality (AR) implemented on a tablet with an RGB-Depth camera attached. Compared with conventional device-centric interaction methods like keypad, stylus, or touchscreen input, natural gesture-based interfaces offer a more intuitive experience for AR applications. Combining with depth information, gesture interfaces can extend handheld AR interaction into full 3D space. In our system we retrieve the 3D hand skeleton from color and depth frames, mapping the results to corresponding manipulations of virtual objects in the AR scene. Our method allows users to control virtual objects in 3D space using their bare hands and perform operations such as translation, rotation, and zooming. <s> BIB004
Moreover, Hurst and Wezel BIB001 did a comprehensive study to utilize the mid-air gestures-based interaction technique towards 3D object selection and manipulation. Results from their user studies showed that participants felt fun and were willing to use their technique because of its intuitiveness and naturality. However, this technique was limited to the translation distance since it was constrained within the arm length and the self-occlusion problem was instead less accurate compared with touch-based interaction. Bai et al. BIB002 ,  then suggested the markerless fingertip-based 3D interaction consisting of seven (7) components which are 1) fingertip detection; 2) fingertip depth acquisition; 3) marker tracking; 4) fingertip coordinate transformation; 5) data communication; 6) fingertip-based interaction and 7) virtual content, rendering in solving the depth dimension problem. Their system needed additional hardware such as Kinect sensor and desktop computer as a server to offer complete 3D object manipulation. BIB003 and BIB004 . (a) Processes involved in BIB003 . (b) Finger tracking in BIB004 .