3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> This paper presents a set of interaction techniques for use in headtracked immersive virtual environments. With these techniques, the user interacts with the 2D projections that 3D objects in the scene make on his image plane. The desktop analog is the use of a mouse to interact with objects in a 3D scene based on their projections on the monitor screen. Participants in an immersive environment can use the techniques we discuss for object selection, object manipulation, and user navigation in virtual environments. CR Categories and Subject Descriptors: 1.3.6 [Computer Graphics]: Methodology and Techniques - InteractionTechniques; 1.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism - VirtualReality. Additional Keywords: virtual worlds, virtual environments, navigation, selection, manipulation. <s> BIB001 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> Virtual environments provide a whole new way of viewing and manipulating 3D data. Current technology moves the images out of desktop monitors and into the space immediately surrounding the user. Users can literally put their hands on the virtual objects. Unfortunately techniques for interacting with such environments have yet to mature. Gloves and sensor based trackers are unwieldy, constraining and uncomfortable to use. A natural, more intuitive method of interaction would be to allow the user to grasp objects with their hands and manipulate them as if they were real objects. We are investigating the use of computer vision in implementing a natural interface based on hand gestures. A framework for a gesture recognition system is introduced along with results of experiments in colour segmentation, feature extraction and template matching for finger and hand tracking and hand pose recognition. Progress in the implementation of a gesture interface for navigation and object manipulation in virtual environments is discussed. <s> BIB002 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> Tabletop computing techniques are using physically familiar force-based interactions to enable compelling interfaces that provide a feeling of being embodied with a virtual object. We introduce an interaction paradigm that has the benefits of force-based interaction complete with full 6DOF manipulation. Only multi-touch input, such as that provided by the Microsoft Surface and the SMART Table, is necessary to achieve this interaction freedom. This paradigm is realized through sticky tools: a combination of sticky fingers, a physically familiar technique for moving, spinning, and lifting virtual objects; opposable thumbs, a method for flipping objects over; and virtual tools, a method for propagating behaviour to other virtual objects in the scene. We show how sticky tools can introduce richer meaning to tabletop computing by drawing a parallel between sticky tools and the discussion in Urp [20] around the meaning of tangible devices in terms of nouns, verbs, reconfigurable tools, attributes, and pure objects. We then relate this discussion to other force-based interaction techniques by describing how a designer can introduce complexity in how people can control both physical and virtual objects, how physical objects can control both physical and virtual objects, and how virtual objects can control virtual objects. <s> BIB003 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> The vision-based detection of hand gestures is one technological enabler for Natural User Interfaces which try to provide a natural and intuitive interaction with computers. In particular, mobile devices might benefit from such a less device-centric but more natural input possibility. In this paper, we introduce our ongoing work on the visual markerless detection of fingertips on mobile devices. Further, we shed light on the potential of mobile hand gesture detection and present several promising use cases and respective demo applications based on the presented engine. <s> BIB004 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> Multitouch input devices afford effective solutions for 6DOF (six Degrees of Freedom) manipulation of 3D objects. Mainly focusing on large-size multitouch screens, existing solutions typically require at least three fingers and bimanual interaction for full 6DOF manipulation. However, single-hand, two-finger operations are preferred especially for portable multitouch devices (e.g., popular smartphones) to cause less hand occlusion and relieve the other hand for necessary tasks like holding the devices. Our key idea for full 6DOF control using only two contact fingers is to introduce two manipulation modes and two corresponding gestures by examining the moving characteristics of the two fingers, instead of the number of fingers or the directness of individual fingers as done in previous works. We solve the resulting binary classification problem using a learning-based approach. Our pilot experiment shows that with only two contact fingers and typically unimanual interaction, our technique is comparable to or even better than the state-of-the-art techniques. © 2012 Wiley Periodicals, Inc. <s> BIB005 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> Three-dimensional capabilities on mobile devices are increasing, and the interactivity is becoming a key feature of these tools. It is expected that users will actively engage with the 3D content, instead of being passive consumers. Because touch-screens provide a direct means of interaction with 3D content by directly touching and manipulating 3D graphical elements, touch-based interaction is a natural and appealing style of input for 3D applications. However, developing 3D interaction techniques for handheld devices using touch-screens is not a straightforward task. One issue is that when interacting with 3D objects, users occlude the object with their fingers. Furthermore, because the user's finger covers a large area of the screen, the smallest size of the object users can touch is limited. In this paper, we first inspect existing 3D interaction techniques based on their performance with handheld devices. Then, we present a set of precise Dual-Finger 3D Interaction Techniques for a small display. Finally, we present the results of an experimental study, where we evaluate the usability, performance, and error rate of the proposed and existing 3D interaction techniques. <s> BIB006 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> Multitouch displays represent a promising technology for the display and manipulation of data. While the manipulation of 2D data has been widely explored, 3D manipulation with multitouch displays remains largely unexplored. Based on an analysis of the integration and separation of degrees of freedom, we propose a taxonomy for 3D manipulation techniques with multitouch displays. Using that taxonomy, we introduce Depth-Separated Screen-Space (DS3), a new 3D manipulation technique based on the separation of translation and rotation. In a controlled experiment, we compared DS3 with Sticky Tools and Screen-Space. Results show that separating the control of translation and rotation significantly affects performance for 3D manipulation, with DS3 performing faster than the two other techniques. <s> BIB007 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> Rotating 3D objects is a difficult task on mobile devices, because the task requires 3 degrees of freedom and (multi-)touch input only allows for an indirect mapping. We propose a novel style of mobile interaction based on mid-air gestures in proximity of the device to increase the number of DOFs and alleviate the limitations of touch interaction with mobile devices. While one hand holds the device, the other hand performs mid-air gestures in proximity of the device to control 3D objects on the mobile device's screen. A flat hand pose defines a virtual surface which we refer to as the PalmSpace for precise and intuitive 3D rotations. We constructed several hardware prototypes to test our interface and to simulate possible future mobile devices equipped with depth cameras. We conducted a user study to compare 3D rotation tasks using the most promising two designs for the hand location during interaction -- behind and beside the device -- with the virtual trackball, which is the current state-of-art technique for orientation manipulation on touch-screens. Our results show that both variants of PalmSpace have significantly lower task completion times in comparison to the virtual trackball. <s> BIB008 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> Rotating 3D objects is a difficult task. We present a new rotation technique based on collision-free "mating" to expedite 3D rotations. It is specifically designed for one-handed interaction on tablets or touchscreens. A user study found that our new technique decreased the time to rotate objects in 3D by more than 60% in situations where objects align. We found similar results when users translated and rotated objects in a 3D scene. Also, angle errors were 35% less with mating. In essence, our new rotation technique improves both the speed and accuracy of common 3D rotation tasks. <s> BIB009 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> Now that 3D interaction is available on tablets and smart phones, it becomes critical to provide efficient 3D interaction techniques for novice users. This paper investigates interaction techniques for 3D rotation with two fingers of a single hand, on multitouch mobile devices. We introduce two new rotation techniques that allow integral control of the 3 axes of rotation. These techniques also satisfy a new criterion that we introduce: surjection. We ran a study to compare the new techniques with two widely used rotation techniques from the literature. Results indicate that surjection and integration lead to a performance improvement of a group of participants who had no prior experience in 3D interaction. Qualitative results also indicate participants' preference for the new interaction techniques. <s> BIB010 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> In mobile applications it is crucial to provide intuitive means for 2D and 3D interaction. A large number of techniques exist to support a natural user interface (NUI) by detecting the user's hand posture in RGB+D (depth) data. Depending on a given interaction scenario, each technique has its advantages and disadvantages. To evaluate the performance of the various techniques on a mobile device, we conducted a systematic study by comparing the accuracy of five common posture recognition approaches with varying illumination and background. To be able to perform this study, we developed a powerful software framework that is capable of processing and fusing RGB and depth data directly on a handheld device. Overall results reveal best recognition rate of posture detection for combined RGB+D data at the expense of update rate. Finally, to support users in choosing the appropriate technique for their specific mobile interaction task, we derived guidelines based on our study. <s> BIB011 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS <s> Virtual Reality environments are able to offer natural interaction metaphors. However, it is difficult to accurately place virtual objects in the desired position and orientation using gestures in mid-air. Previous research concluded that the separation of degrees-of-freedom (DOF) can lead to better results, but these benefits come with an increase in time when performing complex tasks, due to the additional number of transformations required. In this work, we assess whether custom transformation axes can be used to achieve the accuracy of DOF separation without sacrificing completion time. For this, we developed a new manipulation technique, MAiOR, which offers translation and rotation separation, supporting both 3-DOF and 1-DOF manipulations, using personalized axes for the latter. Additionally, it also has direct 6-DOF manipulation for coarse transformations, and scaled object translation for increased placement. We compared MAiOR against an exclusively 6-DOF approach and a widget-based approach with explicit DOF separation. Results show that, contrary to previous research suggestions, single DOF manipulations are not appealing to users. Instead, users favored 3-DOF manipulations above all, while keeping translation and rotation independent. <s> BIB012
Instead of large-scale multi-touch display common in tabletops as discussed above, the touch-based interaction technique has also been investigated of its possibility after the obtainability of touchscreen handheld mobile devices commonly referred to as smartphones due to its popularity and mobility nowadays. In the early stage, Liu et al. BIB005 conducted a study relating to touch-based interaction that considers small touch-based display. In their research, they proposed a two-finger gesture for full 6DOF 3D object manipulation on a portable small touch-based display. They had improved the work done by Hancock et al. BIB003 in which two moving fingers on a single hand are used for 3D object translation and rotation on one axis, totally consisting of 4DOF of 3D object manipulation, while another 2DOF (2 more axes for 3D object rotation) are done by using one moving finger with another fixed finger (1f-1m) as shown in Fig. 3 . Their technique focuses on small touch-based display but the gesture to rotate a 3D object using the 1f-1m technique is sometimes difficult since the action itself may cause the difficulty in making sure that a finger is really fixed on a position. Slice motion occurs and causes the inaccuracy of the 3D object manipulation. In the same context, Telkenaroglu and Capin BIB006 also did a similar study that focuses on limited touch display due to the small screen size. They evaluated their technique with several VOLUME 7, 2019 well-known 3D interaction techniques that are not completely touch-based. For 3D object selection, they suggested combining the touch-based technique with ray casting to improve the accuracy in selecting single objects within multiple-object environments. As for 3D object manipulation, the same control structure suggested by Liu et al. BIB005 was implemented. Apart from the difficulty in using 2-finger touch on small touch display for 3D object rotation especially, occlusion is another highlighted issue BIB001 . Compared with 3D object translation, 3D object rotation is a crucial problem in touch-based interaction as discussed previously. Therefore, Scheurich and Stuerzlinger BIB009 did similar research focusing on 3D object rotation by applying z-technique BIB007 on a small touchscreen display. Besides, Rousset et al. BIB010 also came out with a relevant study by introducing surjection as the main criterion instead of integration for 3D object rotation. By improving the existing 3D object rotation technique produced by BIB010 considering surjection, they claimed that it would provide better results. Instead of both hands interaction as discussed above, the midair gestures-based interaction technique and then been investigated of its possibility towards using single hand after the obtainability of handheld mobile devices commonly referred to as smartphones due to its popularity and mobility nowadays that require the user to hold the device with one hand. Therefore, apart from the mid-air gestures-based interaction utilizes both hands, interaction in mid-air using single hand also had been studied by Segen and Kumar  , O'Hagan and Zelinsky BIB002 , and Baldauf et al. BIB004 . These earlier researches showed that one-hand can be used for mid-air interaction by applying the finger pose and motion estimation. Since the 3D object rotation seem much difficult compared with 3D object translation due to the self-occlusion  , BIB012 , Kratz et al. BIB008 proposed PalmSpace that used the user's palm to perform 3D object rotation. PalmSpace had been compared with the multi-touch technique in 3D object rotation task. They claimed that their prototype performed well and could be extended to 3D object translation. However, due to the constraint of the wrist joint, only small-range rotation can be performed. Difference mid-air gestures used to perform 3D object manipulation causing postures recognition became crucial important, the evaluation had been conducted by Fritz et al. BIB011 lately. Three (3) types of hand or finger postures (five (5) fingers outstretched; pointer finger outstretched and hand palm) had been utilized to form difference control structures to do 3D interaction. RGB-D (depth) data had been used to improve the detection's accuracy of hand palm postures. Results from their pose recognition test showed that RGB+D hand posture detection methods can improve the hand postures recognition within a complicated background (cluttered).