3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> This paper surveys the field of augmented reality AR, in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality. <s> BIB001 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> An experiment was conducted to investigate the relationship between object transportation and object orientation by the human hand in the context of humancomputer interaction (HCI). This work merges two streams of research: the structure of interactive manipulation in HCI and the natural hand prehension in human motor control. It was found that object transportation and object orientation have a parallel, interdependent structure which is generally persistent over different visual feedback conditions. The notion of concurrency and interdependence of multidimensional visuomotor control structure can provide a new framework for human-computer interface evaluation and design. <s> BIB002 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> This paper introduces a new sensor architecture for making interactive surfaces that are sensitive to human hand and finger gestures. This sensor recognizes multiple hand positions and shapes and calculates the distance between the hand and the surface by using capacitive sensing and a mesh-shaped antenna. In contrast to camera-based gesture recognition systems, all sensing elements can be integrated within the surface, and this method does not suffer from lighting and occlusion problems. This paper describes the sensor architecture, as well as two working prototype systems: a table-size system and a tablet-size system. It also describes several interaction techniques that would be difficult to perform without using this architecture <s> BIB003 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> A digital tabletop offers several advantages over other groupware form factors for collaborative applications. However, users of a tabletop system do not share a common perspective for the display of information: what is presented right side up to one participant is upside down for another. In this paper, we survey five different rotation and translation techniques for objects displayed on a direct touch digital tabletop display. We analyze their suitability for interactive tabletops in light of their respective input and output degrees of freedom, as well as the precision and completeness provided by each. We describe various tradeoffs that arise when considering which, when and where each of these techniques might be most useful. <s> BIB004 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> Tabletop computing techniques are using physically familiar force-based interactions to enable compelling interfaces that provide a feeling of being embodied with a virtual object. We introduce an interaction paradigm that has the benefits of force-based interaction complete with full 6DOF manipulation. Only multi-touch input, such as that provided by the Microsoft Surface and the SMART Table, is necessary to achieve this interaction freedom. This paradigm is realized through sticky tools: a combination of sticky fingers, a physically familiar technique for moving, spinning, and lifting virtual objects; opposable thumbs, a method for flipping objects over; and virtual tools, a method for propagating behaviour to other virtual objects in the scene. We show how sticky tools can introduce richer meaning to tabletop computing by drawing a parallel between sticky tools and the discussion in Urp [20] around the meaning of tangible devices in terms of nouns, verbs, reconfigurable tools, attributes, and pure objects. We then relate this discussion to other force-based interaction techniques by describing how a designer can introduce complexity in how people can control both physical and virtual objects, how physical objects can control both physical and virtual objects, and how virtual objects can control virtual objects. <s> BIB005 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> Rotate-Scale-Translate (RST) interactions have become the de facto standard when interacting with two-dimensional (2D) contexts in single-touch and multi-touch environments. Because the use of RST has thus far focused almost entirely on 2D, there are not yet standard techniques for extending these principles into three dimensions. In this paper we describe a screen-space method which fully captures the semantics of the traditional 2D RST multi-touch interaction, but also allows us to extend these same principles into three-dimensional (3D) interaction. Just like RST allows users to directly manipulate 2D contexts with two or more points, our method allows the user to directly manipulate 3D objects with three or more points. We show some novel interactions, which take perspective into account and are thus not available in orthographic environments. Furthermore, we identify key ambiguities and unexpected behaviors that arise when performing direct manipulation in 3D and offer solutions to mitigate the difficulties each presents. Finally, we show how to extend our method to meet application-specific control objectives, as well as show our method working in some example environments. <s> BIB006 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> Multi-touch displays represent a promising technology for the display and manipulation of 3D data. To fully exploit their capabilities, appropriate interaction techniques must be designed. In this paper, we explore the design of free 3D positioning techniques for multi-touch displays to exploit the additional degrees of freedom provided by this technology. Our contribution is two-fold: first we present an interaction technique to extend the standard four view-ports technique found in commercial CAD applications, and second we introduce a technique designed to allow free 3D positioning with a single view of the scene. The two techniques were evaluated in a preliminary experiment. The first results incline us to conclude that the two techniques are equivalent in term of performance showing that the Z-technique provides a real alternative to the statu quo viewport technique. <s> BIB007 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> 3D transformation widgets are commonly used in many 3D applications operated from mice and keyboards. These user interfaces allow independent control of translations, rotations, and scaling for manipulation of 3D objects. In this paper, we study how these widgets can be adapted to the tactile paradigm. We have explored an approach where users apply rotations by means of physically plausible gestures, and we have extended successful 2D tactile principles to the context of 3D interaction. These investigations led to the design of a new 3D transformation widget, tBox, that can been operated easily and efficiently from gestures on touch-screens. <s> BIB008 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> Multitouch displays represent a promising technology for the display and manipulation of data. While the manipulation of 2D data has been widely explored, 3D manipulation with multitouch displays remains largely unexplored. Based on an analysis of the integration and separation of degrees of freedom, we propose a taxonomy for 3D manipulation techniques with multitouch displays. Using that taxonomy, we introduce Depth-Separated Screen-Space (DS3), a new 3D manipulation technique based on the separation of translation and rotation. In a controlled experiment, we compared DS3 with Sticky Tools and Screen-Space. Results show that separating the control of translation and rotation significantly affects performance for 3D manipulation, with DS3 performing faster than the two other techniques. <s> BIB009 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> Multitouch input devices afford effective solutions for 6DOF (six Degrees of Freedom) manipulation of 3D objects. Mainly focusing on large-size multitouch screens, existing solutions typically require at least three fingers and bimanual interaction for full 6DOF manipulation. However, single-hand, two-finger operations are preferred especially for portable multitouch devices (e.g., popular smartphones) to cause less hand occlusion and relieve the other hand for necessary tasks like holding the devices. Our key idea for full 6DOF control using only two contact fingers is to introduce two manipulation modes and two corresponding gestures by examining the moving characteristics of the two fingers, instead of the number of fingers or the directness of individual fingers as done in previous works. We solve the resulting binary classification problem using a learning-based approach. Our pilot experiment shows that with only two contact fingers and typically unimanual interaction, our technique is comparable to or even better than the state-of-the-art techniques. © 2012 Wiley Periodicals, Inc. <s> BIB010 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> III. TOUCH-BASED INTERACTION <s> In the field of scientific visualization, 3D manipulation is a fundamental task for many different scientific datasets, such as particle data in physics and astronomy, fluid data in aerography, and structured data in medical science. Current researches show that large multi-touch interactive displays serve as a promising device providing numerous significant advantages for displaying and manipulating scientific data. Those benefits of direct-touch devices motivate us to use touch-based interaction techniques to explore scientific 3D data. However, manipulating object in 3D space via 2D touch input devices is challenging for precise control. Therefore, we present a novel multi-touch approach for manipulating structured objects in 3D visualization space, based on multi-touch gestures and an extra axis for the assistance. Our method supports 7-DOF manipulations. Moreover, with the help from the extra axis and depth hints, users can have better control of the interactions. We report on a user study to make comparisons between our method and standard mouse-based 2D interface. We show in this work that touch-based interactive displays can be more effective when applied to complex problems if the interactive visualizations and interactions are designed appropriately. <s> BIB011
Touch-based interaction becomes the main research focus recently after the introduction of touchscreen handheld mobile devices  . In the earlier approach on the mobile touchscreen, it is designed to use fingers to navigate on the screen which alters the interaction experience with mobile applications  so users can directly interact with the onscreen objects through their fingers  . In AR, 3D interaction is one of the main concepts defined in its definition BIB001 -  . 3D object manipulation is a challenge for interface designer since it involves the control of six degrees of freedom (6DOF) which include 3DOF for object translation (x, y, and z-axes) and 3DOF for object rotation (x, y, and z-axes), so does AR. On account that 3D object manipulation is essential in AR interfaces, touch-based interaction provides another option for users to perform 3D object manipulation. The touch-based concept is utilized by using touch input through multiple contact points instead of using the traditional keyboard or mouse inputs that have already been verified much slower than real object manipulation BIB002 . Besides, by performing touch-based interaction, users can perform direct manipulation by touching the 3D data directly on screen BIB003 . Finger gestures that include multi-contact points especially for object rotation are used to manipulate on-screen objects as proposed by Hancock et al. BIB004 , which is an example of touch-based interaction technique used to manipulate 2D data on touchscreen tabletops using RNT (Rotate N'Translate) mechanism. However, touch-based interaction has only been widely explored to do 2D object manipulation that includes x and y-axes only while depth manipulation (z-axis) is still unexplored. This arises due to the difficulty of mapping 2D touch points to 3D attributes to perform the complete 3D object manipulation while the 3D object manipulations consist of 6DOF BIB009 are mostly executed significantly in virtual environments. Thus, to understand touch-based interaction better, firstly we need to learn more about its basic concepts that rely early on the multi-touch tabletop display. Based on several previous studies, additional hardware is needed to provide in-depth information to enable manipulation along z-axis integrally. For example, Wilson et al.  claimed that new range-sensing camera is needed to provide in-depth information to construct a rich 3D model, assisting the user when lifting a 3D object up or down along the z-axis to place it on or below another object. The requirement of new hardware assistance may cause difficulty on both compatibility and cost-effectiveness BIB009 ,  . Due to the similar issue of in-depth information, Martinet et al. BIB007 introduced the Z-technique to provide a more immersive experience to the user while only a single viewport is available on multitouch tabletop display and the user can perform 3D object positioning using two (2) finger touches. The movement of the second finger backward or forward provides information to control the depth position. Their study focused on the 3D positioning task and has not been extended to full 3D object manipulation. The effort to perform full 3D object manipulation was then realized by applying touch control separately instead to perform integral 3D object manipulation as stated previously. In this context, Hancock et al. BIB004 introduced the use of three-touches interaction called Shallow-depth to perform the FIGURE 3. Two-finger gestures for 3D object manipulation on small touch-based display suggested by BIB010 . 5DOF of translating and rotating 3D objects on the multitouch tabletop display. This technique requires utilizing both hands of the user to map in-depth information and all the axes. By calculating the distance between two fingers, the user can perform 3D object manipulation along the z-axis. Separating the inputs used for different tasks allows the user to perform 3D object manipulation. Their research found that their technique is faster and more precise, also preferred by their users. Extended from the previous work, Hancock et al. BIB005 introduced sticky tools to consummate the previous technique to perform the complete 6DOF manipulation by extending the standard RNT mechanism into 3D using the screen-space approach. Oppositely, Reisman et al. BIB006 suggested several interaction techniques based on the same control structure mapped along x, y, and z-axes in order to provide integral 3D object rotation. Based on both researches done by Hancock et al. BIB005 and Reisman et al. BIB006 , a new touch-based interaction technique, tBox BIB008 was then proposed. tBox widget appeared as a wireframe box to ease the user when performing 3D object manipulation, where a single finger was used to do 3D object translation and rotation for direct access while scaling was done by applying two fingers. By using tBox, the user understands accordingly which axis is involved when a single manipulation task is activated through separating the DOF of 3D object translation, rotation, and scaling. Since DOF integration and separation are two aspects affect the performances in touch-based interaction discussed widely in BIB005 and BIB006 . Martinet et al. BIB009 provided a comparison between the touch-based interaction techniques which are sticky tools BIB005 , screen-space BIB006 and DS3 BIB009 while DS3 separates the DOF of 3D object translation and rotation. DS3 combines the z-technique and screen-space technique to perform the 3D object rotation. These three (3) different control structures were compared towards the 3D object manipulation task designed. The results showed that the separation of 3D object translation and rotation increases the performance in task completion time compared with just performing both 3D object translation and rotation integrally (all 6DOF integration) or separately (all 6DOF separation). These studies BIB009 , BIB005 , BIB006 provide a certain degree of inspiration for the followers, as an example, the effectiveness of the integration of DOF and the total DOF that can be controlled by the fingers of a single hand were evaluated by Brouet et al.  , and the results showed that the restrictions between each finger such as the motion and their interferences cause difficulty in using all fingers in controlling 3D object manipulation. Besides that, users like to limit themselves to using one to a maximum of three (3) fingers per hand to complete complex interactions such as lifting or scaling and also manipulations that consist of all x, y, and z-axes for 3D object translation and rotation. Due to the difficulty in performing full 6DOF of 3D object manipulation, Guo et al. BIB011 proposed another touch-based technique that combines multitouch gestures with an additional assistant axis performed on touch-based tabletops. They claimed that their technique can perform 7DOF of 3D object manipulation including uniform zooming instead of 3D object translation and rotation. However, their technique requires the operation of both hands with indirect contact points.