3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Retrieving the stylus of a pen-based device takes time and requires a second hand. Especially for short intermittent interactions many users therefore choose to use their bare fingers. Although convenient, this increases targeting times and error rates. We argue that the main reasons are the occlusion of the target by the user's finger and ambiguity about which part of the finger defines the selection point. We propose a pointing technique we call Shift that is designed to address these issues. When the user touches the screen, Shift creates a callout showing a copy of the occluded screen area and places it in a non-occluded location. The callout also shows a pointer representing the selection point of the finger. Using this visual feedback, users guide the pointer into the target by moving their finger on the screen surface and commit the target acquisition by lifting the finger. Unlike existing techniques, Shift is only invoked when necessary--over large targets no callout is created and users enjoy the full performance of an unaltered touch screen. We report the results of a user study showing that with Shift participants can select small targets with much lower error rates than an unaided touch screen and that Shift is faster than Offset Cursor for larger targets. <s> BIB001 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Balloon selection is a 3D interaction technique that is modeled after the real world metaphor of manipulating a helium balloon attached to a string. Balloon selection allows for precise 3D selection in the volume above a tabletop surface by using multiple fingers on a multi-touch-sensitive surface. The 3DOF selection tasks is decomposed in part into a 2DOF positioning task performed by one finger on the tabletop in an absolute 2D Cartesian coordinate system and a 1DOF positioning task performed by another finger on the tabletop in a relative 2D polar coordinate system. We have evaluated balloon selection in a formal user study that compared it to two well-known interaction techniques for selecting a static 3D target: a 3DOF tracked wand and keyboard cursor keys. We found that balloon selection was significantly faster than using cursor keys and had a significantly lower error rate than the wand. The lower error rate appeared to result from the user's hands being supported by the tabletop surface, resulting in significantly reduced hand tremor and arm fatigue. <s> BIB002 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> In this paper, we present Art of Defense (AoD), a cooperative handheld augmented reality (AR) game. AoD is an example of what we call an AR Board Game, a class of tabletop games that combine handheld computers (such as camera phones) with physical game pieces to create a merged physical/virtual game on the table-top. This paper discusses the technical aspects of the game, the design rationale and process we followed, and the resulting player experience. The goal of this research is to explore the affordances and constraints of handheld AR interfaces for collaborative social games, and to create a game that leverages them as fully as possible. The results from the user study show that the game is fun to play, and that by tightly registering the virtual content with the tangible game pieces, tabletop AR games enable a kind of social play experience unlike non-AR computer games. We hope this research will inspire the creation of other handheld augmented reality games in the future, both on and off the tabletop. <s> BIB003 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> In this paper, we explore how to add pointing input capabilities to very small screen devices. On first sight, touchscreens seem to allow for particular compactness, because they integrate input and screen into the same physical space. The opposite is true, however, because the user's fingers occlude contents and prevent precision. We argue that the key to touch-enabling very small devices is to use touch on the device backside. In order to study this, we have created a 2.4" prototype device; we simulate screens smaller than that by masking the screen. We present a user study in which participants completed a pointing task successfully across display sizes when using a back-of device interface. The touchscreen-based control condition (enhanced with the shift technique), in contrast, failed for screen diagonals below 1 inch. We present four form factor concepts based on back-of-device interaction and provide design guidelines extracted from a second user study. <s> BIB004 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Rotate-Scale-Translate (RST) interactions have become the de facto standard when interacting with two-dimensional (2D) contexts in single-touch and multi-touch environments. Because the use of RST has thus far focused almost entirely on 2D, there are not yet standard techniques for extending these principles into three dimensions. In this paper we describe a screen-space method which fully captures the semantics of the traditional 2D RST multi-touch interaction, but also allows us to extend these same principles into three-dimensional (3D) interaction. Just like RST allows users to directly manipulate 2D contexts with two or more points, our method allows the user to directly manipulate 3D objects with three or more points. We show some novel interactions, which take perspective into account and are thus not available in orthographic environments. Furthermore, we identify key ambiguities and unexpected behaviors that arise when performing direct manipulation in 3D and offer solutions to mitigate the difficulties each presents. Finally, we show how to extend our method to meet application-specific control objectives, as well as show our method working in some example environments. <s> BIB005 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> In this paper, we present a Fitts' law-based formal evaluation process and the corresponding results for 3D object manipulation techniques based on a virtual hand metaphor in a tangible augmented reality (TAR) environment. Specifically, we extend the design parameters of the 1D scale Fitts' law to 3D scale and then refine an evaluation model in order to bring generality and ease of adaptation to various TAR applications. Next, we implement and compare standard TAR manipulation techniques using a cup, a paddle, a cube, and a proposed extended paddle prop. Most manipulation techniques were well-modeled in terms of linear regression according to Fitts' law, with a correlation coefficient value of over 0.9. Notably, the throughput by ISO 9241-9 of the extended paddle technique peaked at around 1.39 to 2 times higher than in the other techniques, due to the instant 3D positioning of the 3D objects. In the discussion, we subsequently examine the characteristics of the TAR manipulation techniques in terms of stability, speed, comfort, and understanding. As a result, our evaluation process, results, and analysis can be useful in guiding the design and implementation of future TAR interfaces. <s> BIB006 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Multi-touch displays represent a promising technology for the display and manipulation of 3D data. To fully exploit their capabilities, appropriate interaction techniques must be designed. In this paper, we explore the design of free 3D positioning techniques for multi-touch displays to exploit the additional degrees of freedom provided by this technology. Our contribution is two-fold: first we present an interaction technique to extend the standard four view-ports technique found in commercial CAD applications, and second we introduce a technique designed to allow free 3D positioning with a single view of the scene. The two techniques were evaluated in a preliminary experiment. The first results incline us to conclude that the two techniques are equivalent in term of performance showing that the Z-technique provides a real alternative to the statu quo viewport technique. <s> BIB007 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device’s video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device’s touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks. <s> BIB008 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> The rising popularity of digital table surfaces has spawned considerable interest in new interaction techniques. Most interactions fall into one of two modalities: 1) direct touch and multi-touch (by hand and by tangibles) directly on the surface, and 2) hand gestures above the surface. The limitation is that these two modalities ignore the rich interaction space between them. To move beyond this limitation, we first contribute a unification of these discrete interaction modalities called the continuous interaction space. The idea is that many interaction techniques can be developed that go beyond these two modalities, where they can leverage the space between them. That is, we believe that the underlying system should treat the space on and above the surface as a continuum, where a person can use touch, gestures, and tangibles anywhere in the space and naturally move between them. Our second contribution illustrates this, where we introduce a variety of interaction categories that exploit the space between these modalities. For example, with our Extended Continuous Gestures category, a person can start an interaction with a direct touch and drag, then naturally lift off the surface and continue their drag with a hand gesture over the surface. For each interaction category, we implement an example (or use prior work) that illustrates how that technique can be applied. In summary, our primary contribution is to broaden the design space of interaction techniques for digital surfaces, where we populate the continuous interaction space both with concepts and examples that emerge from considering this space as a continuum. <s> BIB009 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> In this paper we present an Augmented Reality (AR) authoring tool for smartphones which facilitates intuitive interactions that manipulate the augmented virtual objects in real-time. A novel 3D interaction method using multi-touch interface and camera pose is proposed for intuitive authoring. With the gestures of two fingers on the touch screen, the user can adjust 3 DOF translation and 3 DOF rotation to a selected virtual object. The capabilities of the authoring tool are demonstrated on a smartphone. <s> BIB010 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Interaction techniques for handheld mobile Augmented Reality (AR) often focus on device-centric methods based around touch input. However, users may not be able to easily interact with virtual objects in mobile AR scenes if they are holding the handheld device with one hand and touching the screen with the other, while at the same time trying to maintain visual tracking of an AR marker. In this paper we explore novel interaction methods for handheld mobile AR that overcomes this problem. We investigate two different approaches; (1) freeze view touch and (2) finger gesture based interaction. We describe how each method is implemented and present findings from a user experiment comparing virtual object manipulation with these techniques to more traditional touch methods. <s> BIB011 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Modern mobile devices allow a rich set of multi-finger interactions that combine modes into a single fluid act, for example, one finger for panning blending into a two-finger pinch gesture for zooming. Such gestures require the use of both hands: one holding the device while the other is interacting. While on the go, however, only one hand may be available to both hold the device and interact with it. This mostly limits interaction to a single-touch (i.e., the thumb), forcing users to switch between input modes explicitly. In this paper, we contribute the Fat Thumb interaction technique, which uses the thumb's contact size as a form of simulated pressure. This adds a degree of freedom, which can be used, for example, to integrate panning and zooming into a single interaction. Contact size determines the mode (i.e., panning with a small size, zooming with a large one), while thumb movement performs the selected mode. We discuss nuances of the Fat Thumb based on the thumb's limited operational range and motor skills when that hand holds the device. We compared Fat Thumb to three alternative techniques, where people had to precisely pan and zoom to a predefined region on a map and found that the Fat Thumb technique compared well to existing techniques. <s> BIB012 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Over the past few years, Augmented Reality has become widely popular in the form of smart phone applications, however most smart phone-based AR applications are limited in user interaction and do not support gesture-based direct manipulation of the augmented scene. In this paper, we introduce a new AR interaction methodology, employing users' hands and fingers to interact with the virtual (and possibly physical) objects that appear on the mobile phone screen. The goal of this project was to support different types of interaction (selection, transformation, and fine-grain control of an input value) while keeping the methodology for hand detection as simple as possible to maintain good performance on smart phones. We evaluated our methods in user studies, collecting task performance data and user impressions about this direct way of interacting with augmented scenes through mobile phones. <s> BIB013 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> In this paper, we present a novel gesture-based interaction method for handheld Augmented Reality (AR) implemented on a tablet with an RGB-Depth camera attached. Compared with conventional device-centric interaction methods like keypad, stylus, or touchscreen input, natural gesture-based interfaces offer a more intuitive experience for AR applications. Combining with depth information, gesture interfaces can extend handheld AR interaction into full 3D space. In our system we retrieve the 3D hand skeleton from color and depth frames, mapping the results to corresponding manipulations of virtual objects in the AR scene. Our method allows users to control virtual objects in 3D space using their bare hands and perform operations such as translation, rotation, and zooming. <s> BIB014 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Nowadays, handheld devices are capable of displaying augmented environments in which virtual content overlaps reality. To interact with these environments it is necessary to use a manipulation technique. The objective of a manipulation technique is to define how the input data modify the properties of the virtual objects. Current devices have multi-touch screens that can serve as input. Additionally, the position and rotation of the device can also be used as input creating both an opportunity and a design challenge. In this paper we compared three manipulation techniques which namely employ multi-touch, device position and a combination of both. A user evaluation on a docking task revealed that combining multi-touch and device movement yields the best task completion time and efficiency. Nevertheless, using only the device movement and orientation is more intuitive and performs worse only in large rotations. <s> BIB015 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> In this paper, we present a prototype for exploring natural gesture interaction with Handheld Augmented Reality (HAR) applications, using visual tracking based AR and freehand gesture based interaction detected by a depth camera. We evaluated this prototype in a user study comparing 3D gesture input methods with traditional touch-based techniques, using canonical manipulation tasks that are common in AR scenarios. We collected task performance data and user feedback via a usability questionnaire. The 3D gesture input methods were found to be slower, but the majority of the participants preferred them and gave them higher usability ratings. Being intuitive and natural was the most common feedback about the 3D freehand interface. We discuss implications of this research and directions for further work. <s> BIB016 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Mid-air interactions are prone to fatigue and lead to a feeling of heaviness in the upper limbs, a condition casually termed as the gorilla-arm effect. Designers have often associated limitations of their mid-air interactions with arm fatigue, but do not possess a quantitative method to assess and therefore mitigate it. In this paper we propose a novel metric, Consumed Endurance (CE), derived from the biomechanical structure of the upper arm and aimed at characterizing the gorilla-arm effect. We present a method to capture CE in a non-intrusive manner using an off-the-shelf camera-based skeleton tracking system, and demonstrate that CE correlates strongly with the Borg CR10 scale of perceived exertion. We show how designers can use CE as a complementary metric for evaluating existing and designing novel mid-air interactions, including tasks with repetitive input such as mid-air text-entry. Finally, we propose a series of guidelines for the design of fatigue-efficient mid-air interfaces. <s> BIB017 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> In this paper we present a novel framework for simultaneous detection of click action and estimation of occluded fingertip positions from egocentric viewed single-depth image sequences. For the detection and estimation, a novel probabilistic inference based on knowledge priors of clicking motion and clicked position is presented. Based on the detection and estimation results, we were able to achieve a fine resolution level of a bare hand-based interaction with virtual objects in egocentric viewpoint. Our contributions include: (i) a rotation and translation invariant finger clicking action and position estimation using the combination of 2D image-based fingertip detection with 3D hand posture estimation in egocentric viewpoint. (ii) a novel spatio-temporal random forest, which performs the detection and estimation efficiently in a single framework. We also present (iii) a selection process utilizing the proposed clicking action detection and position estimation in an arm reachable AR/VR space, which does not require any additional device. Experimental results show that the proposed method delivers promising performance under frequent self-occlusions in the process of selecting objects in AR/VR space whilst wearing an egocentric-depth camera-attached HMD. <s> BIB018 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Lately, mobile augmented reality (AR) has become very popular and is used for many commercial and product promotional activities. However, in almost all mobile AR applications, the user only views annotated information or the preset motion of the virtual object in an AR environment and is unable to interact with the virtual objects as if he/she were interacting with real objects in the real environment. In this paper, in an attempt to realize enhanced intuitive and realistic object manipulation in the mobile AR environment, we propose an integrated view-input AR interaction method, which integrates user device manipulation and virtual object manipulation. The method enables the user to hold a 3D virtual object by touching the displayed object on the 2D touch screen of a mobile device, and to move and rotate the object by moving and rotating the mobile device while viewing the held object by way of the 2D screen of the mobile device. Based on this concept, we implemented three types of integrated methods, namely the Rod, Center, and Touch methods, and conducted a user study to investigate the baseline performance metrics of the proposed method on an AR object manipulation task. The Rod method achieved the highest success rate (91%). Participants' feedback indicated that this is because the Rod method is the most natural, and evoked a fixed mental model that is conceivable in the real environment. These results indicated that visualizing the manipulation point on the screen and restricting the user's interactivity with virtual objects from the user's position of view based on a conceivable mental model would be able to aid the user to achieve precise manipulation. <s> BIB019 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Interaction for Handheld Augmented Reality (HAR) is a challenging research topic because of the small screen display and limited input options. Although 2D touch screen input is widely used, 3D gesture interaction is a suggested alternative input method. Recent 3D gesture interaction research mainly focuses on using RGB-Depth cameras to detect the spatial position and pose of fingers, using this data for virtual object manipulations in the AR scene. In this paper we review previous 3D gesture research on handheld interaction metaphors for HAR. We present their novelties as well as limitations, and discuss future research directions of 3D gesture interaction for HAR. Our results indicate that 3D gesture input on HAR is a potential interaction method for assisting a user in many tasks such as in education, urban simulation and 3D games. <s> BIB020 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Mobile augmented reality (AR) has been widely used in smart and mobile device-based applications such as entertainment, games, visual experience, and information visualization. However, most of the mobile AR applications have limitations in natural user interaction and do not fully support the direct manipulation of 3D AR objects. This paper proposes a new method for naturally and directly manipulating 3D AR objects through touch and hand gesture-based interactions in handheld devices. The touch gesture is used for the AR object selection and the natural hand gesture is used for the direct and interactive manipulation of the selected objects. Thus, the hybrid interaction makes the user more accurately interact with and manipulate AR objects in the real 3D space, not in the 2D space. In particular, natural hand gestures are detected by the Leap Motion sensor attached to the front or back of mobile devices. Thus the user can easily interacts with 3D AR objects for 3D transformation to enhance usability and usefulness. In this research, comprehensive comparative analyses were performed among the proposed approach and the widely used screen touch-based approach and vision-based approach in terms of quantitative and qualitative aspects. Quantitative analysis was conducted by measuring task completion time and failure rate to perform given tasks such as 3D object matching and grasp-hang-release operation. Both tasks require simultaneous 3D translation and 3D rotation. In addition, we have compared the gesture performance depending on whether the gesture sensor is located in the front or the back of the mobile device. Furthermore, to support other complex operations, an assembly task has also been evaluated. The assembly task consists of a sequence of combining parts into a sub-assembly. Qualitative analysis was performed through enquiring questionnaire after the experiment that examines factors such as ease-of-use, ease-of-natural interaction, etc. Both analyses showed that the proposed approach can provide more natural and intuitive interaction and manipulation of mobile AR objects. Several implementation results will also be given to show the advantage and effectiveness of the proposed approach. <s> BIB021 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Hand-held smart devices are equipped with powerful processing units, high resolution screens and cameras, that in combination makes them suitable for video see-through Augmented Reality. Many Augmented Reality applications require interaction, such as selection and 3D pose manipulation. One way to perform intuitive, high precision 3D pose manipulation is by direct or indirect mapping of device movement. There are two approaches to device movement interaction; one fixes the virtual object to the device, which therefore becomes the pivot point for the object, thus makes it difficult to rotate without translate. The second approach avoids latter issue by considering rotation and translation separately, relative to the object's center point. The result of this is that the object instead moves out of view for yaw and pitch rotations. In this paper we study these two techniques and compare them with a modification where user perspective rendering is used to solve the rotation issues. The study showed that the modification improves speed as well as both perceived control and intuitiveness among the subjects. <s> BIB022 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> VI. REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR <s> Abstract Handheld Augmented Reality (HAR) has the potential to introduce Augmented Reality (AR) to large audiences due to the widespread use of suitable handheld devices. However, many of the current HAR systems are not considered very practical and they do not fully answer to the needs of the users. One of the challenging areas in HAR is the in-situ AR content creation where the correct and accurate positioning of virtual objects to the real world is fundamental. Due to the hardware limitations of handheld devices and possible restrictions in the environment, the correct 3D positioning of objects can be difficult to achieve we are unable to use AR markers or correctly map the 3D structure of the environment. We present SlidAR, a 3D positioning for Simultaneous Localization And Mapping (SLAM) based HAR systems. SlidAR utilizes 3D ray-casting and epipolar geometry for virtual object positioning. It does not require a perfect 3D reconstruction of the environment nor any virtual depth cues. We have conducted a user experiment to evaluate the efficiency of SlidAR method against an existing device-centric positioning method that we call HoldAR. Results showed that SlidAR was significantly faster, required significantly less device movement, and also got significantly better subjective evaluation from the test participants. SlidAR also had higher positioning accuracy, although not significantly. <s> BIB023
Interaction techniques for 3D object manipulation can be roughly divided into three (3) main categories as stated in the previous section which are the touch-based interaction techniques, mid-air gestures-based interaction techniques and device-based interaction techniques although there are some other interaction techniques such as the real object-based interaction techniques which have been studied in BIB003 and BIB006 for handheld mobile AR. In real object-based interaction, the virtual object is coordinated with the real object marked by AR as the controller to translate and rotate the virtual object, but the position and orientation of virtual objects are restricted by those of the corresponding real objects (Fig. 12) , where virtual objects are unable to retain the same posture and height as the real objects. Furthermore, the user needs to make sure all the corresponding real objects are within the trackable range to avoid any AR marker tracking failure which limits its potential such as size and translation distance. Since there VOLUME 7, 2019 have been no recent studies about the real object-based interaction technique thus it is not be discussed further in this paper. After a detailed investigation towards the three (3) main interaction categories, several primary issues are highlighted as shown in Table 5 . Based on Table 5 , it shows the striving Occlusion (Fig. 13) is one of the common issues that has been mentioned frequently in the previous researches BIB010 - BIB015 , BIB008 , BIB013 - BIB016 . Occlusion always exists in the touch-based interaction techniques when users directly touch the 3D object on the screen while the 3D object touched may be occluded. On the other hand, self-occlusion occurs in mid-air gestures-based interaction when the user's own hand or finger causes the 3D object being occluded while bare-hand or fingers actions behind the camera scene but in front of the AR marker may also cause occlusion on the AR marker and then leads to the 3D object registration error. Due to the occlusion problem, several solutions had been suggested such as BIB001 , BIB004 , and BIB018 . In BIB001 , callout display method is used to display the occluded area near the finger touch but the entire finger below the touch still occluded while Paudisch and Chu BIB004 suggested new hardware design to perform touch input behind the touchscreen display that not yet been launched in the consumer market. In BIB018 , self-occlusion had been spatially solved by estimating the occluded hand and finger motions and gestures, but it is expensive due to the huge calculation requirement and also the limitation of the numbers of dataset according to FIGURE 13 . Some examples of occlusion cases happened in the touch-based and mid-air gestures-based interaction techniques; a virtual object occluded by the user's fingers BIB010 , b indirect touch to avoid occlusion BIB011 , c i-virtual object on screen ii-virtual object occluded by finger  , d Self-occlusions BIB008 , BIB013 , BIB016 , BIB021 , e Major features of AR marker had been occluded BIB014 . FIGURE 14. Occlusion had been excluded in the device-based interaction techniques. (a) In BIB022 . (b) In BIB019 . (c) In BIB023 . each of the potential hand and fingers motions. In this context, device-based interaction technique category is the only occlusion-free category showed in Fig. 14. 40596 VOLUME 7, 2019 For fatigue phenomenon, it is linked with the ability for the user to hold the handheld mobile device with both hands when interacting. In BIB020 , fatigue phenomenon has been highlighted and supported with BIB002 that holding device with one hand and using another hand for interaction might cause fatigue to the user. Based on BIB002 , hand tremor and arm fatigue may occur when the user cannot rest their arms and hands. This effect is obvious in mid-air gestures-based interaction when a user needs to stretch his/her hand to perform long distance 3D object translation BIB017 or spin the wrist when performing 3D object rotation. In touch-based interaction, the user needs to hold the handheld mobile device and free up one hand to perform finger touch actions on the display screen. When multi-touch actions are required such as for 3D object rotation, fatigue for fingers occur BIB012 . Also, interactions that utilize both hands can effectively reduce fatigue compared with single-hand interaction BIB002 . Since fatigue phenomenon may affect the user's AR experience that directly determines the effective interaction time in prolonging the activity period. It is one of the problems that require serious attention. Compared with touch-based and mid-air gestures-based interaction techniques, the device-based interaction technique category seems more robust in this context because the user can interact using both hands in handheld mobile AR environments. Prior knowledge is needed when the user uses multitouch actions to perform 3D object manipulation in handheld mobile AR interfaces. Twist, drag, unhinged dual fingers are some of the control structures in the touch-based interaction techniques. Besides the difficulty when using multi-touch actions on the touchscreen display, the user also needs to know in advance the control structure implemented for a certain touch-based interaction technique. However, the midair gestures-based interaction techniques have been proposed due to its intuitiveness and naturality. Users utilize his/her hand and fingers to manipulate 3D objects as he/she does in the real world thus no prior knowledge is needed. In the device-based interaction, handheld mobile devices have been treated as sticky/holding tools to translate or rotate 3D object simulating the real case in our daily life such as using our own hand/fingers to pick an object to move to another position or rotate it. In the real world, our own hand and fingers are the sticky/holding tools while in the device-based interaction technique category, it is changed to our handheld mobile device. Thus, both mid-air gesture-based and devicebased interaction techniques are intuitive and natural as stated in BIB023 , BIB019 , BIB008 , and BIB014 . Position mismatch and low precision problem are another two (2) aspects in our investigation. These two (2) problems exist in the mid-air gesture-based interaction techniques when independently compared with the touch-based and device-based interaction technique categories for 3D object manipulation. Because of the occlusion, the position deviation problem might occur although users thought that they move the 3D object to the exact position BIB004 . There is also low precision in mid-air gestures-based interaction as stated previously in BIB004 and BIB008 . Although inaccuracy may also be present in touch-based interaction when multi-touch inputs are used causing instability and failure of the AR marker's tracking stage BIB008 while objects near the screen corners and edges are difficult to be selected and manipulated BIB008 , solutions had been given by  to solve this problem through one-finger touch and drag separating DOF of 3D object translation and rotation to relief the difficulty of dual-finger action. Oppositely, device-based techniques have been proved more accurate and even faster compared with touch-based techniques when users perform 3D translation  , BIB015 . Even so, position deviation may occur when the same input set is used for both 3D object translation and rotation BIB022 , BIB007 , BIB005 ,  in touch-based and device-based interaction techniques, and this position deviation problem appears more in mid-air gestures-based interaction since it happens not only for 3D object manipulation (both translation and rotation performed integrally) but also for 3D object translation and rotation independently. For 3D fundamental tasks, 3D object translation and rotation are the basis. Both 3D object translation and rotation can be done through all the three main interaction techniques whereas users may feel difficult to perform largerange translation using the mid-air gestures-based interaction technique because of the restriction by the person's reach of hand BIB009 . Moreover, constraints towards large-range rotation also occur when using the mid-air gestures-based interaction techniques, limited by the movement around the wrist joint BIB009 , which also happens within device-based interaction BIB022 ,  , BIB015 because 3D object registration error occurs when the user rotates the handheld mobile device outside the viewable range or AR marker's trackable range. In this context, the touch-based interaction techniques seem to be more robust without limitation towards large-range 3D object translation and rotation.