3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Mobile phones are an ideal platform for augmented reality. In this paper we describe how they also can be used to support face to face collaborative AR applications. We have created a custom port of the ARToolKit library to the Symbian mobile phone operating system and then developed a sample collaborative AR game based on this. We describe the game in detail and user feedback from people who have played it. We also provide general design guidelines that could be useful for others who are developing mobile phone collaborative AR applications. <s> BIB001 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> In this paper we describe a mobile phone based Augmented Reality application for 3D scene assembly. Augmented Reality on mobile phones extends the interaction capabilities on such handheld devices. It adds a 6 DOF isomorphic interaction technique for manipulating 3D content. We give details of an application that we believe to be the first where 3D content can be manipulated using both the movement of a camera tracked mobile phone and a traditional button interface as input for transformations. By centering the scene in a tangible marker space in front of the phone we provide a mean for bimanual interaction. We describe the implementation, the interaction techniques we have developed and initial user response to trying the application. <s> BIB002 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> A heat insulating box, such as a refrigerator box, comprises a heat insulator and a box member that is in contact with the heat insulator. The heat insulator is formed of a urethane foam using either HCFC-123 (CHCl2CF3) or HCFC-141b (CH3CCl2F) or both as a forming agent, and the box member is formed of an acrylonitrile/ethylene- PROPORTIONAL -olefinic rubbery polymer/styrene resin (A/epdm/S resin), an acrylonitrile/alkyl acrylate ester rubbery polymer/styrene resin (ASA resin), a mixture of an A/epdm/s resin and an ASA resin, or a mixture of an ASA resin with an acrylonitrile/butadiene/styrene resin. <s> BIB003 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Nowadays, handheld devices are capable of displaying augmented environments in which virtual content overlaps reality. To interact with these environments it is necessary to use a manipulation technique. The objective of a manipulation technique is to define how the input data modify the properties of the virtual objects. Current devices have multi-touch screens that can serve as input. Additionally, the position and rotation of the device can also be used as input creating both an opportunity and a design challenge. In this paper we compared three manipulation techniques which namely employ multi-touch, device position and a combination of both. A user evaluation on a docking task revealed that combining multi-touch and device movement yields the best task completion time and efficiency. Nevertheless, using only the device movement and orientation is more intuitive and performs worse only in large rotations. <s> BIB004 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Lately, mobile augmented reality (AR) has become very popular and is used for many commercial and product promotional activities. However, in almost all mobile AR applications, the user only views annotated information or the preset motion of the virtual object in an AR environment and is unable to interact with the virtual objects as if he/she were interacting with real objects in the real environment. In this paper, in an attempt to realize enhanced intuitive and realistic object manipulation in the mobile AR environment, we propose an integrated view-input AR interaction method, which integrates user device manipulation and virtual object manipulation. The method enables the user to hold a 3D virtual object by touching the displayed object on the 2D touch screen of a mobile device, and to move and rotate the object by moving and rotating the mobile device while viewing the held object by way of the 2D screen of the mobile device. Based on this concept, we implemented three types of integrated methods, namely the Rod, Center, and Touch methods, and conducted a user study to investigate the baseline performance metrics of the proposed method on an AR object manipulation task. The Rod method achieved the highest success rate (91%). Participants' feedback indicated that this is because the Rod method is the most natural, and evoked a fixed mental model that is conceivable in the real environment. These results indicated that visualizing the manipulation point on the screen and restricting the user's interactivity with virtual objects from the user's position of view based on a conceivable mental model would be able to aid the user to achieve precise manipulation. <s> BIB005 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Hand-held smart devices are equipped with powerful processing units, high resolution screens and cameras, that in combination makes them suitable for video see-through Augmented Reality. Many Augmented Reality applications require interaction, such as selection and 3D pose manipulation. One way to perform intuitive, high precision 3D pose manipulation is by direct or indirect mapping of device movement. There are two approaches to device movement interaction; one fixes the virtual object to the device, which therefore becomes the pivot point for the object, thus makes it difficult to rotate without translate. The second approach avoids latter issue by considering rotation and translation separately, relative to the object's center point. The result of this is that the object instead moves out of view for yaw and pitch rotations. In this paper we study these two techniques and compare them with a modification where user perspective rendering is used to solve the rotation issues. The study showed that the modification improves speed as well as both perceived control and intuitiveness among the subjects. <s> BIB006 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> A. 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR <s> Abstract Handheld Augmented Reality (HAR) has the potential to introduce Augmented Reality (AR) to large audiences due to the widespread use of suitable handheld devices. However, many of the current HAR systems are not considered very practical and they do not fully answer to the needs of the users. One of the challenging areas in HAR is the in-situ AR content creation where the correct and accurate positioning of virtual objects to the real world is fundamental. Due to the hardware limitations of handheld devices and possible restrictions in the environment, the correct 3D positioning of objects can be difficult to achieve we are unable to use AR markers or correctly map the 3D structure of the environment. We present SlidAR, a 3D positioning for Simultaneous Localization And Mapping (SLAM) based HAR systems. SlidAR utilizes 3D ray-casting and epipolar geometry for virtual object positioning. It does not require a perfect 3D reconstruction of the environment nor any virtual depth cues. We have conducted a user experiment to evaluate the efficiency of SlidAR method against an existing device-centric positioning method that we call HoldAR. Results showed that SlidAR was significantly faster, required significantly less device movement, and also got significantly better subjective evaluation from the test participants. SlidAR also had higher positioning accuracy, although not significantly. <s> BIB007
In early 2005, the device-based interaction technique was firstly implemented in handheld mobile AR interfaces as a 3D object manipulation tool BIB001 - BIB002 . For example, by using cell phones as tennis racquets in BIB001 as shown in Fig. 8a and using the cell phone to translate and rotate 3D objects in an augmented environment BIB003 , BIB002 as shown in Fig. 8b . Comparisons have been made towards traditional keypad and VOLUME 7, 2019 FIGURE 8. Using the cell phones as tennis racquets in AR tennis game and mapping the position of cell phone's built-in camera to translate and rotate a 3D object. (a) AR tennis BIB001 . (b) 3D object manipulation BIB003 , BIB002 . Arcball techniques with device-based techniques and results showed that the device-based interaction technique is effective in 3D object translation. These early researches become the basement for recent studies while the device-based interaction technique becomes a potential interaction solution for 3D object manipulation in handheld mobile AR due to its robustness such as the ability to hold the handheld mobile device with both hands and be free from occlusion BIB005 , BIB006 until several studies were carried out within the last few years BIB005 , BIB006 ,  , BIB004 . For examples in  and BIB004 , device-based interaction was used for 3D object manipulation (Fig. 9) in AR interfaces and results showed that it is a natural and intuitive interaction technique and can translate 3D objects faster than touchbased interaction although the user cannot complete largerange 3D rotation tasks. By mapping the position of the handheld mobile device's built-in camera with the 3D object registered on the AR marker, the user can use the handheld mobile device as a moving and rotating tool for manipulation. Since the same input (device movement) has been used in both 3D translation and rotation, position deviation occurs causing slow and less accurate positioning in performing 3D object manipulation tasks integrally. Unlike mid-air gestures-based techniques that require the calculation towards depth information (translation of z-axis for example), device-based techniques are highly suitable to be applied in handheld mobile AR because handheld mobile device movement in depth axis is mapped with the 3D object, when the device is lifted up, the 3D object also be lifted up by using device-based technique thus, additional calculation (on z-axis) that may exhaust the battery capacity can be reduced. Presently, Tanikawa et al. BIB005 applied the device-based interaction technique in AR Jenga game (shown in Fig. 10 ) while Samini and Palmerius BIB006 focusing on 3D object rotation issue that stated in BIB005 ,  , and BIB004 for devicebased interaction. In BIB006 , the hold function was proposed to assist the device-based interaction technique that utilizes the device's movement to move and rotate 3D objects in AR interfaces. By presenting the hold and release action, the user can lock the 3D object on different surfaces to complete largerange 3D object rotation (Fig. 11) . However, their solution still consists of several drawbacks such as the user might over-rotate the handheld mobile device causing 3D object registration error and AR marker tracking error when the camera sensor goes beyond the viewable and trackable range instead of slow rotation due to the frequent hold and release actions. Due to the effectiveness of the device-based interaction technique in 3D object translation, Polvi et al. BIB007 did a comparison on their 3D object positioning techniques named SlidAR that used ray casting assisted with epipolar geometry and the existing device-based positioning technique that utilized the device movement through mapping the built-in camera position with the 3D object. Although the results showed that SlidAR is faster than the device-based technique, but it is not applicable to 3D object rotation. Furthermore, touch input is needed in their technique, which is included in the different interaction technique categories. After the wide discussion about the device-based interaction, the important researches had been compressed in a systematic summarized table (Table 3) by keywords either it becomes the basement to AR research works, about 3D object manipulation, or some relevant issues. Comprehensively, those researches stated in Table 1, 2 and 3 and then had been filtered to refine the particular researches that consisting 3D object translation and rotation in handheld mobile AR and compressed in a systematic summarized table (Table 4) categorized on each of their key findings, limitations and strengths in chronological order based on different technique categories. 