3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Computer sensing of hand and limb motion is an important problem for applications in human-computer interaction (HCI), virtual reality, and athletic performance measurement. Commercially available sensors are invasive, and require the user to wear gloves or targets. We have developed a noninvasive vision-based hand tracking system, called DigitEyes. Employing a kinematic hand model, the DigitEyes system has demonstrated tracking performance at speeds of up to 10 Hz, using line and point features extracted from gray scale images of unadorned, unmarked hands. We describe an application of our sensor to a 3D mouse user-interface problem. > <s> BIB001 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> This paper gives an overview of a system for robustly tracking the 3D position and orientation of a finger using a few closely spaced cameras. Accurate results are obtained by combining features of stereo range images and color images. This work also provides a design framework for combining multiple sources of information, including stereo range images, color segmentations, shape information and various constraints. This information is used in robust model fitting techniques to track highly over-constrained models of deformable objects: fingers. <s> BIB002 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Visual tracking of bare fingers allows more direct manipulation of digital objects, multiple simultaneous users interacting with their two hands, and permits the interaction on large surfaces, using only commodity hardware. After presenting related work, we detail our implementation. Its design is based on our modeling of two classes of algorithms that are key to the tracker: Image Differencing Segmentation (IDS) and Fast Rejection Filters (FRF). We introduce a new chromatic distance for IDS and a FRF that is independent to finger rotation. The system runs at full frame rate (25 Hz) with an average total system latency of 80 ms, independently of the number of tracked fingers. When used in a controlled environment such as a meeting room, its robustness is satisfying for everyday use. <s> BIB003 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Balloon selection is a 3D interaction technique that is modeled after the real world metaphor of manipulating a helium balloon attached to a string. Balloon selection allows for precise 3D selection in the volume above a tabletop surface by using multiple fingers on a multi-touch-sensitive surface. The 3DOF selection tasks is decomposed in part into a 2DOF positioning task performed by one finger on the tabletop in an absolute 2D Cartesian coordinate system and a 1DOF positioning task performed by another finger on the tabletop in a relative 2D polar coordinate system. We have evaluated balloon selection in a formal user study that compared it to two well-known interaction techniques for selecting a static 3D target: a 3DOF tracked wand and keyboard cursor keys. We found that balloon selection was significantly faster than using cursor keys and had a significantly lower error rate than the wand. The lower error rate appeared to result from the user's hands being supported by the tabletop surface, resulting in significantly reduced hand tremor and arm fatigue. <s> BIB004 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Although interactive surfaces have many unique and compelling qualities, the interactions they support are by their very nature bound to the display surface. In this paper we present a technique for users to seamlessly switch between interacting on the tabletop surface to above it. Our aim is to leverage the space above the surface in combination with the regular tabletop display to allow more intuitive manipulation of digital content in three-dimensions. Our goal is to design a technique that closely resembles the ways we manipulate physical objects in the real-world; conceptually, allowing virtual objects to be 'picked up' off the tabletop surface in order to manipulate their three dimensional position or orientation. We chart the evolution of this technique, implemented on two rear projection-vision tabletops. Both use special projection screen materials to allow sensing at significant depths beyond the display. Existing and new computer vision techniques are used to sense hand gestures and postures above the tabletop, which can be used alongside more familiar multi-touch interactions. Interacting above the surface in this way opens up many interesting challenges. In particular it breaks the direct interaction metaphor that most tabletops afford. We present a novel shadow-based technique to help alleviate this issue. We discuss the strengths and limitations of our technique based on our own observations and initial user feedback, and provide various insights from comparing, and contrasting, our tabletop implementations <s> BIB005 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Articulated hand-tracking systems have been widely used in virtual reality but are rarely deployed in consumer applications due to their price and complexity. In this paper, we propose an easy-to-use and inexpensive system that facilitates 3-D articulated user-input using the hands. Our approach uses a single camera to track a hand wearing an ordinary cloth glove that is imprinted with a custom pattern. The pattern is designed to simplify the pose estimation problem, allowing us to employ a nearest-neighbor approach to track hands at interactive rates. We describe several proof-of-concept applications enabled by our system that we hope will provide a foundation for new interactions in modeling, animation control and augmented reality. <s> BIB006 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Extending the tabletop display to the third dimension using a stereoscopic projection offers the possibility to improve applications by using the volume above the table surface. The combination of multi-touch input and stereoscopic projection usually requires an indirect technique to interact with objects above the tabletop, as touches can only be detected on the surface. Triangle Cursor is a 3D interaction technique that allows specification of a 3D position and yaw rotation above the interactive tabletop. It was designed to avoid occlusions that disturb the stereoscopic perception. While Triangle Cursor uses an indirect approach, the position, the height above the surface and the yaw rotation can be controlled simultaneously, resulting in a 4 DOF manipulation technique. We have evaluated Triangle Cursor in an initial user study and compared it to a related existing technique in a formal user study. Our experiments show that users were able to perform all tasks significantly faster with our technique without loosing any precision. Most of the subjects considered the technique easy to use and satisfying. <s> BIB007 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> The rising popularity of digital table surfaces has spawned considerable interest in new interaction techniques. Most interactions fall into one of two modalities: 1) direct touch and multi-touch (by hand and by tangibles) directly on the surface, and 2) hand gestures above the surface. The limitation is that these two modalities ignore the rich interaction space between them. To move beyond this limitation, we first contribute a unification of these discrete interaction modalities called the continuous interaction space. The idea is that many interaction techniques can be developed that go beyond these two modalities, where they can leverage the space between them. That is, we believe that the underlying system should treat the space on and above the surface as a continuum, where a person can use touch, gestures, and tangibles anywhere in the space and naturally move between them. Our second contribution illustrates this, where we introduce a variety of interaction categories that exploit the space between these modalities. For example, with our Extended Continuous Gestures category, a person can start an interaction with a direct touch and drag, then naturally lift off the surface and continue their drag with a hand gesture over the surface. For each interaction category, we implement an example (or use prior work) that illustrates how that technique can be applied. In summary, our primary contribution is to broaden the design space of interaction techniques for digital surfaces, where we populate the continuous interaction space both with concepts and examples that emerge from considering this space as a continuum. <s> BIB008 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Wearable haptic devices with poor position sensing are combined with the Kinect depth sensor by Microsoft. A heuristic hand tracker has been developed. It allows for the animation of the hand avatar in the virtual reality and the implementation of the force rendering algorithm: the position of the fingertips is measured by the hand tracker designed and optimized for Kinect, and the rendering algorithm computes the contact forces for wearable haptic display. Preliminary experiments with qualitative results show the effectiveness of the idea of combining Kinect and wearable haptics. <s> BIB009 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Our work introduces a semi-immersive environment for conceptual design where virtual mockups are obtained from gestures we aim to get closer to the way people conceive, create and manipulate three-dimensional shapes. We present on-and-above-the-surface interaction techniques following Guiard's asymmetric bimanual model to take advantage of the continuous interaction space for creating and editing 3D models in a stereoscopic environment. To allow for more expressive interactions, our approach continuously combines hand and finger tracking in the space above the table with multi-touch on its surface. This combination brings forth an alternative design environment where users can seamlessly switch between interacting on the surface or in the space above it depending on the task. Our approach integrates continuous space usage with bimanual interaction to provide an expressive set of 3D modeling operations. Preliminary trials with our experimental setup show this as a very promising avenue for further work. <s> BIB010 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Commercial 3D scene acquisition systems such as the Microsoft Kinect sensor can reduce the cost barrier of realizing mid-air interaction. However, since it can only sense hand position but not hand orientation robustly, current mid-air interaction methods for 3D virtual object manipulation often require contextual and mode switching to perform translation, rotation, and scaling, thus preventing natural continuous gestural interactions. A novel handle bar metaphor is proposed as an effective visual control metaphor between the user's hand gestures and the corresponding virtual object manipulation operations. It mimics a familiar situation of handling objects that are skewered with a bimanual handle bar. The use of relative 3D motion of the two hands to design the mid-air interaction allows us to provide precise controllability despite the Kinect sensor's low image resolution. A comprehensive repertoire of 3D manipulation operations is proposed to manipulate single objects, perform fast constrained rotation, and pack/align multiple objects along a line. Three user studies were devised to demonstrate the efficacy and intuitiveness of the proposed interaction techniques on different virtual manipulation scenarios. <s> BIB011 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Multitouch displays represent a promising technology for the display and manipulation of data. While the manipulation of 2D data has been widely explored, 3D manipulation with multitouch displays remains largely unexplored. Based on an analysis of the integration and separation of degrees of freedom, we propose a taxonomy for 3D manipulation techniques with multitouch displays. Using that taxonomy, we introduce Depth-Separated Screen-Space (DS3), a new 3D manipulation technique based on the separation of translation and rotation. In a controlled experiment, we compared DS3 with Sticky Tools and Screen-Space. Results show that separating the control of translation and rotation significantly affects performance for 3D manipulation, with DS3 performing faster than the two other techniques. <s> BIB012 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Hand gestures are intuitive ways to interact with a variety of user interfaces. We developed a real-time finger tracking technique using the Microsoft Kinect as an input device and compared its results with an existing technique that uses the K-curvature algorithm. Our technique calculates feature vectors based on Fourier descriptors of equidistant points chosen on the silhouette of the detected hand and uses template matching to find the best match. Our preliminary results show that our technique performed as well as an existing k-curvature algorithm based finger detection technique. <s> BIB013 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Stereoscopic tabletops offer unique visualization capabilities, enabling users to perceive virtual objects as if they were lying above the surface. While allowing virtual objects to coexist with user actions in the physical world, interaction with these virtual objects above the surface presents interesting challenges. In this paper, we aim to understand which approaches to 3D virtual object manipulations are suited to this scenario. To this end, we implemented five different techniques based on the literature. Four are mid-air techniques, while the remainder relies on multi-touch gestures, which act as a baseline. Our setup combines affordable non-intrusive tracking technologies with a multi-touch stereo tabletop, providing head and hands tracking, to improve both depth perception and seamless interactions above the table. We conducted a user evaluation to find out which technique appealed most to participants. Results suggest that mid-air interactions, combining direct manipulation with six degrees of freedom for the dominant hand, are both more satisfying and efficient than the alternatives tested. <s> BIB014 </s> 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review <s> IV. MID-AIR GESTURES-BASED INTERACTION <s> Virtual Reality environments are able to offer natural interaction metaphors. However, it is difficult to accurately place virtual objects in the desired position and orientation using gestures in mid-air. Previous research concluded that the separation of degrees-of-freedom (DOF) can lead to better results, but these benefits come with an increase in time when performing complex tasks, due to the additional number of transformations required. In this work, we assess whether custom transformation axes can be used to achieve the accuracy of DOF separation without sacrificing completion time. For this, we developed a new manipulation technique, MAiOR, which offers translation and rotation separation, supporting both 3-DOF and 1-DOF manipulations, using personalized axes for the latter. Additionally, it also has direct 6-DOF manipulation for coarse transformations, and scaled object translation for increased placement. We compared MAiOR against an exclusively 6-DOF approach and a widget-based approach with explicit DOF separation. Results show that, contrary to previous research suggestions, single DOF manipulations are not appealing to users. Instead, users favored 3-DOF manipulations above all, while keeping translation and rotation independent. <s> BIB015
Mid-air gestures-based interaction is powerful in providing a more natural and intuitive interaction mechanism for the user. One of the earlier researches that investigates the potential to model human hands and limbs for interaction purpose was conducted by Rehg and Kanade BIB001 . In their study, human hands and limbs were modeled as articulated mechanisms and formulated as a real-time visual tracking called DigitEyes. Since their modeling approach was based on 3D kinematics, this made it possible to track any subset of hand or body states with the same basic algorithm and it was possible to track in total 27 DOF of hand model from a two-camera image sequence under orthographic projection. However, DigitEyes could not be applied in complicated backgrounds and suffered an occlusion problem. Jennings BIB002 is another researcher focusing on mid-air gestures tracking system. Rehg and Kanade BIB001 focused on finger tracking. He combined the stereo-range images, color segmentation and shape information to track the highly overconstrained models of fingers assisted with multiple enclosed cameras. It can deal with complex backgrounds and can recover quickly when tracking is lost in the real-time system. However, the combination of too much information leads to slow real-time response. His research and then is followed by Letessier and Bérard BIB003 that focused on bare-finger tracking, claiming that their system can provide a more robust interaction that only requires a low standard camera and without a large interactive surface through image differencing segmentation and novel shape filtering algorithm although their works were limited within controlled environments and had not yet been studied in providing more information such as finger orientations for more complicated tasks especially for 3D object manipulation. There had been a wide exploration of hand pose tracking and estimation researches until Erol et al.  did a solid analysis of the existing studies. Based on their study, hand and finger motions can be captured based on two (2) types of system; a) data glove sensing system and b) computer vision-based sensing system. Computer vision-based sensing has become a promising alternative to data glove sensing due to its naturality, unencumbered and non-contact interaction. In their study, they focused on the difficulties of the vision-based hand pose estimation system which consists of the high-dimensional problem, self-occlusions, processing speed, uncontrolled environments, and rapid hand motion. They also studied the hand modeling-related issues and concluded that vision-based hand tracking system can achieve robustness, accuracy and high processing speed. Although there have been many researches relating to hand and finger tracking, but studies that focus on 3D object selection and manipulation can be found in Benko and Feiner BIB004 , which is an earlier research relating to 3D object selection. In their study, a mid-air multi-finger technique called balloon selection was proposed to perform 3D object selection. By closing two hands, the user could move the balloon up while moving two hands apart bringing the balloon downwards on a stereoscopic tabletop. While Hilliges et al. BIB005 explored the possibility to perform 3D object manipulation on interactive tabletops. In their study, two prototypes were proposed with different setups, whereby the first prototype was set up without additional hardware by applying shadow mapping, using the shadows to provide the partial depth information and allow the user to perform 4DOF manipulation while another prototype assisted by an in-depth camera called ZSense camera enables higher DOFs. Both techniques represented in virtual environments on tabletop display require new hardware assistance which might cause difficulty on both compatibility and cost-effectiveness BIB005 . Strothoff et al. BIB007 then proposed another potential midair gestures-based interaction technique that uses a triangle cursor to allow the specification of a 3D object position and yaw rotation above the interactive tabletop. The user can use the thumb and index finger in mid-air to control the triangle cursor for 3D object manipulation tasks and move their hand up and down to translate the 3D object. Due to the potential of mid-air gestures-based interaction in providing an intuitive and natural interaction, the concept to use sensing glove was proposed by Wong and Popovic BIB006 . Color gloves had been used in BIB006 to ease hand pose estimation and tracking in order to provide a foundation in developing new interaction techniques in several platforms such as AR and animation modeling for 3D object manipulation purpose, whereas the technique called the continuous interaction space BIB008 combined the multi-touch function with mid-air gestures-based tracking to enable 6DOF 3D object manipulation although it is still constrained by the person's reach of the hand, and the rotation is limited by the movement around the wrist joint. Afterward, the Kinect depth sensor by Microsoft was launched in November 2010. Due to its robustness to provide a heuristic hand tracker, quite a large number of researches have been carried out based on Kinect sensor. For example, Frati and Prattichizzo BIB009 suggested to combine wearable haptic devices with Kinect to solve the poor position sensing issue faced by the usage of wearable haptic devices while Kulshreshth et al. BIB013 proposed a Kinect-based finger detection technique that focused on the finger pose recognition and claimed that their technique was more accurate in tracking the number of fingers raised up compared with the existing technique that uses the K-curvature algorithm. Besides, Araùjo et al. BIB010 also introduced a 3D modeling technique above tabletops supported with a Kinect tracker named Mockup builder by using both hands supported with several widgets to perform 3D object manipulation. Also, Song et al. BIB011 proposed the handler bar metaphor to perform 3D object manipulation. With the support of the Kinect sensor, users can perform 3D object translation and rotation in mid-air with both hands. Since there had been many different mid-air interaction techniques proposed previously BIB004 , BIB007 , BIB010 , BIB014 , comparisons were made by Mendes et al. BIB014 . For example, to compare four mid-air interaction techniques with one touch-based interaction technique as the baseline (Fig. 5) . Results showed that self-occlusion is still the main issue in mid-air gestures-based interaction while the handler bar technique BIB011 can spatially solve this problem. The quality of depth camera used might also affect the hand tracking accuracy while new generations of hardware such as Kinect for examples can be explored in this topic. Presently, Mendes et al. BIB015 again conducted a study to separate the DOF of the mid-air interaction technique due to the results gained by Martinet et al. BIB012 showing that separating DOF can increase the performance either accuracy or task completion time. They suggested to use custom transformation axes and developed MAiOR system to evaluate their works. However, their results showed that DOF separation was still weak in achieving high precision when performing complex tasks compared with touch-based interaction.