Aerial robots are becoming popular among general public, and with the development of artificial intelligence (AI), there is a trend to equip aerial robots with a natural user interface (NUI). Hand arm gestures are an intuitive way to communicate for humans, and various research works have focused on controlling an aerial robot with natural gestures. However, the techniques in this area are still far from mature. Many issues in this area have been poorly addressed, such as the principles of choosing gestures from the design point of view, hardware requirements from an economic point of view, considerations of data availability, and algorithm complexity from a practical perspective. Our work focuses on building an economical monocular system particularly designed for gesture-based piloting of an aerial robot. Natural arm gestures are mapped to rich target directions and convenient fine adjustment is achieved. Practical piloting scenarios, hardware cost and algorithm applicability are jointly considered in our system design. The entire system is successfully implemented in an aerial robot and various properties of the system are tested.
There are many inspiring algorithms solely concerning vision-based gesture recognition @cite_7 @cite_21 @cite_13 @cite_16 @cite_5 , and various works have addressed the human-machine interaction issue @cite_27 @cite_24 @cite_0 @cite_11 @cite_25 @cite_22 @cite_9 @cite_15 @cite_12 @cite_19 @cite_26 @cite_18 @cite_14 @cite_10 @cite_23 . The design philosophies, however, are usually just generally mentioned. In @cite_18 , W.S. Ng and E. Sharlin particularly emphasizes that their design is inspired by falconeering, and it is evaluated by users. Kevin P. in @cite_9 explore multiple sets of upper body 3D interaction metaphors, and the evaluation by users shows a preference toward intuitive and easy gestures. Our command design aims at recognizing simple and intuitive gestures, and mapping them to as many piloting commands (i.e. target directions) as possible.
@cite_22 @cite_5 @cite_15 @cite_10 @cite_18 @cite_21 @cite_23 @cite_26 @cite_7 @cite_19 @cite_27 @cite_16 @cite_25 @cite_12 @cite_14 @cite_9 @cite_24 @cite_0 @cite_13 @cite_11
2024210451 2142515858 2018369130 1568156700 1624618595 2142729340 1966086309 2128220885 2060280062 2118706391 2055341873 2102486715 1881418671 1584239336 1969595749 2134654371
This paper presents a real-time and robust approach to recognize two types of gestures consisting of seven motional gestures and six finger spelling gestures. This approach utilizes stereo images captured by a stereo webcam to achieve robust recognition under realistic lighting conditions and in various backgrounds. It incorporates several existing computationally efficient techniques and introduces a rule-based approach to merge the information from a pair of stereo images leading to an improved hand detection compared to using single images. The results obtained indicate that high recognition rates under realistic conditions are obtained in real-time on PC platforms at the rate of 30 frames per second. It is shown that its outcome is comparable to two existing approaches while it is computationally more efficient than these approaches. The evolution of input device technologies led to identification of the natural user interface (NUI) as the clear evolution of the human-machine interaction, following the shift from command-line interfaces (CLI) to graphical user interfaces (GUI). The design of user interfaces requires a careful mapping of complex user “actions” in order to make the human-computer interaction (HCI) more intuitive, usable, and receptive to the user’s needs: in other words, more user-friendly and, why not, fun. NUIs constitute a direct expression of mental concepts and the naturalness and variety of gestures, compared with traditional interaction paradigms, can offer unique opportunities also for new and attracting forms of human-machine interaction. In this paper, a kinect-based NUI is presented; in particular, the proposed NUI is used to control the Ar.Drone quadrotor. In this paper, we present an approach that allows a quadrocopter to follow a person and to recognize simple gestures using an onboard depth camera. This enables novel applications such as hands-free filming and picture taking. The problem of tracking a person with an onboard camera however is highly challenging due to the self-motion of the platform. To overcome this problem, we stabilize the depth image by warping it to a virtual-static camera, using the estimated pose of the quadrocopter obtained from vision and inertial sensors using an Extended Kalman filter. We show that such a stabilized depth video is well suited to use with existing person trackers such as the OpenNI tracker. Using this approach, the quadrocopter not only obtains the position and orientation of the tracked person, but also the full body pose - which can then for example be used to recognize hand gestures to control the quadrocopter's behaviour. We implemented a small set of example commands (“follow me”, “take picture”, “land”), and generate corresponding motion commands. We demonstrate the practical performance of our approach in an extensive set of experiments with a quadrocopter. Although our current system is limited to indoor environments and small motions due to the restrictions of the used depth sensor, it indicates that there is large potential for such applications in the near future. Recent advances in depth sensing provide exciting opportunities for the development of new methods for human activity understanding. Yet, little work has been done in the area of hand gesture recognition which has many practical applications. In this paper we propose a real-time system for dynamic hand gesture recognition. It is fully automatic and robust to variations in speed and style as well as in hand orientations. Our approach is based on action graph, which shares similar robust properties with standard HMM but requires less training data by allowing states shared among different gestures. To deal with hand orientations, we have developed a new technique for hand segmentation and orientation normalization. The proposed system is evaluated on a challenging dataset of twelve dynamic American Sign Language (ASL) gestures. This paper introduces a pantomimic gesture interface, which classifies human hand gestures using unmanned aerial vehicle (UAV) behavior recordings as training data. We argue that pantomimic gestures are more intuitive than iconic gestures and show that a pantomimic gesture recognition strategy using micro-UAV behavior recordings can be more robust than one trained directly using hand gestures. Hand gestures are isolated by applying a maximum information criterion, with features extracted using principal component analysis and compared using a nearest neighbor classifier. These features are biased in that they are better suited to classifying certain behaviors. We show how a Bayesian update step accounting for the geometry of training features compensates for this, resulting in fairer classification results, and introduce a weighted voting system to aid in sequence labeling. Real-time gesture recognition is mandatory for gesture based human-robot natural interaction. Several glove-based approaches are introduced for this purpose, but they are un-natural and force user to wear some cumbersome glove. This paper presents a vision-based gesture recognition method for human-robot interface. Hand poses and face are detected using multiple feature-based template matching techniques from the three largest skin-like regions that are segmented using skin color information from YIQ color space. Gesture is recognized using rule-based approach whenever the three skin-like regions at a particular image frame match with the predefined gesture. Gesture command is sent to robot through a knowledge based software platform for human-robot interface. In this paper, a method has also been proposed to detect left hand and right hand relative to face position as well as, to detect and locate its position. The effectiveness of the proposed method has been demonstrated over the interaction with a pet robot named AIBO In this paper, we propose a real-time hand gesture recognition system using a thinning method utilizing a stereo camera. We implement a depth map in the hand detection portion that uses a sum of absolute differences method based on the acquired right-left image to detect the foreground object. We use a convex hull to detect the region of interests (ROI) and calculate the depth of object in ROI to obtain hand images that are more accurate. Then, we remove the background image in ROI and get the foreground image as a hand image. Finally, we use a blob labeling method to obtain the clean hand image, without the noise caused by computing distance from stereo matching. The hand gesture recognition system uses the Zhang and Suen thinning algorithm to obtain the feature point, angle and distance. It recognizes five kinds of hand gestures. The proposed method achieves an average recognition rate of 82.93 . We present our approach for visual tracking of head, hands and head orientation. Given the images provided by a calibrated stereo-camera, color and disparity information are integrated into a multi-hypotheses tracking framework in order to find the 3D-positions of the respective body parts. Based on the hands' motion, an HMM-based approach is applied to recognize pointing gestures. We show experimentally, that the gesture recognition performance can be improved significantly by using visually gained information about head orientation as an additional feature. Our system aims at applications in the field of human-robot interaction, where it is important to do run-on recognition in real-time, to allow for robot's egomotion and not to rely on manual initialization. We propose a new method to quickly and accurately predict human pose---the 3D positions of body joints---from a single depth image, without depending on information from preceding frames. Our approach is strongly rooted in current object recognition strategies. By designing an intermediate representation in terms of body parts, the difficult pose estimation problem is transformed into a simpler per-pixel classification problem, for which efficient machine learning techniques exist. By using computer graphics to synthesize a very large dataset of training image pairs, one can train a classifier that estimates body part labels from test images invariant to pose, body shape, clothing, and other irrelevances. Finally, we generate confidence-scored 3D proposals of several body joints by reprojecting the classification result and finding local modes. The system runs in under 5ms on the Xbox 360. Our evaluation shows high accuracy on both synthetic and real test sets, and investigates the effect of several training parameters. We achieve state-of-the-art accuracy in our comparison with related work and demonstrate improved generalization over exact whole-skeleton nearest neighbor matching. This paper presents a new visual gesture recognition method for the human-machine interface of mobile robot teleoperation. The interface uses seven static hand gestures, each of which represents an individual control command for the motion control of the remote robot. All the important aspects to develop such an interface are explored, including image acquisition, adaptive object segmentation with color image in RGB, HLS representation, morphological filtering, hand finding and labeling, and recognition with edge codes, template matching, and skeletonizing. By choosing processing methods and procedures properly, a higher ratio of correct recognition and a faster speed are achieved from the experiments. We propose an unmanned aerial vehicle (UAV) control mechanism, called a "Flying Head" which synchronizes a human head and the UAV motions. The accurate manipulation of UAVs is difficult as their control typically involves hand-operated devices. We can incorporate the UAV control using human motions such as walking, looking around and crouching. The system synchronizes the operator and UAV positions in terms of the horizontal and vertical positions and the yaw orientation. The operator can use the UAV more intuitively as such manipulations are more in accord with kinesthetic. Finally, we discuss flying telepresence applications. We present a study exploring upper body 3D spatial interaction metaphors for control and communication with Unmanned Aerial Vehicles (UAV) such as the Parrot AR Drone. We discuss the design and implementation of five interaction techniques using the Microsoft Kinect, based on metaphors inspired by UAVs, to support a variety of flying operations a UAV can perform. Techniques include a first-person interaction metaphor where a user takes a pose like a winged aircraft, a game controller metaphor, where a user's hands mimic the control movements of console joysticks, "proxy" manipulation, where the user imagines manipulating the UAV as if it were in their grasp, and a pointing metaphor in which the user assumes the identity of a monarch and commands the UAV as such. We examine qualitative metrics such as perceived intuition, usability and satisfaction, among others. Our results indicate that novice users appreciate certain 3D spatial techniques over the smartphone application bundled with the AR Drone. We also discuss the trade-offs in the technique design metrics based on results from our study. Since a variety of changes in both robotic hardware and software suggests that service robots will soon become possible, to find "natural" ways of communication between human and robots is of fundamental importance for the robotic field. The paper describes a gesture-based interface for human-robot interaction, which enables people to instruct robots through easy-to-perform arm gestures. Such gestures might be static pose gestures, which involve only a specific configuration of the person's arm, or they might be dynamic motion gestures, that is, they involve motion (such as waving). Gestures are recognized in real-time at approximate frame rate, using neural networks. A fast, color-based tracking algorithm enables the robot to track and follow a person reliably through office environments with drastically changing lighting conditions. Results are reported in the context of an interactive clean-up task, where a person guides the robot to specific locations that need to be cleaned, and the robot picks up trash which it then delivers to the nearest trash-bin. Gestures provide a rich and intuitive form of interaction for controlling robots. This paper presents an approach for controlling a mobile robot with hand gestures. The system uses hidden Markov models (HMMs) to spot and recognize gestures captured with a data glove. To spot gestures from a sequence of hand positions that may include nongestures, we have introduced a "wait state" in the HMM. The system is currently capable of spotting six gestures reliably. These gestures are mapped to robot commands under two different modes of operation: local and global control. In the local control mode, the gestures are interpreted in the robot's local frame of reference, allowing the user to accelerate, decelerate, and turn. In the global control mode, the gestures are interpreted in the world frame, allowing the robot to move to the location at which the user is pointing. The recently developed depth sensors, e.g., the Kinect sensor, have provided new opportunities for human-computer interaction (HCI). Although great progress has been made by leveraging the Kinect sensor, e.g., in human body tracking, face recognition and human action recognition, robust hand gesture recognition remains an open problem. Compared to the entire human body, the hand is a smaller object with more complex articulations and more easily affected by segmentation errors. It is thus a very challenging problem to recognize hand gestures. This paper focuses on building a robust part-based hand gesture recognition system using Kinect sensor. To handle the noisy hand shapes obtained from the Kinect sensor, we propose a novel distance metric, Finger-Earth Mover's Distance (FEMD), to measure the dissimilarity between hand shapes. As it only matches the finger parts while not the whole hand, it can better distinguish the hand gestures of slight differences. The extensive experiments demonstrate that our hand gesture recognition system is accurate (a 93.2 mean accuracy on a challenging 10-gesture dataset), efficient (average 0.0750 s per frame), robust to hand articulations, distortions and orientation or scale changes, and can work in uncontrolled environments (cluttered backgrounds and lighting conditions). The superiority of our system is further demonstrated in two real-life HCI applications. We present a solution to repeatability assessment of an arm gesture recognition system using multiple orientation sensors. We focus specifically on the problem of controlling unmanned aerial vehicles (UAVs) in the presence of manned aircrafts on an aircraft deck. Our goal is to design a robust UAV control with the same gesture signals as used by current flight directors for controlling manned vehicles. Given the fact that such a system has to operate 24 hours a day in a noisy and harsh environment, for example, on a Navy carrier deck, our approach to this problem is based on arm gesture recognition rather than on speech recognition. We have investigated real-time and system design issues for a particular choice of active sensors, such as, the orientation sensors of the IS-300 Pro Precision Motion Tracker manufactured by InterSense. Our work consists of (1) scrutinizing sensor data acquisition parameters and reported arm orientation measurements, (2) choosing the optimal attachment and placement of sensors, (3) measuring repeatability of movements using dynamic time warping (DTW) metric, and (4) testing performance of a template-based gesture classification algorithm and robot control mechanisms, where the robot represents an UAV surrogate in a laboratory environment.