In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulated depth images of a robotic hand. The final training set includes 400K pairwise depth images and joint angles of a Shadow C6 robotic hand. The network evaluation results verify the superiority of TeachNet, especially regarding the high-precision condition. Imitation experiments and grasp tasks teleoperated by novice users demonstrate that TeachNet is more reliable and faster than the state-of-the-art vision-based teleoperation method.
Conventional teleoperation mapping methods are divided into three main categories: joint mapping which is useful for power grasps @cite_14 , fingertip mapping which is suitable for precision grasps @cite_18 , and pose mapping which interprets the function of the human grasp rather than replicating hand position @cite_15 . However, in most cases considering only one type of mapping method is not enough @cite_5 . For example, fingertip mapping neglects the position and orientation of the phalanges and does not consider the special mechanical difference between the slave and the master.
@cite_5 @cite_18 @cite_14 @cite_15
2785929605 1489279189 2963508807 2564823405
Human-in-the-loop manipulation is useful in when autonomous grasping is not able to deal sufficiently well with corner cases or cannot operate fast enough. Using the teleoperator's hand as an input device can provide an intuitive control method but requires mapping between pose spaces which may not be similar. We propose a low-dimensional and continuous teleoperation subspace which can be used as an intermediary for mapping between different hand pose spaces. We present an algorithm to project between pose space and teleoperation subspace. We use a non-anthropomorphic robot to experimentally prove that it is possible for teleoperation subspaces to effectively and intuitively enable teleoperation. In experiments, novice users completed pick and place tasks significantly faster using teleoperation subspace mapping than they did using state of the art teleoperation methods. We study the problem of human to robot grasp mapping as a basic building block of a learning by imitation system. The human hand posture, including both the grasp type and hand orientation, is first classified based on a single image and mapped to a specific robot hand. A metric for the evaluation based on the notion of virtual fingers is proposed. The first part of the experimental evaluation, performed in simulation, shows how the differences in the embodiment between human and robotic hand affect the grasp strategy. The second part, performed with a robotic system, demonstrates the feasibility of the proposed methodology in realistic applications. Most of the existing deep learning-based methods for 3D hand and human pose estimation from a single depth map are based on a common framework that takes a 2D depth map and directly regresses the 3D coordinates of keypoints, such as hand or human body joints, via 2D convolutional neural networks (CNNs). The first weakness of this approach is the presence of perspective distortion in the 2D depth map. While the depth map is intrinsically 3D data, many previous methods treat depth maps as 2D images that can distort the shape of the actual object through projection from 3D to 2D space. This compels the network to perform perspective distortion-invariant estimation. The second weakness of the conventional approach is that directly regressing 3D coordinates from a 2D image is a highly nonlinear mapping, which causes difficulty in the learning procedure. To overcome these weaknesses, we firstly cast the 3D hand and human pose estimation problem from a single depth map into a voxel-to-voxel prediction that uses a 3D voxelized grid and estimates the per-voxel likelihood for each keypoint. We design our model as a 3D CNN that provides accurate estimates while running in real-time. Our system outperforms previous methods in almost all publicly available 3D hand and human pose estimation datasets and placed first in the HANDS 2017 frame-based 3D hand pose estimation challenge. The code is available in1. Abstract This paper describes the development of a remote handling control of an anthropomorphic robotic hand, the SCHUNK S5FH, using the human hand as master by measuring its motion with OptiTrack Technology. The goal of this work is to enhance manipulation studies on the human hand and to instantly transfer those studies on robotic hands. A preliminary study on methods and devices used for fingers tracking led to the choice of a simplified kinematic model of the human hand on the basis of the available motion tracking system. Using the same criteria, the analysis of protocols for markers allocation led to define the number and a method for their arrangement on the fingers and palm. In order to overcome the limitation of the Motion Capture System, a method for identification and labeling has been developed according to their anatomical arrangement. Afterwards, the tracking is performed using the constraints between marker positions on the kinematic chain of the hand and a dynamic labeling algorithm robust with respect to noise, outliers and loss of markers. The validation is performed using the right hand of different subjects and considering different tasks involving flexion extension and abduction adduction of fingers and thumb opposition. For testing and validation, preliminary studies on synergies for manipulation tasks such as screwing a cup, has been conducted on the human hand and transferred on the robotic hand.