In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulated depth images of a robotic hand. The final training set includes 400K pairwise depth images and joint angles of a Shadow C6 robotic hand. The network evaluation results verify the superiority of TeachNet, especially regarding the high-precision condition. Imitation experiments and grasp tasks teleoperated by novice users demonstrate that TeachNet is more reliable and faster than the state-of-the-art vision-based teleoperation method.
We present a comparative experimental evaluation of three methods that estimate the 3D position, orientation and articulation of the human body from markerless visual observations obtained by RGBD cameras. The evaluated methods are representatives of three broad 3D human pose estimation tracking methods. Specifically, the first is the discriminative approach adopted by OpenNI. The second is a hybrid approach that depends on the input of two synchronized and extrinsically calibrated RGBD cameras. Finally, the third one is a recently developed generative method that depends on input provided by a single RGBD camera. The experimental evaluation of these methods has been based on a publicly available data set that is annotated with ground truth. The obtained results expose the characteristics of the three methods and provide evidence that can guide the selection of the most appropriate one depending on the requirements of a certain application domain.