In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulated depth images of a robotic hand. The final training set includes 400K pairwise depth images and joint angles of a Shadow C6 robotic hand. The network evaluation results verify the superiority of TeachNet, especially regarding the high-precision condition. Imitation experiments and grasp tasks teleoperated by novice users demonstrate that TeachNet is more reliable and faster than the state-of-the-art vision-based teleoperation method.
Conventional teleoperation mapping methods are divided into three main categories: joint mapping which is useful for power grasps @cite_14 , fingertip mapping which is suitable for precision grasps @cite_18 , and pose mapping which interprets the function of the human grasp rather than replicating hand position @cite_15 . However, in most cases considering only one type of mapping method is not enough @cite_5 . For example, fingertip mapping neglects the position and orientation of the phalanges and does not consider the special mechanical difference between the slave and the master.
@cite_5 @cite_18 @cite_14 @cite_15
2015028575 205184021 2012368703 2785929605
During past two decades many efforts have been made by different researchers in developing robotic grippers. Some of these grippers are robust and used for handling large objects. On the contrary, certain grippers are adroit enough even to handle biological cells. Wide varieties of grippers are now-a-days available featuring different kinematic ability, dexterity, mode of actuation, usage of sensors, maximum weight carrying capabilities and many more attributes. But they all accord to a single issue, i.e. inspiration. Essentially the goal of developing grippers focuses mainly on the manipulation ability of the human hand. Subsequently the designs have continuously become more and more complicated, which in turn have increased the programming complexity to keep abreast with the advances. Cognition in the field of robotics refers to sensing, generation and interpretation. To inculcate kinesthetic cognition to a robot hand unequivocally implies mapping. In this paper a hybrid methodology based on the existing grasp mapping algorithm has been proposed to increase the efficacy of the robotic hand. Keywords—Grasp mapping, forward kinematics, fuzzy clustering. An optimized fingertip mapping OFM algorithm has been developed to transform human hand poses into robot hand poses. It has been implemented to teleoperate the Utah MIT Dextrous Hand by a new hand master: the Utah Dextrous Hand Master. The keystone of the algorithm is the mapping of both the human fingertip positions and orientations to the robot fingers. Robot hand poses are generated by minimizing the errors between desired human fingertip positions and orientations and possible robot fingertip positions and orientations. Differences in the fingertip workspaces that arise from kinematic dissimilarities between the human and robot hands are accounted for by the use of a priority based mapping strategy. The OFM gives first priority to the human fingertip position goals and the second to orientation. The multi-fingered robot hand has much attention in various fields. Many robot hands have been proposed so far. However, the robot hand cannot execute any tasks autonomously because the robot hand does not have enough motion and sensing ability to task complexities. Therefore, we have developed a robot hand teleoperation system with the motion capture data glove CyberGlove. Here, each joint of the robot hand is controlled according to the corresponding joint of the human hand. In this paper, we develop the teleoperation method that the robot hand joint is controlled to the multiple human hand joints. The positional error of the fingertips can be decrease by using the multiple joints reference. We show the effectiveness of the developed method through some experiments. Human-in-the-loop manipulation is useful in when autonomous grasping is not able to deal sufficiently well with corner cases or cannot operate fast enough. Using the teleoperator's hand as an input device can provide an intuitive control method but requires mapping between pose spaces which may not be similar. We propose a low-dimensional and continuous teleoperation subspace which can be used as an intermediary for mapping between different hand pose spaces. We present an algorithm to project between pose space and teleoperation subspace. We use a non-anthropomorphic robot to experimentally prove that it is possible for teleoperation subspaces to effectively and intuitively enable teleoperation. In experiments, novice users completed pick and place tasks significantly faster using teleoperation subspace mapping than they did using state of the art teleoperation methods.