In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulated depth images of a robotic hand. The final training set includes 400K pairwise depth images and joint angles of a Shadow C6 robotic hand. The network evaluation results verify the superiority of TeachNet, especially regarding the high-precision condition. Imitation experiments and grasp tasks teleoperated by novice users demonstrate that TeachNet is more reliable and faster than the state-of-the-art vision-based teleoperation method.
Abstract This paper describes the development of a remote handling control of an anthropomorphic robotic hand, the SCHUNK S5FH, using the human hand as master by measuring its motion with OptiTrack Technology. The goal of this work is to enhance manipulation studies on the human hand and to instantly transfer those studies on robotic hands. A preliminary study on methods and devices used for fingers tracking led to the choice of a simplified kinematic model of the human hand on the basis of the available motion tracking system. Using the same criteria, the analysis of protocols for markers allocation led to define the number and a method for their arrangement on the fingers and palm. In order to overcome the limitation of the Motion Capture System, a method for identification and labeling has been developed according to their anatomical arrangement. Afterwards, the tracking is performed using the constraints between marker positions on the kinematic chain of the hand and a dynamic labeling algorithm robust with respect to noise, outliers and loss of markers. The validation is performed using the right hand of different subjects and considering different tasks involving flexion extension and abduction adduction of fingers and thumb opposition. For testing and validation, preliminary studies on synergies for manipulation tasks such as screwing a cup, has been conducted on the human hand and transferred on the robotic hand. Operators' intelligent and skillful decisions are necessary for the teleoperation of a mobile robot when there are many scattered obstacles. Among the sensors used for environment recognition, the camera is the most popular and powerful. However, there are several limitations in the camera-based teleoperation of a mobile robot. For example, shadowed and curved areas cannot be viewed using a narrow view-angle camera, especially in an environment with bad illumination and several obstacles. Therefore, it is necessary to have other sensory information for reliable teleoperations. In this study, 16 ultrasonic sensors are attached around a mobile robot in a ring pattern to measure the distances to the obstacles and a collision vector is introduced as a new tool for obstacle avoidance, which is defined as the normal vector from an obstacle to the mobile robot. Based on this collision vector, a virtual reflection force is generated to avoid the obstacles and then the reflection force is transferred to the operator who is holding the joystick used to control the mobile robot. Based on this reflection force, the operator can control the mobile robot more smoothly and safely. For this bidirectional teleoperation, a master joystick system using a two-axis hall sensor was designed to eliminate the nonlinear region, which exists in a general joystick with two motors and potentiometers. The effectiveness of the collision vector and force-reflection joystick is verified by comparing two vision-based teleoperation experiments, with and without force reflection. Teleoperation is of great importance in the area of robotics especially when people’s presence at the robot working space is unavailable. It provides an alternative to employ human intelligence in the control of the robot remotely. We establish robotic teleoperation systems with a wearable multimodal fusion device. The device is integrated with 18 low-cost inertial and magnetic measurement units, which cover all segments of the arm and hand. The multimodal fusion algorithm based on extended Kalman filter is deduced to determine the orientations and positions of each segment. Then, the robotic teleoperation systems using the proposed device are designed. The novel teleoperation schemes can be applied for 11DOF robotic arm–hand system and 10DOF robotic arm–hand system, in which the operator’s fingers are used for robotic hand teleoperation, and the arms with palm are used for robotic arm teleoperation. Meanwhile, the proposed robotic teleoperation systems are fully realized with a user-friendly human–machin... Programming by demonstration, also called "imitation learning," offers the possibility of flexible, easily modifiable robotic systems. Full-fledged robotic imitation learning comprises many difficult subtasks. However, we argue that, at its core, imitation learning reduces to a regression problem. We propose a two-step framework in which an imitating agent first performs a regression from a high-dimensional observation space to a low-dimensional latent variable space. In the second step, the agent performs a regression from the latent variable space to a high-dimensional space representing degrees of freedom of its motor system. We demonstrate the validity of the approach by learning to map motion capture data from human actors to a humanoid robot. We also contrast use of several low-dimensional latent variable spaces, each covering a subset of agents' degrees of freedom, with use of a single, higher-dimensional latent variable space. Our findings suggest that compositing several regression models together yields qualitatively better imitation results than using a single, more complex regression model We describe the design of a modular system for untethered real-time kinematic motion capture using sensors with inertial measuring units (IMU). Our system is comprised of a set of small and lightweight sensors. Each sensor provides its own global orientation (3 degrees of freedom) and is physically and computationally independent, requiring only external communication. Orientation information from sensors is communicated via wireless to host computer for processing. We present results of the real-time usage of our untethered motion capture system for teleoperating the NASA Robonaut. We also discuss potential applications for untethered motion capture with respect to humanoid robotics.