In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulated depth images of a robotic hand. The final training set includes 400K pairwise depth images and joint angles of a Shadow C6 robotic hand. The network evaluation results verify the superiority of TeachNet, especially regarding the high-precision condition. Imitation experiments and grasp tasks teleoperated by novice users demonstrate that TeachNet is more reliable and faster than the state-of-the-art vision-based teleoperation method.
Human teleoperation of robots has usually been implemented through contacting devices such as tracking sensors @cite_22 , gloves instrumented with angle sensors @cite_3 , inertial sensors @cite_10 and joysticks @cite_7 . @cite_2 suggest an end-to-end teleoperation on a 23 degree of freedom (DOF) robot by training a feed-forward neural network for each DOF of the robot to learn the mapping between sensor data from the motion capture suit and the angular position of the robot actuator to which each neural network is allocated. However, wearable devices are customized for a certain size range of the human hand or the human body, and contacting methods may hinder natural human-limb motion. Compared to these methods, markerless vision-based teleoperation is less invasive and performs natural and comfortable gestures.
@cite_22 @cite_7 @cite_3 @cite_2 @cite_10
2767539324 2169900819 2068006087 2588104504
Teleoperation is of great importance in the area of robotics especially when people’s presence at the robot working space is unavailable. It provides an alternative to employ human intelligence in the control of the robot remotely. We establish robotic teleoperation systems with a wearable multimodal fusion device. The device is integrated with 18 low-cost inertial and magnetic measurement units, which cover all segments of the arm and hand. The multimodal fusion algorithm based on extended Kalman filter is deduced to determine the orientations and positions of each segment. Then, the robotic teleoperation systems using the proposed device are designed. The novel teleoperation schemes can be applied for 11DOF robotic arm–hand system and 10DOF robotic arm–hand system, in which the operator’s fingers are used for robotic hand teleoperation, and the arms with palm are used for robotic arm teleoperation. Meanwhile, the proposed robotic teleoperation systems are fully realized with a user-friendly human–machin... Operators' intelligent and skillful decisions are necessary for the teleoperation of a mobile robot when there are many scattered obstacles. Among the sensors used for environment recognition, the camera is the most popular and powerful. However, there are several limitations in the camera-based teleoperation of a mobile robot. For example, shadowed and curved areas cannot be viewed using a narrow view-angle camera, especially in an environment with bad illumination and several obstacles. Therefore, it is necessary to have other sensory information for reliable teleoperations. In this study, 16 ultrasonic sensors are attached around a mobile robot in a ring pattern to measure the distances to the obstacles and a collision vector is introduced as a new tool for obstacle avoidance, which is defined as the normal vector from an obstacle to the mobile robot. Based on this collision vector, a virtual reflection force is generated to avoid the obstacles and then the reflection force is transferred to the operator who is holding the joystick used to control the mobile robot. Based on this reflection force, the operator can control the mobile robot more smoothly and safely. For this bidirectional teleoperation, a master joystick system using a two-axis hall sensor was designed to eliminate the nonlinear region, which exists in a general joystick with two motors and potentiometers. The effectiveness of the collision vector and force-reflection joystick is verified by comparing two vision-based teleoperation experiments, with and without force reflection. As robots begin to enter our homes and workplaces, they will have to deal with the devices and appliances that are already there. Unfortunately, devices that are easy for humans to operate often cause problems for robots [3]. In teleoperation settings, the lack of tactile feedback often makes manipulation of buttons and switches awkward and clumsy [7]. Also, the robot's gripper often occludes the control, making teleoperation difficult. In the autonomous setting, perception of small buttons and switches is often difficult due to sensor limitations and poor lighting conditions. Adding depth information does not help much, since many of the controls we want to manipulate are small, and often close to the noise threshold of currently-available depth sensors typically installed on a mobile robot. This makes it extremely difficult to segment the controls from the other parts of the device. In this paper, we present a shared autonomy approach to the operation of physical device controls. A human operator gives high-level guidance, helps identify controls and their locations, and sequences the actions of the robot. Autonomous software on our robot performs the lower-level actions that require closed-loop control, and estimates the exact positions and parameters of controls. We describe the overall system, and then give the results of our initial evaluations, which suggest that the system is effective in operating the controls on a physical device. Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with Brain---Computer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the operators capabilities and their perception of control authority. Additionally, a custom servo controller design allow for safe interactions of the robotic arm with the environment. We present experimental results demonstrating significant performance improvement using our shared-control assistance framework on adapted rehabilitation benchmarks with two subjects at various timepoints relative to their implantation with intracortical BCIs. Our results indicate that shared assistance mitigates perceived user difficulty in using a seven-degree of freedom robotic arm as a prosthetic and enables successful performance on previously infeasible tasks. We showcase the extensibility of our architecture with applications to quality-of-life tasks such as opening a door, pouring liquids from containers, and manipulation with objects previously unknown to the system in densely cluttered environments.