FIELD

Methods and apparatuses consistent with embodiments relate to configuring a neural network to estimate an in-grasp sliding velocity and characteristics of an object that is held and slid by a tool (e.g., a robot end effector, a robot gripper, etc.) based on tactile sensing data detected from the tool and an captured image of the object, and also relate to manipulating the tool based on the estimated sliding velocity and the estimated characteristics of the object.

BACKGROUND

In-hand manipulation may be easy for humans but still challenging for robots. In humans, tactile perception provides critical feedback for grasping and manipulation. As a result, tactile sensors have become increasingly popular in robotic grippers. Existing work extensively studies grasp stability assessment, slip detection, and contact force estimation. Beyond simple feedback, such as gripping more tightly in response to detecting slip, the existing work on using tactile measurements for in-hand manipulation remains limited.

In controlling a robot arm and a robot end effector, one of the important manipulation skills is in-grasp sliding—releasing a held object from a grip in a controlled manner. This capability is useful for gentle placement in cluttered environments. The presence of obstructing items may render infeasible direct placement via pick-and-place methods. When direct placement is infeasible, a robot may still accomplish a placement task by releasing the object from its grasp and gently sliding it into place at a specified velocity.

SUMMARY

One or more embodiments provide a method and an apparatus for controlling an in-grasp sliding velocity of an object that is held by a tool (e.g., a robot end effector, a robot gripper, etc.), by using a neural network configured to estimate the velocity of the object based on tactile sensing data detected from the tool.

According to an aspect of the disclosure, there is provided a tool control system including: a tactile sensor configured to, when a tool holds a target object and slides the target object downward across the tool, obtain tactile sensing data from the tool; one or more memories configured to store a target velocity and computer-readable instructions; and one or more processors configured execute the computer-readable instructions to: receive the tactile sensing data from the tactile sensor; estimate a velocity of the target object based on the tactile sensing data, by using one or more neural networks that are trained based on a training image of an sample object captured while the sample object is sliding down; and generate a control parameter of the tool based on the estimated velocity and the target velocity.

The tactile sensing data may include at least one of an impedance value of an electrode mounted on the tool, a static pressure value that represents a force exerted to the tool by the target object, and a dynamic pressure value that represents a vibration that occurs in the tool when the target object slides across the tool.

The tool may include two fingers configured to hold and slide the target object downward over the two fingers. The control parameter may be a command value indicating a distance between the two fingers of the tool.

The tactile sensing data may include an impedance value of an electrode mounted on the tool, a static pressure value that represents a force exerted to the tool by the target object, and a dynamic pressure value that represents a vibration that occurs in the tool when the target object slides across the tool. The one or more neural networks may include: a first neural network configured to process the impedance value; a second neural network configured to process the static pressure value; a third neural network configured to process the dynamic pressure value; a concatenation layer configured to combine the processed impedance value, the processed the static pressure value, and the processed dynamic pressure value; and one or more fully connected layers configured to output the estimated velocity based on a combination of the processed impedance value, the processed the static pressure value, and the processed dynamic pressure value.

The one or more neural networks may further include: a max pooling layer provided between the third neural network and the concatenation layer to down-sample the processed dynamic pressure value, and output a down-sampling result of the processed dynamic pressure value to the concatenation layer.

The tactile sensor may include: an impedance sensor including a plurality of sensing electrodes and configured to detect an impedance value of each of the plurality of sensing electrodes; and a pressure sensor configured to detect a static pressure value that represents a force exerted to the tool by the target object, and a dynamic pressure value that represents a vibration that occurs in the tool when the target object slides across the tool.

The one or more memories may be further configured to store an activation velocity that is less than the target velocity. The one or more processors may be further configured execute the computer-readable instructions to: generate the control parameter of the tool based on the estimated velocity and the activation velocity, in a pre-sliding stage starting from a first point in time where the estimated velocity of the target object is zero, and ending at a second point in time where the estimated velocity of the target object becomes a predetermined percentage of the activation velocity; and generate the control parameter of the tool based on the estimated velocity and the target velocity, in a sliding stage starting from the second point in time, and ending a third point in time where the estimated velocity of the target object becomes the target velocity.

The tool control system may further include a vison sensor configured to capture an image of the target object. The one or more processors may be further configured execute the computer-readable instructions to: estimate a physical property of the target object based on the captured image of the target object, by using another neural network that is trained based on either one or both of depth information of the sample object and tactile sensing information of the sample object; and generate the control parameter of the tool based on the estimated physical property of the target object, the estimated velocity of the target object, and the target velocity.

The physical property of the target object may include at least one of shape, texture, material, fragility, strength, and size of the target object.

The tool may include at least one of a robot end-effector, a robot gripper, and a robot tweezer.

The one or more processors may be further configured execute the computer-readable instructions to: transmit, to the tool, the control parameter that enables the tool to gradually increase a distance between two fingers of the tool based on the control parameter, until the estimated velocity becomes the target velocity.

According to an aspect of other embodiments, there is provided a tool control method including: obtaining tactile sensing data from a tool when the tool holds a target object and slides the target object downward across the tool; estimating a velocity of the target object based on the tactile sensing data, by using one or more neural networks that are trained based on a training image of an sample object captured while the sample object is sliding down; and generating a control parameter of the tool based on the estimated velocity and a predetermined target velocity.

The tactile sensing data may include at least one of an impedance value of an electrode mounted on the tool, a static pressure value that represents a force exerted to the tool by the target object, and a dynamic pressure value that represents a vibration that occurs in the tool when the target object slides across the tool.

The control parameter may be a command value indicating a distance between two fingers of the tool configured to hold and slide the target object downward.

The tactile sensing data may include at least one of an impedance value, a static pressure value, and a dynamic pressure value detected from the tool. The one or more networks may include a first neural network, a second neural network, and a third neural network. The estimating the velocity of the target object may include: processing the impedance value, the static pressure value, and the dynamic pressure value, through the first neural network, the second neural network, and the third neural network, respectively; and obtaining the estimated velocity by processing a combination of the processed impedance value, the processed the static pressure value, and the processed dynamic pressure value, through one or more fully connected layers.

The estimating the velocity of the target object may further include: down-sampling the processed dynamic pressure value to be combined with the processed impedance value and the processed static pressure value, so that the estimated velocity is obtained from the combination of the down-sampled processed dynamic pressure value, the processed impedance value, and the processed static pressure value.

The generating the control parameter of the tool may include: entering a pre-sliding stage at a first point in time where the estimated velocity of the target object is zero; generating the control parameter of the tool based on the estimated velocity and a predetermined activation velocity that is less than the predetermined target velocity, in the pre-sliding stage starting until a second point in time where the estimated velocity of the target object becomes the predetermined activation velocity; transitioning from the pre-sliding stage to a sliding stage at the second point time; and generating the control parameter of the tool based on the estimated velocity and the predetermined target velocity, in the sliding stage until the estimated velocity of the target object becomes the predetermined target velocity.

The tool control method may further include capturing an image of the target object; estimating a physical property of the target object based on the captured image of the target object, by using another neural network that is trained based on either one or both of depth information of the sample object and tactile sensing information of the sample object; and generating the control parameter of the tool based on the estimated physical property of the target object, the estimated velocity of the target object, and the predetermined target velocity.

The physical property of the target object may include at least one of shape, texture, material, fragility, strength, and size of the target object.

The tool control method may further include: transmitting, to the tool, the control parameter that enables the tool to gradually increase a distance between two fingers of the tool based on the control parameter, until the estimated velocity becomes the predetermined target velocity.

While the afore described methods, devices, and non-transitory computer-readable mediums have been described individually, these descriptions are not intended to suggest any limitation as to the scope of use or functionality thereof. Indeed these methods, devices, and non-transitory computer-readable mediums may be combined in other aspects of the disclosure.

