Computer Science
Computer Science-2021.gz:143259
Computer Science
5e54096d311476d58d34334f4bffaac3c8c8793d
Robust Regrasping against Error of Grasping for Bin-picking and Kitting

—This study proposes a method of robust regrasping an object using a dual-arm robot with general-purpose hands, which is robust against the error of grasping. In this paper, one arm is assigned to hand over the object to the other arm that is named a receiver arm. The grasping error must be considered to increase the success rate of the regrasping since a hand-over arm ﬁrst picks up the object with the general- purpose hand. In an online phase, the proposed method performs object positioning at an optimal pose at the time of regrasping using an image-based visual servoing (IBVS) approach to reduce the effect of the grasping error. In the planning phase, the proposed method computes the optimal pose for regrasping by maximizing the minimum singular values of the image Jacobian of IBVS to achieve a high positioning accuracy using a 3D model of the target object. To achieve the regrasping objects with various shapes robustly against image noises and changes in light environments, the image Jacobian of IBVS is computed by numerical differential using an actual data set. A large number of data sets corresponding to each candidate grasp are usually required for computing the image Jacobian. To reduce the number of data sets, we propose a conversion method of the image Jacobian requiring only one data set corresponding to one representative grasp. The experimental results show that the proposed method achieves regrasping of target objects with the general-purpose hands with 100% success rates and performs target object positioning with less than 0.7[mm] positioning error.


I. INTRODUCTION
M ANY types of dual-arm robots, such as NEXTAGE [1], duAro [2], and YuMi [3], have been developed, and the use of the dual-arm robots has been expanded for industrial applications. Compared to a single-arm robot, the dual-arm robots can perform various operations done by workers in a smaller space. This study focuses on "regrasping objects" in picking and placement tasks. The motivation why we focus on it is that the regrasping objects in the picking and placement tasks include important and essential technologies for robotics, such as 3D measurement [4], [5], random bin piking [6]- [8], grasping [9], [10], positioning [11], [12], and assembling industrial parts [13].
As shown in Fig. 1, we consider one of the most conventional strategies for regrasping a target object in the picking and placement tasks. In this strategy, we first need to prepare fixtures and robot hands that are specially designed for the target object. The relative poses of the target object to the fixture and the robot hand are uniquely determined by these tools. Next, a specially trained human operator performs teachings for the motion of the robot. In the teachings, the operator manually controls the pose of the robot and its hand and finds movement to achieve object picking, regrasping, and placing. In addition, all robot movements are recorded by measuring the joint velocity with encoders during the teaching. After the teaching phase, the robot can automatically execute the picking, regrasping, and placing the object by playbacking the recorded data. Thus, this approach takes a lot of time and costs for changing the target object because special tools, such as the fixtures and the robot hands, need to be changed according to the shape of the target object. Furthermore, the teaching also needs to be performed again.
To solve these problems, we consider herein the regrasping conducted by general-purpose hands instead of the specially designed hands and any fixtures. Recently a lot of generalpurpose hands are proposed and developed to grasp various shapes of objects [14]- [17]. In more detail, we assume the situation illustrated in Fig. 2, where the target objects are randomly piled up; a three-dimensional (3D) sensor measures a profile of the piled-up objects, pose estimation of the objects is performed; one object is picked up by the general-purpose Hand A based on the estimated pose; then, regrasping is done by general-purpose hand B. In this situation, we have to consider the grasping error of Hand A. The grasping error is a deviation between the pre-assumed and actual relative poses of the object to the robot hand. The grasping error occurs because of various factors, such as the error of the 3D measurement and pose estimation, the calibration error for relative pose between a 3D sensor and the robot arm, and the deviation of the object during the picking and transfer.
To perform the regrasping robustly against the grasping error, we propose a novel regrasping method using imagebased visual servoing (IBVS) for the dual-arm robot with the general-purpose hands. Fig. 2 shows an outline of a regrasping procedure proposed in this paper. As shown in Fig. 2, after Robot Arm A picks up one object by the general-purpose Hand A with the grasping error, the object pose is then controlled to coincide with the pre-planned desired pose by IBVS. Then the pose of the other general-purpose robot hand (named Hand B) is controlled by position control to the goal pose set by the user beforehand and Hand B grasps the object. After that, the grasping by Hand A is released.
The positioning accuracy of the object by the proposed method depends on the relative pose of the grasped object to a camera. Thus, the proposed method in a planning phase computes the optimal relative pose that maximizes the minimum singular value of the image Jacobian using a 3D model of the target object. IBVS requires the image Jacobian related to the motion of the robot to the image. One of the most common approaches of finding the image Jacobian is to derive it analytically. In this approach, the image coordinates of keypoints, such as edges and corners, in the image plane are used for the image features of IBVS. However, selecting appropriate image features according to objects and detecting the keypoints for objects with various types of shapes robustly against image noises and changes of the lighting environment are difficult. Thus, the proposed method directly uses luminance values of the pixels as the image feature for IBVS and computes the image Jacobian using the pre-obtained images numerically. Even if the desired pose of the target object in the camera coordinate system at the time of regrasping is unique, the image Jacobian changes according to the grasping. In other words, a large number of data sets consisting of images and joint angles are necessary to compute the image Jacobian for multiple grasping. To avoid this high-cost and time-consuming process for data collecting, we propose a conversion method of multiple image Jacobian using only one data set of one representative grasping.
The symbols used in this paper are as follows. Joint angles of a robot arm with N degrees of freedom are denoted by θ := [θ 1 , θ 2 , · · · , θ N ] ⊤ . The symbol A T B represents a homogeneous transformation matrix that represents the pose of coordinate system B in coordinate system A. A set of real numbers is represented by R.

II. RELATED WORKS
We can mainly classify related works for the regrasping into the three following categories: 1) in-hand manipulation, 2) dynamic regrasping, and 3) regrasping by a dual-arm robot.
In-hand manipulation is a method of changing the pose of an object in a robot hand after grasping the object. Various in-hand manipulation methods have been proposed [18]- [23]. T. Okada [18] has developed the hand with three fingers to realize the operation of rotating a bar-like object. D. Rus [19] has proposed a method to change the pose of a simple-shaped object considering the fingertip slip. H. V. Hoof et al. [20] have proposed a data-driven method for in-hand manipulation and conducted an experiment to rotate a simple-shaped object in a hand with a tactile sensor. In addition, N. C. Dafle et al. [21] and A. Sintov et al. [22], [23] have proposed in-hand manipulation methods that use contact force between an object and a surrounding environment for changing the object pose. T. Yamawaki et al. [24], [25] have proposed effective learning methods for in-hand manipulation without any precise model and manual tuning of the learning and have validated the proposed method by using an asymmetric hexagonal object in a two-dimensional space.
Dynamic regrasping is a method of changing the pose of an object by releasing the grasped object, then grasping it again with a different pose [26], [27]. In [28], a robot throws an object upward using a multi-fingered hand that can operate at a high speed, detects the object with a high-speed camera, and regrasps the object. A. Sintov et al. [22], [23] have proposed a method of dropping an object, changing its pose, and grasping it again. Regrasping methods using dual-arm robots have been also proposed. Methods of changing the pose of an object using multiple manipulators have been also presented in [29], [30]. W. Wan et al. [30] have proposed a method of regrasping by computing grasping candidates of a target object, finding state transition between the grasping candidates in a planning phase, and finally regrasping the object based on the found state transition to achieve the goal pose of the object. Moreover, a technique to regrasp clothes using a dual-arm robot has been proposed in [29].
This paper focuses on regrasping objects using the dual-arm robot from the viewpoint of success rate, reproducibility, and versatility for shapes of target objects. The contributions of this paper compared with other methods are as follows: • A variety of researches for regrasping using the dualarm robot has focused on the planning of grasping and regrasping and has proposed useful and practical methods for the planning. On the other hand, this paper proposes a robust regrasping method for the grasping error which has to be considered when an object is picked from a bin by a robotic general-purpose hand. • This research is a first attempt to achieve the regrasping robustly against the grasping error using the dual-arm robot with the general-purpose hands by IBVS. • The proposed method can be applied to the positioning of various object types regardless of the complexity of the shapes of the objects since the image Jacobian of IBVS is computed numerically by using the actual data set. • The proposed method requires only one data set for a representative grasping candidate to compute the image Jacobian even if a large number of candidate grasps exists. In addition, the image Jacobian is computed in real-time in validation experiments. • The experimental results have shown a 100% success rate of the regrasping.
III. PROPOSED REGRASPING METHOD As described in sec. I, this study aims to regrasp an object with a dual-arm robot equipped with general-purpose hands. This paper considers the following problem of the regrasping.

Problem
It is assumed that the 3D model of the object and the relative pose of two robot arms are given. Parallel grippers are used as Hand A and B. Hand A picks one object from piled-up objects with grasping error and then hand over it to Hand B. A final goal of the regrasping is to grasp the object by represents the jth grasp candidate, and G B is the given goal set of the grasp candidates. Assuming that hand B is a parallel gripper, we can express G Bi by G Bi = (w Bi , B T ob ), where w Bi ∈ R 2 denotes the opening and closing widths of the parallel gripper and B T ob represents the object pose at the time of grasping in the Hand B coordinate system. Consider a control framework to regrasp an object robustly against the grasping error. □ As shown in Fig. 2, this paper proposes a regrasping scheme as follows: We use the general-purpose hands herein to grasp objects with various shapes. In addition, we do not utilize any fixtures, as described in sec. I. Thus, we must consider the influence of the grasping error for Hand A that grasps the not positioned object. The grasping error is defined as follows: we denote the desired and actual relative poses of the object to the robot hand by A T * ob and A T ob , respectively. The grasping error is defined by E grasp : In a case where the grasping error can be ignored, the regrasping can be achieved by controlling the pose of Hand A to the planned desired pose by the position control of Robot Arm A. To perform the regrasping robustly against the grasping error, the proposed method performed positioning not Hand A, but the object using IBVS.
The proposed method consists of the planning and online phases as shown below.

A. Planning Phase
The proposed method first computes the optimal relative pose of the object to the camera at the time of regrasping (hereinafter called the "optimal regrasping pose of the object" or "optimal regrasping pose") based on the 3D model of the object and the camera arrangement. The positioning accuracy of IBVS depends on the relative pose between the camera and the object. The details are explained in sec. V.
Next, a set of candidate grasps of regrasping is calculated. In this step, we first generate a set of candidate grasps with Hand A, for the object denoted by G A := {G A1 , G A2 , · · · , G AN }. For the grasps G Ai and G B j , its grasp qualities denoted by s Ai = f (G Ai ), s B j = f (G B j ) are then computed, respectively, where f is a function of the grasp to calculate the grasp quality. Note that the proposed method has no restriction on grasp quality measures. One of the most common grasping qualities is the radius of the inscribed sphere of the grasp wrench space (GWS) [34]. We adopt it as the grasp quality in the verification experiment shown in sec. VII. Finally, for each grasp candidate G Ai , we select the grasp G Bℓ(i) with the highest grasp quality among the candidate grasps of Hand B without any overlap of Hand A with grasp G Ai and B with grasp G Bℓ(i) . As a consequence, the sets of candidate grasps of the regrasping are represented by After finding the sets of candidate grasps of the regrasping, the image Jacobian of IBVS is computed by using actual data set including actual images and joint angles of Robot Arm A. The image Jacobian J robot used for the regrasping relates the joint angle θ of Robot Arm A to the image of the object I ob captured by the camera and is defined by where p ob and p A represent the pose of the object and Hand A, respectively 1 . As shown in (3), the image Jacobian J robot depends on grasping because ∂ p ob /∂ p A relies on grasping. Therefore, even if the optimal regrasping pose of the object is unique, the same number of the image Jacobian as the grasp candidates of Hand A need to be prepared. However, for computing the image Jacobian, acquiring the same number of the sets of actual images as the grasp candidates of Hand A is not feasible from the viewpoint of time and cost. We solve this problem herein by proposing an effective method for computing the image Jacobian using only one data set of actual images corresponding to one representative grasp in sec. VI.

B. Online Phase
In the online phase, we first estimate the poses of the objects. Next, Hand A grasps one object with the grasp candidate G A,i * with the highest grasp quality among the feasible candidate grasps for the piled-up objects. The object is then positioned by IBVS at the optimal pose computed in the planning phase. Finally, Hand B grasps the object with G B,ℓ(i * ) and then Hand A releases the grasps of the object.
In the following sections, sec. IV describes the object positioning by IBVS; sec. V represents the computation of the optimal regrasping pose of the object; and sec. VI presents an efficient calculation method of the image Jacobian.

IV. REGRASPING BY IBVS
In conventional IBVS methods, the coordinates of the 2D keypoints computed from the feature quantity, such as edges, corners, SIFT [31], SURF [32], and AKAZE [33], are usually used for the image feature of IBVS. However, this approach is effective only when the image feature can be extracted robustly against image noises, changes of the lighting environment, and shape of the object. To solve this problem, the proposed method uses the intensity of the image as the image feature directly. In addition, we set Region of Interest (ROI) in the captured image such that ROI does not include the appearance of the hand but the object. By setting ROI in this way, we need to prepare only one goal image for the optimal regrasping pose of the object. In other words, if we use the entire image for IBVS, multiple goal images are required for only one optimal pose, since the goal image changes according to the grasping.
A controller of IBVS in the proposed regrasping is given as:θ whereθ d ∈ R N , N, λ ∈ R, J † robot , I ob (t), and I * ob are the control input of the joint's velocity of Robot Arm A, the degrees of freedom of Robot Arm A, the controller gain, a pseudo-inverse matrix of the image Jacobian J robot , the image in ROI captured by the camera at time t, and the image in ROI with the object being located at the optimal regrasping pose, respectively.
A controller similar to (4) has been proposed in [35]. In [35], the image Jacobian has been successfully derived by introducing an optical reflection model of the object surface and identifying the light environment. This study does not assume any reflection model since it is difficult to identify the reflection models of various shapes of objects and light environment. Thus this paper numerically computes the image Jacobian. In this case, we need to numerically compute the same number of image Jacobian as the grasp candidates of Hand A. However, preparing many sets of the actual images for computing the image Jacobian is not feasible. We propose herein a feasible computation method of the image Jacobian in sec. VI.

V. OPTIMAL OBJECT POSE FOR REGRASPING
As described in sec. I and III, Robot Arm A performs object positioning at the optimal pose by IBVS. The positioning accuracy by IBVS depends on captured image of the object at the optimal pose. This study computes the optimal pose of the object by maximizing the minimum singular value of the image Jacobian of the object in a numerical simulation, as follows; p * ob = arg max p ob,i ( σ min (J ob (p ob,1 )), σ min (J ob (p ob,2 )), · · · , σ min (J ob (p ob,N )) ) , (5) where σ min (X), J ob (p ob,i ), and p * ob are the minimum singular value of the matrix X, the image Jacobian of the object at the pose p ob,i , and the optimal pose of the object, respectively. The image Jacobian J ob (p ob,i ) defined by (3) relates the object pose p ob to the image I ob .

VI. CONVERSION OF THE IMAGE JACOBIAN
In this section, we present a conversion method of the image Jacobian of IBVS in (4).
First, the proposed method selects a candidate grasp G rep from the set of the candidate grasp G A as a representative grasp.
Then, the proposed method computes the image Jacobian for the representative grasp. We set Robot Arm A, Hand A, and object coordinate systems as shown in Fig. 3. When the object grasped by Hand A with the grasp G rep is positioned at the optimal regrasping pose, the image I ob ( r T ob * ) and the joint angle θ * rep of the arm A are recorded 2 . After that, the object pose is shifted by ∆ r T ob,i , i = 1, 2, · · · , K from the optimal regrasping pose. In this situation, the image I( r T ob,i ), i = 1, 2, · · · , K and the joint angles θ rep,i , i = 1, 2, · · · , K are also recorded. By using the recorded data, a pseudo-inverse of the image Jacobian for the representative grasp G rep can be computed as follows: where In the proposed method, the optimal regrasping pose of the object is uniquely determined, as shown in sec. V. The image Jacobian defined by (3) depends on the grasp. Therefore, the additional data set denoted by (7) and (9) needs to be experimentally obtained to compute the image Jacobian for all grasps belonging to G A . However, this process is not feasible when a large number of candidate grasps exists.
To solve this problem, we here propose the conversion method of the image Jacobian that does not require additional data sets. The purpose here is to compute the pseudo-inverse matrix of the image Jacobian J † (G X , r T ob * ) for the object located at r T ob * and grasped by G X ∈ G A (G X ̸ = G rep ) (See Fig. 4) using the obtained data set shown in (7) and (9). For computing the pseudo-inverse matrix of the image Jacobian, we need to prepare the deviation of the images and the joint angles represented by (7) and (9), respectively.
First, we can use the same data for the deviation of the images ∆I( r T ob * ) in the additional data sets for grasp G X . Since the images in ROI do not change since ROI includes only the target object, not the gripper, even if the grasp is changed, Next, we compute the joint angles of the robot arm denoted by θ * X to prepare the data ∆Θ(G X , r T ob * ). When grasping the object with pose r T ob * by G rep , the pose of Hand A at this time can be computed as follows using forward kinematics: where f is a function of the joint angles to compute the pose of Hand A. We can obtain the pose of the object in the robot coordinate system by substituting the pose r T A (G rep , r T ob * ) to the following equation: Note here that A T ob * is found from G rep = (w A , A T ob * ). Hence, the pose of Hand A when it holds the object located at r T ob * with the grasp G X can be computed as: By substituting r T A (G X , r T ob * ) into the inverse kinematics f −1 , the joint angles we want to find are obtained by: The above discussion holds if we replace ob * and θ * rep with ob, i and θ rep,i , respectively. In other words, we can compute θ X,i using the above method. As a consequence, we can compute J † (G X , r T ob * ) using θ * X , θ rep,i , I( r T ob * ), and I( r T ob,i ).

VII. VALIDATION EXPERIMENT
This section shows the experiments that validate the proposed method. Figs. 5 and 6 show the experimental equipments for the regrasping and picking, respectively. A high-speed camera from Photron Inc. with model number IDP-Express R2000 is used for IBVS. Its resolution is 512 × 512 (pixel), and its frame rate is 50 (fps). Two vertically articulated robots with six degrees of freedom are used as robot arms A and B. Two parallel grippers are equipped with robot arms A and B.
Figs. 12-14 shows the target object A, B, and C, which are actual industrial parts made of resin. ENSENSO N35 is used for a 3D vision sensor to measure profiles of the randomly piled up objects, as shown in Fig. 6. We use PPF-MEAM [36]- [39] as the pose estimation algorithm for the piled up objects.
First, we compute the optimal pose using the method proposed in sec. V. In the camera coordinate system, a total of 6912 candidate poses of the objects are created by rotating the objects with 15 (deg.) step for each axis of the object coordinate systems. The minimum singular values of the image Jacobian for each candidate pose of the objects are computed. Figs. 7 and 8 show the poses of the target object A corresponding to the 1st to 4th largest and 6909th to 6912th largest minimum singular values of the image Jacobian, respectively. Consequently, the pose with a large minimum singular value indicates that its surface visible from the camera is large, and    the normal to the major surface is slightly inclined with respect to the optical axis of the camera. This is considered to ensure the sensitivity to the depth direction in the camera coordinate system. The regrasping of target objects A, B, and C is performed according to the proposed method. Figs. 9-11 show one sequence of the regrasping and positioning of target objects A, B, and C by the proposed regrasping, respectively. Fig. 17 shows time-series images captured by the camera at the time of IBVS for target object A. A total of 30 experiments of the regrasping for each object are conducted. The success rates for each object are 100%, and the grasps are different from each other in every trial. This result shows that the accuracy of the proposed conversion method of the image Jacobian is sufficiently high for the regrasping. For comparison, we have performed experiments with the object pose at the time of regrasping being not optimal. In this case, the success rate of the regrasping is 87%.
Finally, we conduct an experiment to evaluate the positioning accuracy of the proposed regrasping for target object A. As shown in Fig. 15, a laser displacement meter with model number LJ-V7300 made by KEYENCE CORPORATION is set for measuring the positioning error. Fig. 16 shows the average positioning errors in four cases. A total of 10 trials are conducted for each case. The grasping error with 5 deg. for one axis of the object coordinate system is added in the two cases. Fig. 16 shows that even if the grasping error exists, the average positioning error by the proposed method is 0.69 [mm], and the pose with a larger minimum singular value provided a much higher positioning accuracy.

VIII. CONCLUSION
This study has proposed a novel robust regrasping method against the grasping error by IBVS. The proposed method mainly consists of the computation of the optimal pose of the object at the time of the regrasping based on the 3D model of the object, the effective computation of a large number of the image Jacobian corresponding to all candidate grasps, the generation of the regrasping candidates, and the positioning of the object at the optimal regrasping pose by the customized IBVS. The experimental results show that the proposed method achieves regrasping for the actual industrial parts with a 100% success rate and exhibited the effectiveness of each component in the proposed method.