Autonomous harvesting may provide a viable solution to mounting labor pressures in the United States's strawberry industry. However, due to bottlenecks in machine perception and economic viability, a profitable and commercially adopted strawberry harvesting system remains elusive. In this research, we explore the feasibility of using deep reinforcement learning to overcome these bottlenecks and develop a practical algorithm to address the sub-objective of viewpoint optimization, or the development of a control policy to direct a camera to favorable vantage points for autonomous harvesting. We evaluate the algorithm's performance in a custom, open-source simulated environment and observe encouraging results. Our trained agent yields 8.7 times higher returns than random actions and 8.8 percent faster exploration than our best baseline policy, which uses visual servoing. Visual investigation shows the agent is able to fixate on favorable viewpoints, despite having no explicit means to propagate information through time. Overall, we conclude that deep reinforcement learning is a promising area of research to advance the state of the art in autonomous strawberry harvesting.
To the best of our knowledge, this work is the first research exploring viewpoint optimization via reinforcement learning for autonomous harvesting applications (any crop). A related idea is visual servoing @cite_0 , which is a class of control algorithms acting on image features that has been applied in the autonomous harvesting domain. Unlike our method, visual servoing typically involves hand-specifying features, whereas our algorithm learns a control policy with a data-driven approach. @cite_16 , Mehta and Burks implement visual servoing by means of two cameras: one in the hand of a citrus harvesting robot and one stationary camera with a wide field of view. The feedback from the cameras is then used to create a perspective image and guide the robotic manipulator towards an artificial citrus fruit. One of the main limitations of this approach is that it requires the target fruit to be visible by the fixed camera, which cannot be guaranteed in unstructured environments.
@cite_0 @cite_16
2072150826 569732077 2586545389 2625680238
The main contribution of this paper is in the development of vision-based estimation and control system for robotic fruit harvesting and rigorous stability analysis to guarantee performance of the closed-loop system. The presented cooperative visual servo controller benefits from the large field-of-view of a fixed camera and the accuracy of a camera-in-hand (CiH). Computationally inexpensive perspective transformation-based range estimation method obtains 3D fruit position using a monocular camera to enable real-time manipulator control. A rotation controller is developed to orient the robot such that the target fruit selected by the fixed camera can be viewed by the CiH attached to the end-effector. Subsequently, the end-effector can be servoed to the target fruit location using the presented pursuit guidance based hybrid translation controller. Lyapunov-based stability analysis guarantees global exponential regulation of the end-effector. Numerical simulations verify the feasibility of the developed controller while the performance is evaluated on a seven degrees-of-freedom kinematically redundant manipulator using an artificial citrus tree. The position of the fruit was randomly selected, and the closed-loop visual servo control experiment was performed 21 times to analyze the repeatability and accuracy of the developed controller. With 95 confidence level the expected position of the robot end-effector is observed to lie within the confidence ellipsoid. The accuracy of the controller was observed to be about 15mm, thus making the system suitable for harvesting medium and large varieties of citrus fruit but may limit operation for small varieties such as page and blood oranges. Abstract This paper concerns the position-based visual servo control of autonomous robotic manipulators in space. It focuses on the development of a real-time vision-based pose and motion estimation algorithm of a non-cooperative target by photogrammetry and extended Kalman filter for robotic manipulators to perform autonomous capture. Optical flow algorithm is adopted to track the target features in order to improve the image processing efficiency. Then, a close-loop position-based visual servo control strategy is devised to determine the desired pose of the end-effector at the rendezvous point based on the estimated pose and motion of the target. The corresponding desired joint angles of the robotic manipulator in the joint space are derived by the inverse kinematics of the robotic manipulator. The developed algorithm and position-based visual servo control strategy are validated experimentally by a custom built robotic manipulator with an eye-in-hand configuration. The experimental results demonstrate the proposed estimation algorithm and control scheme are feasible and effective. In this paper we present a perception system for agriculture robotics that enables an unmanned ground vehicle (UGV) equipped with a multi spectral camera to automatically perform the crop weed detection and classification tasks in real-time. Our approach exploits a pipeline that includes two different convolutional neural networks (CNNs) applied to the input RGB+near infra-red (NIR) images. A lightweight CNN is used to perform a fast and robust, pixel-wise, binary image segmentation, in order to extract the pixels that represent projections of 3D points that belong to green vegetation. A deeper CNN is then used to classify the extracted pixels between the crop and weed classes. A further important contribution of this work is a novel unsupervised dataset summarization algorithm that automatically selects from a large dataset the most informative subsets that better describe the original one. This enables to streamline and speed-up the manual dataset labeling process, otherwise extremely time consuming, while preserving good classification performance. Experiments performed on different datasets taken from a real farm robot confirm the effectiveness of our approach. Abstract The overall goal of this study is to develop an effective, simple, aptly computer vision algorithm to detect and count citrus on the tree using image processing techniques, to estimate the yield, and to compare the yield estimation results obtained through several methods. This new citrus recognition and counting algorithm was utilized the color features (or schemes) to present an estimate of the citrus yield, and the corresponding models are developed to provide an early estimation of the citrus yield. Citrus images were taken from Jeju, South Korea during daylight and the citrus recognition and counting algorithm were tested on 84 images which were collected from 21 trees. The citrus counting algorithm consisted of the following steps: convert RGB image to HSV, thresholding, orange color detection, noise removal, watershed segmentation, and counting. Distance transform and marker-controlled watershed algorithms were evaluated for automated watershed segmentation in citrus fruits to obtain good result. A correlation coefficient R 2 of 0.93 was obtained between the citrus counting algorithm and counting performed through human observation. The proposed algorithm showed great potential for early prediction of the yield of single citrus trees and the possibility of its uses for further fruit crops.