This letter presents a three-dimensional (3-D) visual detection method for the challenging task of detecting peduncles of sweet peppers (Capsicum annuum) in the field. Cutting the peduncle cleanly is one of the most difficult stages of the harvesting process, where the peduncle is the part of the crop that attaches it to the main stem of the plant. Accurate peduncle detection in 3-D space is, therefore, a vital step in reliable autonomous harvesting of sweet peppers, as this can lead to precise cutting while avoiding damage to the surrounding plant. This letter makes use of both color and geometry information acquired from an RGB-D sensor and utilizes a supervised-learning approach for the peduncle detection task. The performance of the proposed method is demonstrated and evaluated by using qualitative and quantitative results [the area-under-the-curve (AUC) of the detection precision-recall curve]. We are able to achieve an AUC of 0.71 for peduncle detection on field-grown sweet peppers. We release a set of manually annotated 3-D sweet pepper and peduncle images to assist the research community in performing further research on this topic.
This section reviews existing methods for detecting peduncles and crops using 3D geometry and visual features. Such techniques are widely used for autonomous crop inspection and detection tasks. Cubero demonstrated the detection of various fruit peduncles using radius and curvature signatures @cite_0 . The Euclidean distance and the angle rate change between each of the points on the contour and the fruit centroid are calculated. The presence of peduncles yields rapid changes in these metrics and can be detected using a specified threshold. Blasco @cite_13 and Ruiz @cite_18 presented peduncle detection of oranges, peaches, and apples using a Bayesian discriminant model of RGB colour information. The size of a colour segmented area was then calculated and assigned to pre-defined classes. These methods are more likely suitable for the quality control and inspection of crop peduncles after the crop have been harvested rather than for harvesting automation, as they require an inspection chamber that provides ideal lighting conditions with a clean background, no occlusions, good viewpoints, and high-quality static imagery.
@cite_0 @cite_18 @cite_13
2501369945 2087877962 2064245748 1993925155
This paper presents a novel approach to fruit detection using deep convolutional neural networks. The aim is to build an accurate, fast and reliable fruit detection system, which is a vital element of an autonomous agricultural robotic platform; it is a key element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). We adapt this model, through transfer learning, for the task of fruit detection using imagery obtained from two modalities: colour (RGB) and Near-Infrared (NIR). Early and late fusion methods are explored for combining the multi-modal (RGB and NIR) information. This leads to a novel multi-modal Faster R-CNN model, which achieves state-of-the-art results compared to prior work with the F1 score, which takes into account both precision and recall performances improving from 0 . 807 to 0 . 838 for the detection of sweet pepper. In addition to improved accuracy, this approach is also much quicker to deploy for new fruits, as it requires bounding box annotation rather than pixel-level annotation (annotating bounding boxes is approximately an order of magnitude quicker to perform). The model is retrained to perform the detection of seven fruits, with the entire process taking four hours to annotate and train the new model per fruit. Yield mapping for tree crops by mechanical harvesting requires automatic detection and counting of fruits in tree canopy. However, partial occlusion, shape irregularity, varying illumination, multiple sizes and similarity with the background make fruit identification a very difficult task to achieve. Therefore, immature green citrus-fruit detection within a green canopy is a challenging task due to all the above-mentioned problems. A novel algorithmic technique was used to detect immature green citrus fruit in tree canopy under natural outdoor conditions. Shape analysis and texture classification were two integral parts of the algorithm. Shape analysis was conducted to detect as many fruits as possible. Texture classification by a support vector machine (SVM), Canny edge detection combined with a graph-based connected component algorithm and Hough line detection, were used to remove false positives. Next, keypoints were detected using a scale invariant feature transform (SIFT) algorithm and to further remove false positives. A majority voting scheme was implemented to make the algorithm more robust. The algorithm was able to accurately detect and count 80.4 of citrus fruit in a validation set of images acquired from a citrus grove under natural outdoor conditions. The algorithm could be further improved to provide growers early yield estimation so that growers can manage grove more efficiently on a site-specific basis to increase yield and profit. We developed a strawberry-harvesting robot, consisting of a cylindrical manipulator, end-effector, machine vision unit, storage unit and travelling unit, for application to an elevated substrate culture. The robot was based on the development concepts of night operation, peduncle handling and task sharing with workers, to overcome the robotic harvesting problems identified by previous studies, such as low work efficiency, low success rate, fruit damage, difficulty of detection in unstable illumination and high cost. In functional tests, the machine vision assessments of fruit maturity agreed with human assessments for the Amaotome and Beni-hoppe cultivars, but the performance for Amaotome was significantly better. Moreover, the machine vision unit correctly detected a peduncle of the target fruit at a rate of 60 . In harvesting tests conducted throughout the harvest season on target fruits with a maturity of 80 or more, the successful harvesting rate of the system was 41.3 when fruits were picked using a suction device before cutting the peduncle, while the rate was 34.9 when fruits were picked without suction. There were no significant differences between the two picking methods in terms of unsuccessful picking rates. The execution time for the successful harvest of a single fruit, including the time taken to transfer the harvested fruit to a tray, was 11.5 s. Automated harvesting requires accurate detection and recognition of the fruit within a tree canopy in real-time in uncontrolled environments. However, occlusion, variable illumination, variable appearance and texture make this task a complex challenge. Our research discusses the development of a machine vision system, capable of recognizing occluded green apples within a tree canopy. This involves the detection of “green” apples within scenes of “green leaves”, shadow patterns, branches and other objects found in natural tree canopies. The system uses both thermal infra-red and color image modalities in order to achieve improved performance. Maximization of mutual information is used to find the optimal registration parameters between images from the two modalities. We use two approaches for apple detection based on low and high-level visual features. High-level features are global attributes captured by image processing operations, while low-level features are strong responses to primitive parts-based filters (such as Haar wavelets). These features are then applied separately to color and thermal infra-red images to detect apples from the background. These two approaches are compared and it is shown that the low-level feature-based approach is superior (74 recognition accuracy) over the high-level visual feature approach (53.16 recognition accuracy). Finally, a voting scheme is used to improve the detection results, which drops the false alarms with little effect on the recognition rate. The resulting classifiers acting independently can partially recognize the on-tree apples, however, when combined the recognition accuracy is increased.