An accurate and reliable image based fruit detection system is critical for supporting higher level agriculture tasks such as yield mapping and robotic harvesting. This paper presents the use of a state-of-the-art object detection framework, Faster R-CNN, in the context of fruit detection in orchards, including mangoes, almonds and apples. Ablation studies are presented to better understand the practical deployment of the detection network, including how much training data is required to capture variability in the dataset. Data augmentation techniques are shown to yield significant performance gains, resulting in a greater than two-fold reduction in the number of training images required. In contrast, transferring knowledge between orchards contributed to negligible performance gain over initialising the Deep Convolutional Neural Network directly from ImageNet features. Finally, to operate over orchard data containing between 100-1000 fruit per image, a tiling approach is introduced for the Faster R-CNN framework. The study has resulted in the best yet detection performance for these orchards relative to previous works, with an F1-score of >0.9 achieved for apples and mangoes.
Analysis of local colours and textures has been used for pixel-wise mango classification, followed by blob extraction to identify individual mangoes @cite_19 . Avoiding the use of hand-engineered features, Convolutional Neural Networks (CNNs) have been used for pixel-wise apple classification @cite_0 , followed by Watershed Segmentation (WS) to identify individual apples. The radial symmetries in the specular reflection in berries have been shown to be useful for RoI extraction @cite_3 , where a KD-forest was used for berry classification. To detect citrus fruit, Circular Hough Transforms (CHT) have been used to extract key-points, which were then classified using a Support Vector Machine (SVM) @cite_10 .
@cite_0 @cite_19 @cite_10 @cite_3
2543665758 2501369945 2064808659 2963523428
Ground vehicles equipped with monocular vision systems are a valuable source of high-resolution image data for precision agriculture applications in orchards. This paper presents an image processing framework for fruit detection and counting using orchard image data. A general-purpose image segmentation approach is used, including two feature learning algorithms; multiscale multilayered perceptrons (MLP) and convolutional neural networks (CNN). These networks were extended by including contextual information about how the image data was captured (metadata), which correlates with some of the appearance variations and or class distributions observed in the data. The pixel-wise fruit segmentation output is processed using the watershed segmentation (WS) and circular Hough transform (CHT) algorithms to detect and count individual fruits. Experiments were conducted in a commercial apple orchard near Melbourne, Australia. The results show an improvement in fruit segmentation performance with the inclusion of metadata on the previously benchmarked MLP network. We extend this work with CNNs, bringing agrovision closer to the state-of-the-art in computer vision, where although metadata had negligible influence, the best pixel-wise F1-score of 0.791 was achieved. The WS algorithm produced the best apple detection and counting results, with a detection F1-score of 0.861. As a final step, image fruit counts were accumulated over multiple rows at the orchard and compared against the post-harvest fruit counts that were obtained from a grading and counting machine. The count estimates using CNN and WS resulted in the best performance for this data set, with a squared correlation coefficient of r2=0.826. This paper presents a novel approach to fruit detection using deep convolutional neural networks. The aim is to build an accurate, fast and reliable fruit detection system, which is a vital element of an autonomous agricultural robotic platform; it is a key element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). We adapt this model, through transfer learning, for the task of fruit detection using imagery obtained from two modalities: colour (RGB) and Near-Infrared (NIR). Early and late fusion methods are explored for combining the multi-modal (RGB and NIR) information. This leads to a novel multi-modal Faster R-CNN model, which achieves state-of-the-art results compared to prior work with the F1 score, which takes into account both precision and recall performances improving from 0 . 807 to 0 . 838 for the detection of sweet pepper. In addition to improved accuracy, this approach is also much quicker to deploy for new fruits, as it requires bounding box annotation rather than pixel-level annotation (annotating bounding boxes is approximately an order of magnitude quicker to perform). The model is retrained to perform the detection of seven fruits, with the entire process taking four hours to annotate and train the new model per fruit. Abstract Fruit and vegetables are normally presented to consumers in batches. The homogeneity and appearance of these have significant effect on consumer decision. For this reason, the presentation of agricultural produce is manipulated at various stages from the field to the final consumer and is generally oriented towards the cleaning of the product and sorting by homogeneous categories. The project ESPRIT 3, reference 9230 ‘Integrated system for handling, inspection and packing of fruit and vegetable (SHIVA)’ developed a robotic system for the automatic, non-destructive inspection and handling of fruit. The aim of this paper is to report on the machine vision techniques developed at the Instituto Valenciano de Investigaciones Agrarias for the on-line estimation of the quality of oranges, peaches and apples, and to evaluate the efficiency of these techniques regarding the following quality attributes: size, colour, stem location and detection of external blemishes. The segmentation procedure used, based on a Bayesian discriminant analysis, allowed fruits to be precisely distinguished from the background. Thus, determination of size was properly solved. The colours of the fruits estimated by the system were well correlated with the colorimetric index values that are currently used as standards. Good results were obtained in the location of the stem and the detection of blemishes. The classification system was tested on-line with apples obtaining a good performance when classifying the fruit in batches, and a repeatability in blemish detection and size estimation of 86 and 93 respectively. The precision and repeatability of the system, was found to be similar to those of manual grading. An accurate and reliable image based fruit detection system is critical for supporting higher level agriculture tasks such as yield mapping and robotic harvesting. This paper presents the use of a state-of-the-art object detection framework, Faster R-CNN, in the context of fruit detection in orchards, including mangoes, almonds and apples. Ablation studies are presented to better understand the practical deployment of the detection network, including how much training data is required to capture variability in the dataset. Data augmentation techniques are shown to yield significant performance gains, resulting in a greater than two-fold reduction in the number of training images required. In contrast, transferring knowledge between orchards contributed to negligible performance gain over initialising the Deep Convolutional Neural Network directly from ImageNet features. Finally, to operate over orchard data containing between 100–1000 fruit per image, a tiling approach is introduced for the Faster R-CNN framework. The study has resulted in the best yet detection performance for these orchards relative to previous works, with an F1-score of > 0.9 achieved for apples and mangoes.