Phenotyping is the process of measuring an organism's observable traits. Manual phenotyping of crops is a labor-intensive, time-consuming, costly, and error prone process. Accurate, automated, high-throughput phenotyping can relieve a huge burden in the crop breeding pipeline. In this paper, we propose a scalable, high-throughput approach to automatically count and segment panicles (heads), a key phenotype, from aerial sorghum crop imagery. Our counting approach uses the image density map obtained from dot or region annotation as the target with a novel deep convolutional neural network architecture. We also propose a novel instance segmentation algorithm using the estimated density map, to identify the individual panicles in the presence of occlusion. With real Sorghum aerial images, we obtain a mean absolute error (MAE) of 1.06 for counting which is better than using well-known crowd counting approaches such as CCNN, MCNN and CSRNet models. The instance segmentation model also produces respectable results which will be ultimately useful in reducing the manual annotation workload for future data.
For agronomy applications, such as ours, understanding the crop is fundamental to developing an effective algorithm. A well executed approach that illustrates this is @cite_41 that count tomatoes by first segmenting the image then a decision tree extracts the fruit segments. The decision tree used color, shape, texture and size features to locate the tomatoes. A similar approach was taken in @cite_57 for counting panicles in Sorghum. This is the only other published work we are aware of for panicle counting.
@cite_41 @cite_57
2625680238 2064352738 2087877962 2765516633 2094025749 2897179582 2543665758 2753403518 2586545389 2064808659 2501369945 1993925155 2578363764 2156585878 2013785102 2091119928 2182749117 2770754200 2047243441 2588938825
Abstract The overall goal of this study is to develop an effective, simple, aptly computer vision algorithm to detect and count citrus on the tree using image processing techniques, to estimate the yield, and to compare the yield estimation results obtained through several methods. This new citrus recognition and counting algorithm was utilized the color features (or schemes) to present an estimate of the citrus yield, and the corresponding models are developed to provide an early estimation of the citrus yield. Citrus images were taken from Jeju, South Korea during daylight and the citrus recognition and counting algorithm were tested on 84 images which were collected from 21 trees. The citrus counting algorithm consisted of the following steps: convert RGB image to HSV, thresholding, orange color detection, noise removal, watershed segmentation, and counting. Distance transform and marker-controlled watershed algorithms were evaluated for automated watershed segmentation in citrus fruits to obtain good result. A correlation coefficient R 2 of 0.93 was obtained between the citrus counting algorithm and counting performed through human observation. The proposed algorithm showed great potential for early prediction of the yield of single citrus trees and the possibility of its uses for further fruit crops. Fully automated yield estimation of intact fruits prior to harvesting provides various benefits to farmers. Until now, several studies have been conducted to estimate fruit yield using image-processing technologies. However, most of these techniques require thresholds for features such as color, shape and size. In addition, their performance strongly depends on the thresholds used, although optimal thresholds tend to vary with images. Furthermore, most of these techniques have attempted to detect only mature and immature fruits, although the number of young fruits is more important for the prediction of long-term fluctuations in yield. In this study, we aimed to develop a method to accurately detect individual intact tomato fruits including mature, immature and young fruits on a plant using a conventional RGB digital camera in conjunction with machine learning approaches. The developed method did not require an adjustment of threshold values for fruit detection from each image because image segmentation was conducted based on classification models generated in accordance with the color, shape, texture and size of the images. The results of fruit detection in the test images showed that the developed method achieved a recall of 0.80, while the precision was 0.88. The recall values of mature, immature and young fruits were 1.00, 0.80 and 0.78, respectively. Yield mapping for tree crops by mechanical harvesting requires automatic detection and counting of fruits in tree canopy. However, partial occlusion, shape irregularity, varying illumination, multiple sizes and similarity with the background make fruit identification a very difficult task to achieve. Therefore, immature green citrus-fruit detection within a green canopy is a challenging task due to all the above-mentioned problems. A novel algorithmic technique was used to detect immature green citrus fruit in tree canopy under natural outdoor conditions. Shape analysis and texture classification were two integral parts of the algorithm. Shape analysis was conducted to detect as many fruits as possible. Texture classification by a support vector machine (SVM), Canny edge detection combined with a graph-based connected component algorithm and Hough line detection, were used to remove false positives. Next, keypoints were detected using a scale invariant feature transform (SIFT) algorithm and to further remove false positives. A majority voting scheme was implemented to make the algorithm more robust. The algorithm was able to accurately detect and count 80.4 of citrus fruit in a validation set of images acquired from a citrus grove under natural outdoor conditions. The algorithm could be further improved to provide growers early yield estimation so that growers can manage grove more efficiently on a site-specific basis to increase yield and profit. Recently, a body of computer vision research has studied the task of high-throughput plant phenotyping (measurement of plant attributes). The goal is to more rapidly and more accurately estimate plant properties as compared to conventional manual methods. In this work, we develop a method to measure two primary yield attributes of interest; stalk count and stalk width that are important for many broad-acre annual crops (sorghum, sugarcane, corn, maize for example). Prior work of using convolutional deep neural networks for plant analysis has either focused on object detection or dense image segmentation. In our work, we develop a novel pipeline that accurately extracts both detected object regions and dense semantic segmentation for extracting both stalk counts and stalk width. A ground-robot called the Robotanist is used to deploy a high-resolution stereo imager to capture dense image data of experimental plots of Sorghum plants. We ground-truth validate data extracted using two humans who assess the traits independently and we compare both accuracy and efficiency of human versus robotic measurements. Our method yields R-squared correlation of 0.88 for stalk count and a mean absolute error of 2.77 mm where average stalk width is 14.354 mm. Our approach is 30 times faster for stalk count and 270 times faster for stalk width measurement. This paper presents a multi-class image segmentation approach to automate fruit segmentation. A feature learning algorithm combined with a conditional random field is applied to multi-spectral image data. Current classification methods used in agriculture scenarios tend to use hand crafted application-based features. In contrast, our approach uses unsupervised feature learning to automatically capture most relevant features from the data. This property makes our approach robust against variance in canopy trees and therefore has the potential to be applied to different domains. The proposed algorithm is applied to a fruit segmentation problem for a robotic agricultural surveillance mission, aiming to provide yield estimation with high accuracy and robustness against fruit variance. Experimental results with data collected in an almond farm are shown. The segmentation is performed with features extracted from multi-spectral (colour and infrared) data. We achieve a global classification accuracy of 88 . Sorghum (Sorghum bicolor L. Moench) is a C4 tropical grass that plays an essential role in providing nutrition to humans and livestock, particularly in marginal rainfall environments. The timing of head development and the number of heads per unit area are key adaptation traits to consider in agronomy and breeding but are time consuming and labor intensive to measure. We propose a two-step machine-based image processing method to detect and count the number of heads from high-resolution images captured by unmanned aerial vehicles (UAVs) in a breeding trial. To demonstrate the performance of the proposed method, 52 images were manually labeled; the precision and recall of head detection were 0.87 and 0.98, respectively, and the coefficient of determination (R-2) between the manual and new methods of counting was 0.84. To verify the utility of the method in breeding programs, a geolocationbased plot segmentation method was applied to pre-processed ortho-mosaic images to extract > 1000 plots from original RGB images. Forty of these plots were randomly selected and labeled manually; the precision and recall of detection were 0.82 and 0.98, respectively, and the coefficient of determination between manual and algorithm counting was 0.56, with the major source of error being related to the morphology of plants resulting in heads being displayed both within and outside the plot in which the plants were sown, i.e., being allocated to a neighboring plot. Finally, the potential applications in yield estimation from UAV-based imagery from agronomy experiments and scouting of production fields are also discussed. Ground vehicles equipped with monocular vision systems are a valuable source of high-resolution image data for precision agriculture applications in orchards. This paper presents an image processing framework for fruit detection and counting using orchard image data. A general-purpose image segmentation approach is used, including two feature learning algorithms; multiscale multilayered perceptrons (MLP) and convolutional neural networks (CNN). These networks were extended by including contextual information about how the image data was captured (metadata), which correlates with some of the appearance variations and or class distributions observed in the data. The pixel-wise fruit segmentation output is processed using the watershed segmentation (WS) and circular Hough transform (CHT) algorithms to detect and count individual fruits. Experiments were conducted in a commercial apple orchard near Melbourne, Australia. The results show an improvement in fruit segmentation performance with the inclusion of metadata on the previously benchmarked MLP network. We extend this work with CNNs, bringing agrovision closer to the state-of-the-art in computer vision, where although metadata had negligible influence, the best pixel-wise F1-score of 0.791 was achieved. The WS algorithm produced the best apple detection and counting results, with a detection F1-score of 0.861. As a final step, image fruit counts were accumulated over multiple rows at the orchard and compared against the post-harvest fruit counts that were obtained from a grading and counting machine. The count estimates using CNN and WS resulted in the best performance for this data set, with a squared correlation coefficient of r2=0.826. Plant Diseases and Pests are a major challenge in the agriculture sector. An accurate and a faster detection of diseases and pests in plants could help to develop an early treatment technique while substantially reducing economic losses. Recent developments in Deep Neural Networks have allowed researchers to drastically improve the accuracy of object detection and recognition systems. In this paper, we present a deep-learning-based approach to detect diseases and pests in tomato plants using images captured in-place by camera devices with various resolutions. Our goal is to find the more suitable deep-learning architecture for our task. Therefore, we consider three main families of detectors: Faster Region-based Convolutional Neural Network (Faster R-CNN), Region-based Fully Convolutional Network (R-FCN), and Single Shot Multibox Detector (SSD), which for the purpose of this work are called “deep learning meta-architectures”. We combine each of these meta-architectures with “deep feature extractors” such as VGG net and Residual Network (ResNet). We demonstrate the performance of deep meta-architectures and feature extractors, and additionally propose a method for local and global class annotation and data augmentation to increase the accuracy and reduce the number of false positives during training. We train and test our systems end-to-end on our large Tomato Diseases and Pests Dataset, which contains challenging images with diseases and pests, including several inter- and extra-class variations, such as infection status and location in the plant. Experimental results show that our proposed system can effectively recognize nine different types of diseases and pests, with the ability to deal with complex scenarios from a plant’s surrounding area. In this paper we present a perception system for agriculture robotics that enables an unmanned ground vehicle (UGV) equipped with a multi spectral camera to automatically perform the crop weed detection and classification tasks in real-time. Our approach exploits a pipeline that includes two different convolutional neural networks (CNNs) applied to the input RGB+near infra-red (NIR) images. A lightweight CNN is used to perform a fast and robust, pixel-wise, binary image segmentation, in order to extract the pixels that represent projections of 3D points that belong to green vegetation. A deeper CNN is then used to classify the extracted pixels between the crop and weed classes. A further important contribution of this work is a novel unsupervised dataset summarization algorithm that automatically selects from a large dataset the most informative subsets that better describe the original one. This enables to streamline and speed-up the manual dataset labeling process, otherwise extremely time consuming, while preserving good classification performance. Experiments performed on different datasets taken from a real farm robot confirm the effectiveness of our approach. Abstract Fruit and vegetables are normally presented to consumers in batches. The homogeneity and appearance of these have significant effect on consumer decision. For this reason, the presentation of agricultural produce is manipulated at various stages from the field to the final consumer and is generally oriented towards the cleaning of the product and sorting by homogeneous categories. The project ESPRIT 3, reference 9230 ‘Integrated system for handling, inspection and packing of fruit and vegetable (SHIVA)’ developed a robotic system for the automatic, non-destructive inspection and handling of fruit. The aim of this paper is to report on the machine vision techniques developed at the Instituto Valenciano de Investigaciones Agrarias for the on-line estimation of the quality of oranges, peaches and apples, and to evaluate the efficiency of these techniques regarding the following quality attributes: size, colour, stem location and detection of external blemishes. The segmentation procedure used, based on a Bayesian discriminant analysis, allowed fruits to be precisely distinguished from the background. Thus, determination of size was properly solved. The colours of the fruits estimated by the system were well correlated with the colorimetric index values that are currently used as standards. Good results were obtained in the location of the stem and the detection of blemishes. The classification system was tested on-line with apples obtaining a good performance when classifying the fruit in batches, and a repeatability in blemish detection and size estimation of 86 and 93 respectively. The precision and repeatability of the system, was found to be similar to those of manual grading. This paper presents a novel approach to fruit detection using deep convolutional neural networks. The aim is to build an accurate, fast and reliable fruit detection system, which is a vital element of an autonomous agricultural robotic platform; it is a key element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). We adapt this model, through transfer learning, for the task of fruit detection using imagery obtained from two modalities: colour (RGB) and Near-Infrared (NIR). Early and late fusion methods are explored for combining the multi-modal (RGB and NIR) information. This leads to a novel multi-modal Faster R-CNN model, which achieves state-of-the-art results compared to prior work with the F1 score, which takes into account both precision and recall performances improving from 0 . 807 to 0 . 838 for the detection of sweet pepper. In addition to improved accuracy, this approach is also much quicker to deploy for new fruits, as it requires bounding box annotation rather than pixel-level annotation (annotating bounding boxes is approximately an order of magnitude quicker to perform). The model is retrained to perform the detection of seven fruits, with the entire process taking four hours to annotate and train the new model per fruit. Automated harvesting requires accurate detection and recognition of the fruit within a tree canopy in real-time in uncontrolled environments. However, occlusion, variable illumination, variable appearance and texture make this task a complex challenge. Our research discusses the development of a machine vision system, capable of recognizing occluded green apples within a tree canopy. This involves the detection of “green” apples within scenes of “green leaves”, shadow patterns, branches and other objects found in natural tree canopies. The system uses both thermal infra-red and color image modalities in order to achieve improved performance. Maximization of mutual information is used to find the optimal registration parameters between images from the two modalities. We use two approaches for apple detection based on low and high-level visual features. High-level features are global attributes captured by image processing operations, while low-level features are strong responses to primitive parts-based filters (such as Haar wavelets). These features are then applied separately to color and thermal infra-red images to detect apples from the background. These two approaches are compared and it is shown that the low-level feature-based approach is superior (74 recognition accuracy) over the high-level visual feature approach (53.16 recognition accuracy). Finally, a voting scheme is used to improve the detection results, which drops the false alarms with little effect on the recognition rate. The resulting classifiers acting independently can partially recognize the on-tree apples, however, when combined the recognition accuracy is increased. This paper describes a fruit counting pipeline based on deep learning that accurately counts fruit in unstructured environments. Obtaining reliable fruit counts is challenging because of variations in appearance due to illumination changes and occlusions from foliage and neighboring fruits. We propose a novel approach that uses deep learning to map from input images to total fruit counts. The pipeline utilizes a custom crowdsourcing platform to quickly label large data sets. A blob detector based on a fully convolutional network extracts candidate regions in the images. A counting algorithm based on a second convolutional network then estimates the number of fruits in each region. Finally, a linear regression model maps that fruit count estimate to a final fruit count. We analyze the performance of the pipeline on two distinct data sets of oranges in daylight, and green apples at night, utilizing human generated labels as ground truth. We also show that the pipeline has a short training time and performs well with a limited data set size. Our method generalizes across both data sets and is able to perform well even on highly occluded fruits that are challenging for human labelers to annotate. This work details the development and validation of an algorithm for estimating the number of apples in color images acquired in orchards under natural illumination. Ultimately, this algorithm is intended to enable estimation of the orchard yield and be part of a management decision support system. The algorithm includes four main steps: detection of pixels that have a high probability of belonging to apples, using color and smoothness; formation and extension of ''seed areas'', which are connected sets of pixels that have a high probability of belonging to apples; segmentation of the contours of these seed areas into arcs and amorphous segments; and combination of these arcs and comparison of the resulting circle with a simple model of an apple. The performance of the algorithm is investigated using two datasets. The first dataset consists of images recorded in full automatic mode of the camera and under various lighting conditions. Although the algorithm detects correctly more than 85 of the apples visible in the images, direct illumination and color saturation cause a large number of false positive detections. The second dataset consists of images that were manually underexposed and recorded under mostly diffusive light (close to sunset). For such images the correct detection rate is close to 95 while the false positive detection rate is less than 5 . Abstract Three image analysis methods were studied and evaluated to solve the problem of removing long stems attached to mechanically harvested oranges: colour segmentation based on linear discriminant analysis, contour curvature analysis, and a thinning process which involves iterating until the stem becomes a skeleton. These techniques are able to determine the presence or absence of a stem with certainty, to locate the stems from random views with more than 90 accuracy and from profile images with an accuracy ranging from 92·4 to 100 depending on the method used. Finally, determination of the length and cutting point of the stem is achieved with only 3·8 of failures. This paper extends a previous study on the use of image analysis to automatically estimate mango crop yield (fruit on tree) (, 2013). Images were acquired at night, using artificial lighting of fruit at an earlier stage of maturation ('stone hardening' stage) than for the previous study. Multiple image sets were collected during the 2011 and 2012 seasons. Despite altering the settings of the filters in the algorithm presented in the previous study (based on colour segmentation using RGB and YCbCr, and texture), the less mature fruit were poorly identified, due to a lower extent of red colouration of the skin. The algorithm was altered to reduce its dependence on colour features and to increase its use of texture filtering, hessian filtering in particular, to remove leaves, trunk and stems. Results on a calibration set of images (2011) were significantly improved, with 78.3 of fruit detected, an error rate of 10.6 and an R^2 value (machine vision to manual count) of 0.63. Further application of the approach on validation sets from 2011 and 2012 had mixed results, with issues related to variation in foliage characteristics between sets. It is proposed the detection approaches within both of these algorithms be used as a 'toolkit' for a mango detection system, within an expert system that also uses user input to improve the accuracy of the system. Innovative concept proposed for apple counting in tall spindle apple trees orchard.Over-the-row system with tunnel structure provide uniform illumination for imaging.Image processing for apple identification on tree canopy images.Repetitive counting of apples from two sides was avoided using 3D information. Accurate crop-load estimation is important for efficient management of pre- and post-harvest operations. This information is crucial for the planning of labor and equipment requirement for harvesting and transporting fruit from the orchard to packing house. Current machine vision-based techniques for crop-load estimation have achieved only limited success mostly due to: (i) occlusion of apples by branches, leaves and or other apples, and (ii) variable outdoor lighting conditions. In order to minimize the effect of these factors, a new sensor system was developed with an over-the-row platform integrated with a tunnel structure which acquired images from opposite sides of apple trees. The tunnel structure minimized illumination of apples with direct sunlight and reduced the variability in lighting condition. Images captured in a tall spindle orchard were processed for identifying apples, which achieved an identification accuracy of 79.8 . The location of apples in three-dimensional (3D) space was used to eliminate duplicate counting of apples that were visible to cameras from both sides of the tree canopy. The error on identifying duplicate apples was found to be 21.1 . Overall, the method achieved an accuracy of 82 on estimating crop-load on trees with dual side imaging compared to 58 with single side imaging. Over-the-row machine vision system showed promise for accurate and reliable apple crop-load estimation in the apple orchards. In this paper we address the task of counting crop plants in a field using Convolutional Neural Networks (CNN). The number of plants in an Unmanned Aerial Vehicle (UAV) image of the field is estimated using regression instead of classification. This avoids to need to know (or guess) the maximum expected number of plants. We also describe a method to extract images of sections or “plots” from an orthorectified image of the entire crop field. These images will be used for training and evaluation of the CNN. Our experiments show that we can obtain a Mean Absolute Percentage Error (MAPE) as low as 6.7 with the Inception-v3 CNN architecture. Agricultural management increasingly uses crop maps based on classification of remotely sensed data. However, classification errors can translate to errors in model outputs, for instance agricultural production monitoring (yield, water demand) or crop acreage calculation. Hence, knowledge on the spatial variability of the classier performance is important information for the user. But this is not provided by traditional assessments of accuracy, which are based on the confusion matrix. In this study, classification uncertainty was analyzed, based on the support vector machines (SVM) algorithm. SVM was applied to multi-spectral time series data of RapidEye from different agricultural landscapes and years. Entropy was calculated as a measure of classification uncertainty, based on the per-object class membership estimations from the SVM algorithm. Permuting all possible combinations of available images allowed investigating the impact of the image acquisition frequency and timing, respectively, on the classification uncertainty. Results show that multi-temporal datasets decrease classification uncertainty for different crops compared to single data sets, but there was no “one-image-combination-fits-all” solution. The number and acquisition timing of the images, for which a decrease in uncertainty could be realized, proved to be specific to a given landscape, and for each crop they differed across different landscapes. For some crops, an increase of uncertainty was observed when increasing the quantity of images, even if classification accuracy was improved. Random forest regression was employed to investigate the impact of different explanatory variables on the observed spatial pattern of classification uncertainty. It was strongly influenced by factors related with the agricultural management and training sample density. Lower uncertainties were revealed for fields close to rivers or irrigation canals. This study demonstrates that classification uncertainty estimates by the SVM algorithm provide a valuable addition to traditional accuracy assessments. This allows analyzing spatial variations of the classifier performance in maps and also differences in classification uncertainty within the growing season and between crop types, respectively. A new image analysis algorithm based on mathematical morphology and pixel classification for grapevine berry counting is presented in this paper. First, a set of berry candidates represented by connected components was extracted. Then, six descriptors were calculated using key features of these components, and were employed for false positive (FP) discrimination using a supervised approach. More specifically, the set of descriptors modelled the grapes' distinctive shape, light reflection pattern and colour. Two classifiers were tested, a three-layer neural network and an optimised support vector machine. A dataset of 152 images was acquired with a low-cost smart phone camera. Images came from seven grapevine varieties, 18 per variety, at the two phenological stages in the Baggiolini scale between berry set (named stage K; 94 images) and cluster-closure (named stage L; 32 images). 126 of these images were kept for external validation and the remaining 26 were used for training (12 at stage L and 14 at K). From these training images, 5438 true false positive samples were generated and labelled in terms of the six descriptors. The neural network performed better than the support vector machine, yielding consistent Recall and Precision average values of 0.9572 and 0.8705, respectively. The presented algorithm, implemented as a smartphone application, can constitute a useful diagnosis tool for the in-the-field and non-destructive yield prediction and berry set assessing for the grape and wine industry.