Rice Diseases Detection and Classification Using Attention Based Neural Network and Bayesian Optimization

In this research, an attention-based depthwise separable neural network with Bayesian optimization (ADSNN-BO) is proposed to detect and classify rice disease from rice leaf images. Rice diseases frequently result in 20 to 40 \% corp production loss in yield and is highly related to the global economy. Rapid disease identification is critical to plan treatment promptly and reduce the corp losses. Rice disease diagnosis is still mainly performed manually. To achieve AI assisted rapid and accurate disease detection, we proposed the ADSNN-BO model based on MobileNet structure and augmented attention mechanism. Moreover, Bayesian optimization method is applied to tune hyper-parameters of the model. Cross-validated classification experiments are conducted based on a public rice disease dataset with four categories in total. The experimental results demonstrate that our mobile compatible ADSNN-BO model achieves a test accuracy of 94.65\%, which outperforms all of the state-of-the-art models tested. To check the interpretability of our proposed model, feature analysis including activation map and filters visualization approach are also conducted. Results show that our proposed attention-based mechanism can more effectively guide the ADSNN-BO model to learn informative features. The outcome of this research will promote the implementation of artificial intelligence for fast plant disease diagnosis and control in the agricultural field.


Introduction
Disease is a leading factor that affects the growth and development of rice. Rice diseases are causing a large quantity of loss in grain yield. FAO (Food and Agriculture Organization, USA) estimates that plant diseases cost the global economy around $220 billion annually and between 20 to 40 percent of global crop production is lost to diseases and pests (Agrios, 2005). Identifying the plant diseases rapidly and precisely is critical to reduce the economic losses. However, disease diagnosis and treatment usually require special skills and equipment, which farmers often don't have, thus resulting in a delay of treatment while waiting for experts to diagnose and treat the disease or due to misdiagnosis. The rapid advancement in mobile device and digital camera technologies have made it highly convenient for people to obtain digital images of crop plants and organs with diseases. Meanwhile, machine learning, deep learning, and computer vision technologies have made image processing and disease diagnosis become feasible as shown in many studies (Wang et al., 2019;Lu et al., 2018;Zhang et al., 2018).
Pesticide treatments are used as the main methods to control corp diseases. However, the use of pesticides highly depends on diagnosis results, and unrestrained pesticides usage causes considerable environmental damage as well (Wu et al., 2017). Therefore, precise and timely diagnosis of the diseases is critical. Currently, the diagnosis of rice diseases is often performed manually, and the process can be extremely labor intense and time-consuming. To overcome the drawbacks, computer-aided diagnosis is applied as an important tool in detecting and classifying rice diseases.
The convolutional neural network (CNN) is a type of popular deep neural networks (DNNs) that has demonstrated high generalization performance in many image analysis studies. For instance, an attention based deep residual network is proposed to detect the type of infection in tomato leaves (Tom, 2020). Performance of different deep learning based approaches is studied to classify skin diseases from colored digital photographs (Goceri, 2019d;Goceri and Karakas, 2020). Neural network modeling associated with adaptive neuro-fuzzy inference system (ANFIS) is used to describe and predict performance of a horizontal ground-coupled heat pump (GCHP) system (Esen et al., 2008(Esen et al., , 2009(Esen et al., , 2017. To obtain high performances from deep networks, appropriate loss functions, activation functions and batch sizes should be used (E and A., 2018;Goceri, 2019a). Challenges and recent solutions for image segmentation in deep learning are well-explained (Goceri, 2019b). Even though DNN techniques have been developed rapidly, seldom has been implemented for rice disease detection.
In other words, a proper model designated for rice disease classification is lacking. In this study, we focus on analyzing the extracted rice disease pattern from a deep learning model. An optimizationbased attention mechanism is developed to help the deep neural networks pay more attention to the critical features. In addition, our model is designed for running on portable devices, such as mobile phones and single board computers, which is a significant improvement from other studies. To extract diagnostic features, a large amount of rice leaf images are typically required in training deep learning models. In this research, we propose an attention mechanism to enhance the DNN model learning process to guide the model focus more on extracting more informative features.
Visualizing and understanding how the neural networks learning are quite essential. In practice, there are many helpful approaches, for instance, intermediate activation visualization, convolution filter visualization, heat map and saliency map. The aim of the feature visualization is to reveal how the model represents the visual world from its perspective. Additionally, it helps us to understand how a pre-trained convolutional network extracts feature, and show intuitively desirable properties such as compositionality, increasing invariance and class discrimination as the layers being ascended. These visualization techniques can also be used to debug problems with the model to obtain better results (Zeiler and Fergus, 2014).
The contributions of our work presented in this article are listed as follows: (1) a novel CNN model structure applying attention-based depthwise separable convolution is proposed and tested; (2) the proposed ADSNN-BO model outperforms all the tested state-of-the-art models in classification accuracy and other performance measurements with regard to rice disease classification mission; (3) the performance of ADSNN-BO model is analyzed and compared with other deep learning models using filter visualization approach. The structure of this paper is organized as following. Section 2 provides an overview of deep learning models and implementations in rice diseases. Section 3 describes different learning architectures as well as modifications for rice diseases classification. Experimental results and related performance evaluation of the deep learning models are illustrated in Section 4.
Approaches used to extract samples and other image processing steps are also discussed in Section 4.
The research findings and future work are concluded in Section 5.

Literature Review
Multiple deep learning techniques have been developed and implemented with rice diseases detection over the past few years.  (Liang et al., 2019). They also established a rice blast disease dataset with the assistance of plant protection experts, which can be combined with other rice disease images to build a content-rich dataset. They conducted comparative experiment and applied t-SNE approach and found that the high-level features extracted by CNN were more discriminative and representative than local binary patterns histograms (LBPH) and Haar-WT (wavelet transform). Their quantitative analysis results indicated that CNN and CNN with support vector machine (SVM) had almost the same performance, which was better than that of LBP + SVM and Haar-WT + SVM.
Ramesh and Vydeki utilized the image processing techniques to minimize the reliance on the farmers to protect the agricultural products (Ramesh and Vydeki, 2020). They proposed a recognition and classification of paddy leaf diseases using optimized DNN with Jaya Algorithm. Their images of rice plant leaves were directly captured from the farm field for healthy, bacterial blight, brown spot, sheath rot and blast diseases. In preprocessing, for the background removal the RGB images were converted into HSV images and based on the hue and saturation parts binary images were extracted to split the diseased and non-diseased part. Classification of diseases is carried out by using Optimized They explored the RiceTalk project that utilized nonimage IoT devices to detect rice blast based on an IoT platform for soil cultivation. Unlike the image-based plant disease detection approaches, their agriculture sensors generated nonimage data that can be automatically trained and analyzed by the AI mechanism in real time. Their approach reduced the platform management cost significantly to provide real-time training and predictions. In their implementation, the accuracy of the RiceTalk prediction on rice blast is 89.4%.
Since machine learning techniques are commonly applied before deep learning becomes popular. The discoloration was considered as a potential risk to the rice-producing countries. Transfer learning is a method of learning in a new prediction task with the transferred knowledge from a related task that has already been learned. By utilizing InceptionV3 model of CNNs and transferred weights from ImageNet to train their collected data, the classification accuracy of 88.2% was achieved. Shrivastava seldom discussed in previous researches, this paper will concentrate more on feature visualization and understanding. More sophisticated deep learning models should rather be understood as an attempt to both search for the minimum necessary ingredients for recognition with CNNs (Springenberg et al., 2014). The connection between the gradient-based CNN visualisation methods and deconvolutional networks can be established via saliency maps (Simonyan et al., 2013).
Advanced optimization method is important for improving deep learning model performance. Similarly, the loss function used in a deep network affects performance of the model. Therefore, cross-entropy, Tversky similarity function or combination of them has been applied, such as in (Goceri, 2020), in the literature to solve classification problems. Bayesian Optimization provides an efficient technique based on Bayes Theorem to guide an optimization search problem. The algorithm functions by building a probabilistic model of the objective function which is searched efficiently with an acquisition function. The candidate samples are determined by being evaluated on the objective function (Snoek et al., 2012).
In summary, a list of related literature has been presented in Table 1 (Goceri, 2019c) to increase generalization performance of deep networks, they cause high computational costs. Therefore, Bayesian optimization has been implemented in this work.
CNN is a multi-layer neural network with a learning architecture which is usually consisted of two parts, including a feature extractor and a trainable classifier. In practice, a CNN learns the values of filters by itself during the training process. More precisely, a convolution layer attempts to learn filters in a three-dimensional space, with two spatial dimensions and a channel dimension. Therefore, a single convolution kernel is conducted with simultaneously mapping cross-channel and spatial correlations.
MobileNet was proposed as an efficient CNN architecture to achieve very small, low latency models that can easily fit to the design requirements for mobile and embedded vision applications (Howard et al., 2017;Göçeri, 2020). The MobileNet model is based on a streamlined architecture using depthwise separable convolutions to build DNNs with light weights. Compared with a standard convolution, a depthwise separable convolution consists of a depthwise convolution and a 1 × 1 pointwise convolution.
The conventional convolution combines inputs into a new set of outputs within one step. The depthwise separable convolution splits this into two layers, a separate layer for filtering and another for combining.
To illustrate, a conventional convolutional layer takes D F × D F × M as an input and produces a D G × D G × N output feature map, where D F is the spatial width and height of an input feature map, M is the number of input channels, D G is the spatial width and height of an output feature map, and N is the number of output channels. The conventional convolutional layer is represented by convolution From the computational cost perspective, standard convolutions have the computational cost of, MobileNet architecture addresses each of these terms and their interactions. Depthwise separable convolutions is used to break the interaction between the number of output channels and the size of the kernel. In this way, the model will learn information within one particular channel and interactions between the channels separately. Depthwise separable convolution involves two kinds of layers: 1) depthwise convolutions and 2) pointwise convolutions. The Depthwise convolution refers to a single filter applied for each input channel. Pointwise convolution refers to a simple 1 × 1 convolution which is used to establish a linear combination of the output of the depthwise layer. Depthwise convolution has a computational cost of, Depthwise convolution performances well inside each particular channel, and pointwise convolution is then employed to address the lack of the relations between channels. Thus, an additional layer that creates a linear combination of the output of depthwise convolution via 1 × 1 convolution is added to generate new features. Overall, depthwise separable convolution has a computational cost of, which refers to the summation of the depthwise and pointwise convolutions.
By presenting convolution as a two-step process with filtering and combining. A reduction in computational cost can be achieved as, Chollet proposed a CNN architecture fully based on depthwise separable convolution layers called Xception. As an improvement regarding to the Inception model, the hypothesis was made that the mapping of cross-channels and spatial correlations in the feature maps can be entirely disengaged (Chollet, 2017). The original Xception model possesses 36 convolutional layers forming the feature extraction base. Compared with the Inception module, it is noted that there are two minor differences.
One is the order of the operations. In Xception module, depthwise separable convolutions as usually are implemented to perform first channel-wise spatial convolution and then perform 1x1 convolution, however the Inception model performs the 1x1 convolution first. The other is that the presence or absence of a non-linearity after the first operation. In Inception module, both operations are followed by a ReLU non-linearity, whereas for Xception, depthwise separable convolutions are usually implemented without non-linearities.
To overcome the limitations of trading off accuracy to reduce size and latency, we propose a new model based on depthwise separable neural network embedded with augmented attention mechanism.
We take the architecture of MobileNetV1 as the bare-bones of the model, and apply attention layers between the last convolution block and the average pooling layer. We keep the original MobileNet structure as well as the hyperparameters settings. The detailed architecture is shown in Figure 1.
Each convolution block is specified in the figure. In addition, the outputs of corresponding convolution operations are indicated.
Neural network models are generally referred to as being opaque. This means that they usually fail to explain the reason why a specific decision or prediction was made. CNNs are designed to work with image data, and their structure and function suggest that should be less inscrutable than other types of neural networks. Specifically, the models are comprised of small linear filters and the result of Corresponding Operations An efficient way to inspect the filters learned by convolutional networks is to display the visual pattern that each filter is meant to respond to (Francois, 2017). This can be conducted with gradient ascent in input space: applying gradient descent to the value of the input image of a convolutional network so that the response of a specific filter is maximized. The procedure usually starts from a blank input image. The resulting image will be one that the chosen filter is maximally responsive to. These filter visualizations reveal a lot about how convolutional layers learn. Each layer in a convolutional network learns a collection of filters so that their inputs can be expressed as a combination of the filters. Further visualization techiques such as heatmap can be followed up to explore and evaluate the model.

Augmented Attention Mechanism
Attention mechanism in deep learning has gathered widespread attention as a computational module for modeling sequences because of its ability to apprehend long distance interactions within a model.
Our attention augmented networks do not rely on pre-training of the fully convolutional counterparts and employ self-attention along the entire architecture. Additionally, the architecture is enhanced with the representational power of self-attention over images by extending relative self-attention to two dimensional inputs (Bello et al., 2019).
For a self-attention procedure, given an input tensor of shape (H, W, F in ), we flatten it to a matrix X ∈ R HW ×Fin and perform multi-head attention. The output of the self-attention mechanism for a single head h can be formulated as: where W q , W k ∈ R Fin×d h k and W v ∈ R Fin×d h v are learned linear transformations that map the input X to queries Q = XW q , keys K = XW k and values V = XW v . N h and d k respectively refer to the number of heads, the depth of values and the depth of queries and keys in multihead-attention. The outputs of all heads are then concatenated and projected again as follows: With an original convolution operator with kernel size k, F in input filters and F out output filters.
The corresponding attention augmented convolution can be expressed as: We employed the augmented convolution layers directly after conventional convolutions which are followed by average pooling computation. Similarly to the traditional convolution, the utilized attention augmented convolution is equivalent to translation and can smoothly work on inputs of different spatial dimensions. Effects on attention parameters such as the number of filters and the number of attention layers implemented are explored using Bayesian optimization approach.

Bayesian Optimization for Hyperparameter Tuning
Bayesian optimization works by constructing a posterior distribution of functions which is gaussian process that best describes the function to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and computing.
Bayesian optimization method consists of two major components: one is a Bayesian statistical model to model the objective function, and the other is an acquisition function to decide where to sample next (Frazier, 2018). In other words, presenting sampling points in the search space is achieved by acquisition function. The function will trade off exploitation and exploration. Exploitation refers to sample where the Bayesian statistical model predicts a high objective score, and exploration means where n is the number of observations.
In this research, we conduct hyperparameters tuning with regard to the embedded augmented attention mechanism. Particularly, the number of filters that are employed in each attention layer is optimized. For hyperparameter tuning with Bayesian optimization, we construct an objective function that takes the number of filters in each attention layer as input and returns a test accuracy score.
Throughout the optimization procedure, the best combination of the filter number for each attention layer can be determined.

Data Preparation
In this paper, we focus on three specific types of rice diseases, which are brown spot, rice hispa damage, and rice leaf blast. The typical manual diagnosis process is based on visualization symptom of each disease. Brown spots are round to oval, dark-brown lesions with yellow halo. As lesions enlarge, they remain round, with center area necrotic, gray and the lesion margin reddish-brown to dark brown.
Rice hispa damage scrapes the upper surface of leaf blades leaving only the lower epidermis. The disease also tunnels through the leaf tissues. When damage is severe, plants become less vigorous. The rice hispa damage can be directly detected by visualizing the bug on the leaf. For rice leaf blast, lesions varying from small round, dark spots to oval spots with narrow reddish-brown margins and gray or white center. Spots become elongated, diamond-shaped or linear with wit pointed ends and gray dead areas in the center surrounded by narrow reddish-brown. We conducted our study on a dataset with 2370 rice leaf samples in total, including all the three classes as well as the healthy rice samples (HuyDo, 2019). The detailed class information is provided in Table 2. 100 samples are selected in each category for training and testing. Additional image preprocessing methods including morphology, edge detection and cropping are involved in sample preprocessing.  Image preprocessing procedure is performed to address a few limitations of the dataset. Specifically, the white background occupies much image space in each image, and rice leaf samples are collected with different angles. Therefore, we first applied Otsu's thresholding approach to detect the leaf and cropped out the white background. Subsequently, we adjusted all the samples horizontally aligned.
This procedure is described in Figure 2.
Step 1 includes Otsu's threshold and opening morphology that would detect the shape of the rice leaf.
Step 2 refers to removing the unrelated background with the detected object and readjust the image position horizontally. In this way, useless information in the original samples is maximally excluded, which is beneficial for the learning performance. All the rice leaf images are then resized to 299×299 pixels to fix the deep learning model input size. The influence of different input size are also discussed in classification performance section.

Performance Measurement
The proposed ADSNN-BO model is applied to rice disease classification problem. In our exper- To measure the model classification performance, a multi-class confusion matrix is introduced (Wang et al., 2019). Given a problem with M classes, a M × M matrix is constructed, where n ij denotes the number of samples with the actual label i that are classified as the predicted label j.
Precision, recall, and F1-score of each class can be calculated based on the returned confusion matrix.
Those measurement equations are indicated in Eq.
[8]- [10]. Although Matthews correlation coefficient is identified to perform better for binary classification, it is not suitable for our multi-class classification tasks (Chicco and Jurman, 2020). The experimental results evaluated based on the performance measurement are shown in Table 3. The table reveals   Compute the gradient of the input picture with regard to this loss. Normalize the gradient. Return the loss and gradients given the input picture. Run gradient ascent for i steps. Utility function to convert a tensor into a valid image to get the output.

Feature Evaluation
Features that deep learning models extract are evaluated based on two major approaches: (1) activation map, and (2) filters visualization. Activation map, considered as the most straight-forward visualization technique, will provide a visual representation of the activation numbers within different layers of the neural network. In order to generate a series of activation maps throughout running the model, we first load and fix the weights of entire network with the best performed model. Then an original rice image sample is input into the model for testing. After each layer's operation, the output will tell what type of input maximally activates the layer. A series of selected activation maps are generated for our proposed ADSNN-BO model in Figure 5. The figures reveal how the model deals with different rice diseases. For example, it is shown that the network captures and extracts the spot pattern successfully when detecting brown spot disease. In addition, activation maps of all the filters within one layer are presented in Figure 6. According to the results, it is apparent that most of the filters are getting activated with regard to the pattern for each disease, and various types of patterns are activated for different disease categories. The blank convolution outputs mean that the pattern encoded by the filters were not found in the input image. These patterns must be complex shapes most likely that are not present in the input image.  Table 1. The results of other disease categories are similar according to the visualization.

Conclusions
In this paper, an overview of deep learning models and implementations in rice diseases is provided.
Six existing deep learning models have been implemented and compared with each other for rice disease classification problem. Pre-trained ImageNet weights are used to provide better results for all CNN architectures. A new model named ADSNN-OB has been proposed,tested, and discussed.
The details of data preparation steps are introduced. Various experiments are conducted which show that our proposed ADSNN-OB model outperforms other deep learning models in various scenarios.
Feature evaluation including activation map and filters visualization is also implemented to illustrate the performance of the model. The results show that a new model which outperforms all tested state-of-the-art models and fits well with the mobile device environment is created.
In the next step, we plan to explore more on our proposed ADSNN-OB model with different optimization methods as well as hyperparameter tuning to improve the performance and make the procedure more effectively. Also, since the dataset we used is different from those in the earlier literature, we tend to apply the model on different public datasets. Integrating location, weather and soil information along with the given rice disease image can be interesting to investigate for a more comprehensive and automated rice disease detection and classification mechanism.