High intra-class diversity and inter-class similarity is a characteristic of remote sensing scene image data sets currently posing significant difficulty for deep learning algorithms on classification tasks. To improve accuracy, post-classification
 methods have been proposed for smoothing results of model predictions. However, those approaches require an additional neural network to perform the smoothing operation, which adds overhead to the task. We propose an approach that involves learning deep features directly over neighboring scene images without requiring use of a cleanup model. Our approach utilizes a siamese network to improve the discriminative power of convolutional neural networks on a pair
 of neighboring scene images. It then exploits semantic coherence between this pair to enrich the feature vector of the image for which we want to predict a label.
 Empirical results show that this approach provides a viable alternative to existing methods. For example, our model improved prediction accuracy by 1 percentage point and dropped the mean squared error value by 0.02 over the baseline, on a disease density estimation task. These performance gains are comparable with results from existing post-classification methods, moreover without implementation overheads. Remote sensing scene image analysis is emerging as an important area of research for application of deep learning algorithms. Application areas include land-use land-cover analysis, urban planning, and natural disaster detection. A deep learning task for labeling a scene image is typically formulated as conditional probability of the form in Eq. 1 Liu et al. (2019) , Albert et al. (2017) , Nogueira et al. (2016) , Castelluccio et al. (2015) , Mnih (2013) , Mnih & Hinton (2010) , where l i is label for image patch s i . This formulation is sufficient for problems where spatial situatedness of a scene, which embodies knowledge of semantic likeness between neighborhoods in the geophysical world, is not important. However, for problems which require knowledge of neighborhood the formulation in Eq. 1 becomes inadequate. An example of such a problem would be estimating disease density for a small geographical region of interest, in which case the probability of label l is likely to depend on the labels for neighboring regions due to semantic coherence among them. The problem of how to improve model prediction by leveraging semantic coherence among neighboring scene images has previously been considered in the literature. Previous studies consider the problem as a post-classification task. For example, Bischof et al. (1992) used a second classifier to do pixel smoothing to refine predictions made by another classifier. Based on a 5x5 window, a filter assigns pixels to the majority class if it had been assigned a different class. In Mnih (2013) , a post-processing architecture is suggested for incorporating structure into image patch prediction. It involves stacking neural networks (NN) such that the output from a previous one becomes input for the next. Idea is for each network to clean up predictions of previous one in order to progressively improve overall accuracy. While improved model performance was achieved by these methods, they have overhead of performing same classification task in at least two stages. In other words, you need a minimum of two NN to perform the same classification task. Unlike post-classification methods, this work considers the problem of improving model accuracy on scene images by exploiting knowledge of neighboring scenes as part of the model training process. We make the assumption that l is conditionally co-dependent on information embedded in scene image i and in other similar, neighboring image j such that the problem is formulated as probability of the form in Eq. 2, where s j is image for a neighboring tile that is most similar to index tile i and P (l i |S i , S j ) is observed probability distribution. We used Convolutional Neural Networks (CNN) for modeling the observed probability distribution in Eq. 2. A network architecture is proposed for training our model consisting of four components: a siamese sub-network, a similarity metric learning component, a convolutional network, and a decision layer. The siamese sub-network takes two neighboring scene images as input and extracts features from each. The similarity learning component evaluates how similar the input images are, using the extracted features. If the two input images are found to be similar the convolutional network learns additional features based on the merged feature vector, otherwise those from the index tile are used alone. We implemented the decision layer to perform classification or regression. A baseline model was implemented that takes a single image, the index tile i, as input. Empirical results show the proposed model consistently outperforms the baseline. In addition to improving predictive performance with a relatively small training set, our model is fast to train since it uses a pre-trained model for the siamese sub-network. Furthermore, it does not require another NN to smooth out its predictions as is the case with post-classification approaches, while achieving comparable performance gain. In summary,our contributions include the following. 1. We propose an approach for training a probabilistic deep learning model to improve prediction accuracy by exploiting semantic coherence between neighboring tiles in aerial scene images. A CNN architecture is suggested for this purpose. 2. We provide empirical evidence that demonstrates the viability of this approach on a disease density estimation task. 3. Lastly, we discovered an important limitation of the synthetic minority over-sampling technique (SMOTE). This method fails when used for oversampling an under-represented class whereby knowledge of spatial proximity between scene image data points must be preserved, an important requirement under the framework of learning deep features over neighboring scene images introduced in this work. Our model performed better than the baseline in both the classification and regression tasks for disease density estimation. For example, our model achieved 1 percentage point gain in accuracy over the baseline model. While a gain as result of deploying a siamese network to boost discriminative power of CNN for aerial scene image classification is consistent with findings in previous studies overall results from our model are poor. For instance, our model was only able to attain a maximum overall accuracy of 34 percent on the classification task. We would like to pin these poor results to a combination of three factors. First, the small data set used to train our model (12,070 images) could have impacted accuracy negatively, despite the use of regularization methods.It is therefore, possible that our model suffered the problem of overfitting. Secondly, our data set was unbalanced. It is likely that the extra parameter we introduced in the loss function to weight classes by giving higher importance to under-represented classes did not work as expected. The result could have been that our model did not learn all the necessary features required to make a prediction but rather could have resorted to guessing the output and hence, failing to generalize well over the test set. Class imbalance in our data set could also have negatively affected feature correlation which in turn could have reduced model performance. Besides, well-known methods for mitigating sample size bias in imbalanced data sets, for example by over-sampling under-represented classes Chawla et al. (2002) could not be applied directly to our data set without modifying the algorithm. That is because it was not immediately clear how to preserve spatial proximity between neighboring tiles, an idea that is central to learning deep features over neighboring scene images.However, despite the low overall performance by our model we have been able to demonstrate that it is possible to improve model accuracy by learning deep features over neighboring scene images in a disease density estimation task. Figure 2: Finding a neighboring image j that is semantically most similar to image i.
Approach for improving prediction accuracy by learning deep features over neighboring scene images in satellite scene image analysis.
{"src": "scitldr-AIC_train", "paper_id": "Skxn-JSYwr"}