Hyperspectral Data Augmentation

Data augmentation is a popular technique which helps improve generalization capabilities of deep neural networks. It plays a pivotal role in remote-sensing scenarios in which the amount of high-quality ground truth data is limited, and acquiring new examples is costly or impossible. This is a common problem in hyperspectral imaging, where manual annotation of image data is difficult, expensive, and prone to human bias. In this letter, we propose online data augmentation of hyperspectral data which is executed during the inference rather than before the training of deep networks. This is in contrast to all other state-of-the-art hyperspectral augmentation algorithms which increase the size (and representativeness) of training sets. Additionally, we introduce a new principal component analysis based augmentation. The experiments revealed that our data augmentation algorithms improve generalization of deep networks, work in real-time, and the online approach can be effectively combined with offline techniques to enhance the classification accuracy.


I. INTRODUCTION
Hyperspectral satellite imaging (HSI) captures a wide spectrum (commonly more than a hundred of bands) of light per pixel (forming an array of reflectance values). Such detailed information is being exploited by the remote sensing, pattern recognition, and machine learning communities in the context of accurate HSI classification (elaborating a class label of an HSI pixel) and segmentation (finding the boundaries of objects) in many fields [1]. Although the segmentation techniques include conventional machine learning algorithms (both unsupervised [2] and supervised [1], [3]), deep learning based techniques became the main stream [4]- [11]. They encompass deep belief networks [4], [7], recurrent neural networks [8], and convolutional neural networks (CNNs) [5], [6], [9]- [11].
Deep neural networks discover the underlying data representation, hence they do not require feature engineering and can potentially capture features which are unknown for humans. However, to train such large-capacity learners (and to avoid overfitting), we need huge amount of ground truth data. Acquiring such training sets is often expensive, timeconsuming, and human-dependent. These problems are very important real-world obstacles in training well-generalizing models (and validating such learners) faced by the remote sensing community-they are manifested by a very small number of ground truth benchmark HSI sets (there are approx. 10 widely-used benchmarks, with the Salinas Valley, Pavia University, and Indian Pines scenes being the most popular).
To combat the problem of limited, non-representative, and imbalanced training sets, data augmentation can be employed.
It is a process of synthesizing new examples following the original data distribution. Since such enhanced training sets can improve generalization of the learners, data augmentation may be perceived as implicit regularization. In computer vision tasks, data augmentation often involves simple affine (e.g., rotation or scaling) and elastic transforms of the image data [12]. These techniques, albeit applicable to HSI, do not benefit from all available information to model useful data.

A. Related literature
The literature on HSI data augmentation is fairly limited (not to mention, only one of the deep-learning HSI segmentation methods discussed earlier in this section used augmentation-simple mirroring of training samples-for improved classification [10]). In [13], the authors calculated perspectral-band standard deviation (for each class) in the training set. The augmented samples are later drawn from a zero-mean multivariate normal distribution N (0, αΣ), where Σ is a diagonal matrix with the standard deviations (for all classes) along its main diagonal, and α is a hyper-parameter (scale factor) of this technique. Albeit its simplicity, this augmentation was shown to be able to help improve generalization.
Li et al. utilized both spectral and spatial information to synthesize new samples in their pixel-block augmentation [14]. Two data-generation approaches: (i) Gaussian smoothing filtering alongside (ii) label-based augmentation were exploited in [15]. The latter technique resembles weak-labeling [16], and builds on an assumption that neighboring HSI pixels should share the same class (the label of a pixel propagates to its neighbors, and these generated examples are inserted into the training set). Thus, it may introduce wrongly-labeled samples.
Generative adversarial networks (GANs) have already attracted research attention in the context of data augmentation due to their ability of introducing invariance of models with respect to affine and appearance variations. GANs model an unknown data distribution based on the provided samples, and they are composed of a generator and discriminator. A generator should generate data samples which follow the underlying data distribution and are indistinguishable from the original data by the discriminator (hence, they compete with each other). In a recent work [17], Audebert et al. applied GAN conditioning to ensure that the synthesized HSI examples (from random distribution) belong to the specified class. Overall, all of the state-of-the-art HSI augmentation methods are aimed at increasing the size and representativeness of training sets which are later fed to train the deep learners.

B. Contribution
In this letter, we propose a novel online augmentation technique for hyperspectral data (Section II-A)-instead of arXiv:1903.05580v1 [cs.CV] 13 Mar 2019 synthesizing samples and adding them to the training set (hence increasing its size which adversely affects the training time of deep learners), we generate new examples during the inference. These examples (both original and artificial) are classified using a deep net trained over the original set, and we apply the voting scheme to elaborate the final label. To our knowledge, such online augmentation has not been exploited in HSI analysis so far (test-time augmentation was utilized in medical imaging [18], where the authors applied affine transforms and noise injection into brain-tumor images for better segmentation). Also, we introduce principal component analysis (PCA) based augmentation (Section II-B) which may be used both offline (before the training) and online. This PCAbased augmentation simulates data variability, yet follows the original data distribution (which GANs are intended to learn [17], but they are not applicable at test-time).
Our rigorous experiments performed over HSI benchmarks revealed that the online approach is very flexible-different augmentation techniques can be exploited in this setting (Section III). The results obtained using a spectral CNN indicated that the test-time augmentation significantly improves abilities of the models when compared with those trained using the original sets, and augmented using other algorithms (also, we compared our CNN with a spectral-spatial CNN from the literature whose capacity is much larger [11]). Our online approach does not sacrifice the inference speed and allows for real-time classification. We showed that the proposed PCA augmentation is extremely fast, and ensures the highest-quality generalization of the deep models for all data-split scenarios. Finally, we demonstrated that the offline and online augmentations can be effectively combined for better classification.

II. PROPOSED HYPERSPECTRAL DATA AUGMENTATION A. Online Hyperspectral Data Augmentation
Our online (test-time) data augmentation involves synthesizing A artificial samples x 1 , x 2 , . . . , x A for each incoming example x during the inference. We traverse the neighborhood of the original example and try to mitigate potential inputdependent uncertainty of the deep model. In contrast to the offline augmentation techniques, the test-time augmentation does not cause increasing the training time of the network, and it does not require defining the number of synthesized samples beforehand (also, for the majority of specific augmentation algorithms, the operation time of a trained learner would not be significantly affected since the inference is fast). We build upon the theory of ensemble learning, where elaborating a combined classifier (encompassing several weak learners) delivers high-quality generalization (it is an efficient regularizer). Here, by creating A artificial data points, we implicitly form a homogeneous ensemble of deep models (trained over the same training set T ). The final class label is elaborated using majority voting (with equal weights) over all (A + 1) samples (for low ensemble confidence, when two or more classes receive the same number of votes, we perform soft votingwe average all class probabilities, and the final class label corresponds to the class with the highest average probability).
The proposed online HSI augmentation may be considered to be a meta-algorithm, in which a specific data augmentation method is applied to synthesize samples on the fly. Although we exploited the noise injection based approach [13], and our principal component analysis based technique (see Section II-B) in this work, we anticipate that other augmentations which are aimed at modifying an existent sample can be straightforwardly utilized here. Finally, the online augmentation may be coupled with any offline technique (Section III).

B. Principal component analysis based data augmentation
In this work, we propose a new augmentation method based on PCA. Let us consider a training set T of N HSI pixels t i , where i = 1, 2, . . . , N , and each t i is b-dimensional (b denotes the number of bands in this HSI). PCA extracts a set of b (b b) projection directions (vectors) by maximizing the projected variance of a given b-dimensional dataset-the first principal component (P C 1 ) accounts for as much of the data variability as possible, and so forth. First, we center the data at the origin (hence, we subtract the average samplē is the matrix with the non-increasingly ordered eigenvalues along its main diagonal, and is the matrix with b corresponding eigenvectors (principal components) as columns. Finally, b principal components form an orthogonal base, and each sample t can be projected onto a new feature space: t = Φ T t. Importantly, each sample t can be projected back to its original space: The first step of our PCA-based data augmentation involves transforming all training samples using PCA (trained over T ). Afterwards, the first principal component 1 P C 1 (of each sample) is multiplied by a random value α drawn from a uniform distribution U(α min , α max ), where α min and α max are the hyper-parameters of our method (α is drawn independently for all original examples). This process is visualized in Fig. 1-we can observe that the synthesized examples (Fig. 1c) preserve the original data distribution (Fig. 1b) projected onto a reduced feature space, and preserve inter-class relationships. Finally, these samples are projected back onto the original space (using all principal components to ensure correct mapping), and they are added to the augmented T (if executed offline). This PCAbased augmentation can be applied in both offline and online settings (in both cases, PCA is trained over the original T ).

III. EXPERIMENTS
The experimental objective was to verify the impact of data augmentation on the deep model generalization. For online augmentation, we applied our PCA augmentation (PCA-ON), and noise injection (Noise-ON) [13], whereas for the offline setting, we used our PCA-based method (PCA), generative adversarial nets (GAN) [17], and noise injection (Noise) [13]. GAN cannot be used online, since it does not modify an incoming example, but rather synthesizes samples which follow an approximated distribution. Finally, we combined online and offline augmentation in PCA/PCA-ON (PCA augmentation is used to both augment the set beforehand, and to generate new samples at test time), and GAN/PCA-ON. For each offline technique, we at most doubled the number of original samples (unless that number would exceed the most numerous classin such case, we augment only by the missing difference). For online augmentation, we synthesize A = 4 samples, and for PCA and PCA-ON, we had α min = 0.9 and α max = 1.1.
We exploit our shallow (thus resource-frugal) 1D spectral CNN (Fig. 2) coded in Python 3.6 [19]. Larger-capacity CNNs require longer training and infer slower, hence are less likely to be deployed for Earth observation, especially on board of a satellite. The training (ADAM, learn. rate of 10 −4 , β 1 = 0.9, and β 2 = 0.999) stops, if after 15 epochs the validation set V (random subset of T ) accuracy plateaus. We train and validate the deep models using: (1) balanced T sets with random pixels (B), (2) imbalanced T sets with random pixels (IB), and (3) our patched sets (P) [19] (for fair comparison, the numbers of pixels in T and Ψ for B and IB are close to those reported in [11]). We also report the results obtained using a spectral-spatial CNN (3D-CNN) [11], trained over the original T (3D-CNN-in contrast to our CNN-does suffer from the training-test information leak problem, and the 3D-CNN results over B and IB are over-optimistic [19]). The results (over the test sets Ψ) obtained for the Salinas Valley, Indian Pines, and Pavia University datasets are gathered in Table I. Introducing augmented samples (in both online and offline settings) helped boost generalization abilities of the deep models in the majority of cases (even up to more than 8% of OA for GAN, PCA, and PCA/PCA-ON in Salinas, B; only the Noise offline augmentation deteriorated both OA and AA for P). Interestingly, exploring the local neighborhood randomly (Noise and Noise-ON) can notably deteriorate OA and AA. It usually occurs for under-represented classes (e.g., C7 in Pavia) since their examples lay close to other-class examples in the discovered feature space (therefore, they can be easily "confused" with each other). This problem is addressed by the data-distribution analysis in our PCA-based augmentations. Coupling offline and online augmentation (PCA/PCA-ON and GAN/PCA-ON) gave consistent high-quality results over all sets and all training-test splits, and dealt well with the HSI imbalance (in P, we did not ensure that examples of all classes are included in the original T , thus P is very challenging [19]).
To verify the statistical significance of the results (and see if the differences in the average per-class accuracy are important in the statistical sense), we executed two-tailed Wilcoxon's tests for each dataset split (B, IB, and P) over per-class AA for all HSI. The results reported in Table II show that applying HSI data augmentation is beneficial in most cases and delivers significant improvements in accuracy. GAN did equally well as e.g., PCA, PCA-ON, and our combined PCA/PCA-ON and GAN/PCA-ON for B, as Noise-ON, PCA-ON, and GAN/PCA-ON for IB, and as Noise-ON and PCA-ON for P. It indicates that employing time-consuming and complex deep-learning engines for data augmentation not necessarily brings larger improvements in the performance of the deep models.
Our combined approaches (PCA/PCA-ON and GAN/PCA-ON) were stable and consistently ensured high-quality generalization (as shown in Table I) of the deep models over all splits. This stability is also manifested in Table III, where we summarize the results across all sets (although PCA gave the best accuracy for B, the differences between PCA and PCA/PCA-ON and GAN/PCA-ON are not statistically significant). We can appreciate that our PCA-based augmentation (offline, online, or combined) allowed us to obtain the best generalizationvery intuitive PCA-based data-distribution analysis for synthesizing samples outperformed or worked on par with GAN in the case of difficult (small and imbalanced) sets. Finally, our CNN surpassed the accuracy elaborated using a significantly larger 3D-CNN from the literature (with a bigger capacity) for P (note that the results obtained using 3D-CNN for B and IB are over-optimistic due to the intrinsic training-test information leak problem, hence they cannot be considered reliable [11]).    To gain better insights into the augmentation performance (and its potential overhead imposed on the deep models in terms of training and/or test times), we collected the average execution times of the most important steps of the investigated methods in Table IV. It can be observed that training of GANs is very time-consuming (it was executed using NVIDIA GeForce GTX 1060), and is of orders of magnitude higher than pre-processing in other offline techniques (PCA and Noise). Although all offline augmentations affect the training time of deep networks, these differences are not dramatic. Finally, the online augmentation allowed us to classify test pixels in realtime (note that we report the inference time in ms). IV. CONCLUSIONS In this letter, we introduced a new online HSI data augmentation approach which synthesizes examples at test time. It is in contrast to other state-of-the-art hyperspectral data augmentation techniques that work offline (i.e., before the deep-network training to increase the training set cardinality and representativeness). Our experimental study, performed over three HSI benchmark sets (with different training-test data splits) and coupled with statistical tests revealed that our online augmentation is very flexible (different augmentations can be applied here), improves the generalization abilities of deep neural networks, and works in real-time. Also, we showed that combining online and offline augmentation leads to consistently well-performing models. Finally, we proposed a principal component analysis based augmentation which operates extremely fast, synthesizes high-quality data, outperforms other augmentations for small and imbalanced sets, and is applicable in online and offline settings.