Discriminative feature generation for classification of imbalanced data

The data imbalance problem is a frequent bottleneck in the classification performance of neural networks. In this paper, we propose a novel supervised discriminative feature generation (DFG) method for a minority class dataset. DFG is based on the modified structure of a generative adversarial network consisting of four independent networks: generator, discriminator, feature extractor, and classifier. To augment the selected discriminative features of the minority class data by adopting an attention mechanism, the generator for the class-imbalanced target task is trained, and the feature extractor and classifier are regularized using the pre-trained features from a large source data. The experimental results show that the DFG generator enhances the augmentation of the label-preserved and diverse features, and the classification results are significantly improved on the target task. The feature generation model can contribute greatly to the development of data augmentation methods through discriminative feature generation and supervised attention methods.


Introduction
Deep learning has achieved significant improvements in various tasks in computer vision applications using open image datasets containing a large amount of data. However, the acquisition of large datasets is a challenge in real-world applications, especially if they pertain to new areas in deep learning. For training deep neural networks, the quantity of data is as important as the quality because training with only a small dataset often results in degradation of the neural network performance or introduces an overfitting problem. Attempts have been made to establish a large training dataset in various fields; however, the annotation of a dataset still remains expensive, laborious, and timeconsuming. Furthermore, the distribution of classes in the dataset is often imbalanced. Training a network under class-imbalanced conditions can produce a detrimental effect on neural network performance and a biased classification result [1,2,3].
Data augmentation enhances the size and quality of the training dataset. Techniques range from simple data augmentation, such as flip, shift, and rotation, to deep generative models based on generative adversarial networks (GAN) [4], where a small training dataset is augmented at the input data level. Because GANs can approximate the distribution of the real input data and generate realistic samples from a generative model, many recent studies have shown that augmenting a small training dataset using a GAN can improve the classification performance in real applications [5,6,7,8,9,10]. However, such data augmentation at input level showed the limited improvement in the performance, because balancing the data distribution have week relation to the enhancement of feature extraction in the minority dataset. Also, the state-of-art GAN has the limited generating capability on large-scale images [11,12]. When the size of input image is large, these augmentation method is not applicable.
Unlike data augmentation, adversarial feature augmentation generates domain-invariant features, increasing the size of the minority classes in the feature space without considering the modality of the input data [13]. This augmentation was adopted in supervised learning, especially for the class imbalance problem [14]. Although the feature augmentation method performs better than the unsupervised domain adaptation method by generating domain-invariant and modality-free features, the generated feature maps from the outer fully connected layer do not have sufficient feature dimensions to represent the data distribution with a small amount of minority class data. In other words, generating features in convolutional layers has a greater capacity to improve classification performance in the case of the class-imbalanced conditions.
Taking these problems into consideration, we propose a novel discriminative feature generation (DFG) method using attention maps in the feature space. The proposed method is a combination of transfer learning and adversarial feature augmentation to complement their drawbacks. The baseline is transfer learning with regularizations, and the features are augmented using a GAN with the weight of the activation level of the feature maps in each class.
The framework is based on a modified GAN structure containing an independent classifier to improve the neural network performance. We extend the adversarial feature augmentation [15] in the following three manners: (1) Our GAN structure is composed of four independent networks: a feature generator, feature discriminator, feature extractor, and feature classifier. The more constituent networks enable the network to be trained for generating more effective features, especially for small sample data, in the feature space. (2) Similar to DELTA [13], we deploy regularizations to the feature extractor and the feature classifier for transfer learning to select the discriminative features from the outer layer outputs and transfer pre-trained knowledge from the large-scale source dataset. (3) In the training phase, we employ a feature generation method in a smallsized target dataset, which augments the selected discriminative features by adopting the attention mechanisms for each class label through supervised learning.
Our main contribution is a novel feature generation and augmentation using the proposed GAN structure to unravel the data imbalanced problem and improve neural network performance. In the proposed GAN structure, a feature extractor and a feature 2 classifier are included to train together with a feature generator and a feature discriminator. For the generation of meaningful features for classification of small-sized target data, transfer learning with regularization and class-wise attention is adopted. We evaluate the proposed method with various pairs of source and target dataset to show general applicability to classification in the class-imbalanced conditions. The remainder of this paper is organized as follows. In Section 2, related works are summarized. The framework and training procedure are presented in Section 3. In Section 4, the benchmark datasets for verification and the experimental results are detailed. Section 5 concludes the paper and provides scope for future research.

Related Work
GAN [4] is a generative model based on a min-max game theory scenario that pits two networks against one another. A generator network, G, competes against a discriminator network, D, that distinguishes between samples generated from G and samples from the training data. GAN can generate synthetic data close to the original; however, the training process of GAN has the instability of loss function convergence and the problem of mode collapse [16]. To avoid such problems, the Wasserstein GAN (WGAN) [17] and the WGAN with a gradient penalty (WGAN-GP) [18] using the Wasserstein-K distance as the loss function were proposed. Recently, GAN has been extended in several ways to control the generated properties [19,20,21] and its utilization, such as data augmentation [6,22,7].
Recently, GAN-based data augmentation methods were proposed to further improve the creation of augmented synthetic training data. Huang et al. [8] proposed stacked GAN (SGAN), which is trained to invert the hierarchical representations of a bottom-up discriminative network. Guo et al. [9] proposed a discriminative variational autoencoding adversarial network, which learns a latent two-component mixture distributor and alleviates the class imbalance for deep imbalanced learning. Cui et al. [23] proposed a class-balanced loss for long-tailed distributions. The class-balanced loss re-weights losses inversely with the effective number of samples per class. Although the generated synthetic data can balance the distribution between the classes, the classification performance is limited due to no guarantee enhancing the feature extraction ability of the classifier, when it is an independent network of the GAN. Even if GAN is trained with the classifier together like [24], GAN has the limitation to generate real-looking synthetic data on large-scale images. Not-qualified synthetic data is not able to improve the performance of the classification significantly.
Unlike GAN-based data augmentation at the input data level, Volpi et al. [15] proposed augmentation in the feature space. A feature extractor is trained with the source dataset under supervised learning, and then a feature generator for the unlabeled target dataset is trained in the convolutional GAN (CGAN) framework [19] against the feature extractor. Zhang et al. [14] developed a more general feature generation framework for imbalanced classification, inspired by the adversarial feature augmentation approach. These methods can generate domain-invariant features without considering the modality of the data. Consequently, they achieved better performance than unsupervised domain adaptation methods. However, the improvement of the classification under the classimbalanced condition is still not significant. The dimension of the features in the feature extractor is considerably lower than the input data. Such a low dimension may not be 3 sufficient to present the data distribution of a small amount of minority class data. The generated features should be domain-invariant and modality-free. Transfer learning is a machine learning method that transfers knowledge learned in a source task to a target task [25,26,27]. The weights of deep neural networks are first pre-trained on a large-scale dataset, which is called the source task, and then fine-tuned using the data from the target task with a small amount of data [26]. In the case of fine-tuning in convolutional neural networks, the weights in the first few convolution layers are fixed by pre-training. The last convolution layers are fine-tuned by the target task. During fine-tuning, the parameters of the target model can be driven far away from the pre-trained parameter values, leading to the incorporation of information relevant to the targeted problem and overfitting to the target task, so-called catastrophic forgetting [28]. This simple method cannot guarantee good performance because it may in many cases burden the network of the target task with irrelevant information. Yosinki et al. [29] quantified the transferability of features from each layer to learn transferable representations. For constraining catastrophic forgetting in inductive transfer learning, L 2 -norm regularization was proposed in [30]. The key concept of L 2 -SP is "starting point as reference" optimization, which tries to drive weight parameters to pre-trained values by regularizing the distance between the parameters of source and target tasks.
Instead of regularizing the weight parameters, deep learning transfer using a feature map with attention (DELTA), which is a regularized transfer learning framework, was proposed by Li et al. [13]. Inspired by knowledge distillation for model compression [31,32,33], DELTA employs the ideas of "inactivated channel re-usage" and feature map regularization with attention. These regularization approaches achieved significant improvement and alleviated the catastrophic forgetting problem by drawing weight parameters close to pre-trained values or aligning transferable channels in feature maps. It constrains the difference between the feature maps generated by the convolution layers of the source and target networks with attention. DELTA selects the discriminative features from the outer layer outputs using a supervised attention mechanism. To further improve the performance of the classification with a small training dataset or class-imbalanced dataset, a DFG method using attention maps in the feature space is proposed in this paper.

Discriminative Feature Generation
Our goal is to generate discriminative features to improve the performance of the neural network on a small number of training data and the class-imbalanced dataset.
Unlike CGAN and ACGAN [19,20], the structure of the feature classifier to the independent network is employed in the proposed GAN structure. The proposed GAN structure is composed of four independent networks: a feature generator, feature discriminator, feature extractor, and feature classifier. The structure of the proposed GAN model is shown in Figure 1. The feature extractor represents a first convolution layer or block in a neural network classifier, and the feature classifier represents the rest of the convolution layers and fully connected layers after the feature extractor in the neural network classifier. The feature extractor and classifier are regularized with transferred weights from the source model, similar to Li et al. [13]. We deploy the knowledge distillation technique to the training procedure of GAN and train the feature generator with supervised attention models to generate discriminative features. 4

Training GAN model with four independent networks
For the stability of the training procedure and the quality of the generated data, we deploy the objective formulation of WGAN-GP for the feature discriminator and the feature generator. The loss functions of the feature discriminator and the feature generator are denoted by L D and L G , respectively.
where z is the noise vector sampled from uniform distribution P z , and feature generator G generates synthetic feature G(z). λ is the penalty coefficient, and Px is the uniform sampling along straight lines between pairs of points from the real data distribution, P r , and the generated data distribution. θ D and θ G are the parameters of the feature discriminator and the feature generator, respectively. Feature discriminator D is trained to minimize L D to distinguish between real and generated features. Generator G is trained to minimize L G .
To generate the distribution of features similar to the real features from the feature extractor, a total loss function of the feature classifier contains a loss function of the 5 feature classifier on the features concatenating the real and generated features as well. The objective functions of the feature extractor and classifier are represented as follows: In (4), the loss functions of L r C , L g C , and L c C are defined as where L E and L C are the loss functions of the feature extractor and feature classifier, respectively. θ E and θ C are the parameters E and C, respectively. ⊕ denotes a concatenation operation. L c C is the loss function of the feature classifier on the feature maps concatenated with the real and generated features. α, β, and γ are hyperparameters that control the importance of the classification for the real and generated data. In L E and L r C , the regularization term, which characterizes the differences between the source and target network for transfer learning, can be added.
Feature generator G (to minimize (2) and (4c)) and feature classifier C (to minimize (4) and (4a)) are trained simultaneously, whereas feature discriminator D and feature extractor E are trained to minimize (1) and (3), respectively. The reason why feature classifier C is optimized not only with the real data is to prevent the classifier from overfitting to the real data because the performance of the classifier with the real data is not sufficient to improve the performance of the classification under the class-imbalanced condition.
In the training procedure, we define two training parameters. The generator learning parameter controls the ratio between L G and L g C while optimizing the generator parameter, θ G . The classifier learning parameter is used to control the balance of how much the classifier is trained from the feature extracted from the real data L r C or from the real features combined with the generated features, L C . The training details of the proposed method using (1), (2), (3), and (4) with these parameters are summarized in Algorithm 1.

Discriminative feature generation by using the supervised attention model
Algorithm 1 Training procedure of the proposed DFG method. We use the default values of λ = 10, n D = 5, n C1 = 2, n C2 = 10, andn W = 2000 Require: Batch size m, learning rate η, hyperparameter for weight sum ρ, hyperparameters α, β, and γ, and a threshold value for filter weight δ.

11:
Sample {x (i) } m i=1 ∼ P r a batch from the real data and labels {y (i) } m i=1 .

23:
for k = 1,..., n l number of class labels do 24: To generate discriminative features, we adopt supervised attention mechanisms for each class label. To obtain the weights for feature maps, we propose a supervised attention method adopted from [13] for the generator network. Whereas the supervised attention method in [13] is calculated by averaging the filter weight for each filter, we utilize the fact that the importance of each filter varies from class to class. We transform the filter weights for a single data into class-wise filter weights and deactivate channels with low filter weights.
In the supervised attention method from [13], the weights of the features are characterized by the performance loss when removing the convolutional filter for each feature from the feature extractor network. For a conv2d layer in the feature extractor and generator, the parameter form is a four-dimensional tensor with the shape of (c i+1 , c i , k h , k w ), where c i denotes the number of channels of the i-th layer and (k h , k w ) represents the size of the kernel. At the last convolution layer of the feature extractor, the dimension of the output feature maps is (c i+1 , h i+1 , w i+1 ), which is matched with the dimension of the output of the feature generator. We can measure significance through the performance reduction of the feature extractor and classifier when the filter in the last convolution layer is disabled in the feature extractor network. In other words, because it usually causes higher performance loss to remove a filter with a greater capacity for discrimination, we can improve the performance of the classification by generating feature maps focusing on channels with high filter weights. The filter weight is calculated through the feature extractor and classifier and is input into the feature generator. The feature generator generates weighted feature maps with deactivated feature maps using the filter weight. The deactivated feature maps are denoted in black in Figure 1.
The filter weight is expressed in (5), which is used to calculate the gap between the classification losses of the feature extractor and classifier on data (x, y) with and without the j-th filter in the last convolution layer of the feature extraction network.
where θ \j E denotes the modified parameter from θ E with all elements of the j-th filter set to zero.
Because each class usually has different importance of the filter channel, the calculated filter weights for a single data point can be transformed into class-wise filter weights.
where W j (E, c; θ E ) (W ∈ R nc×n f ) is a class-wise filter weight, c is a class label, n c is the number of images in the class, and n f is the number of filters. By multiplying the number of filters n f , we set the average value of the weights to 1. When transfer learning is used for the small training dataset and class imbalance problem, a filter weight on the source model can be obtained through feature extractor E S with parameter θ E S and classifier C S with parameter θ C S trained on the source dataset. Whereas the filter weight in [13] was calculated on the source model only, the proposed method updates the filter weight on the target model. In this case, the filter weight can be expressed in the form of 8 a weighted sum between the filter weight on the source dataset and the target dataset.
By applying the weight to the filters of the last layer in the feature generator, the feature generator can generate discriminative features by following the supervised attention models. The operation of the last layer in the original feature generator is expressed as follows.
Note that f l denotes the features of the l-th layer in the feature generator, norm() performs batch normalization, TransConv(f, θ) is a transpose convolution that takes f as the input with θ as a weight, and b l is a bias vector of the l-th layer. To generate discriminative features by using the class-wise weights of a filter, the operation can be rewritten as follows.
f l (z,ŷ) is a discriminative feature generated by the supervised attention model. To maximize the difference in the weights and inactivate unimportant filter channels, we set the weight values that are smaller than a threshold, δ, to zero and represent as W * j . δ is set to 0.95 of the average value of all filters for each class.
To evaluate the proposed method under the class-imbalanced condition, we adopt the step imbalance type described in [3]. The step imbalance assumes that the classes are divided into minority and majority groups. Then, classes in the same group have the same number of data points, making a step in the data distribution plot. In this evaluation, we set the number of majority classes as two and eight to evaluate under imbalanced conditions. All combinations of class imbalance cases were tested in the 9 evaluation. For small training datasets such as STL-10 and Caltech-256, we did not create an imbalanced condition. Under the class balanced condition, each experiment was repeated 10 times. SVHN: To test the transferability in the same domain, we set EMNIST as the source task and SVHN as the target task, where both are single-digit images. To reduce the image dimension gap, the SVHN images are converted to grayscale. For the imbalance 10:1 ratio set-up in training, the number of majority class images is 5000, and one of the minority class images is 500 in SVHN.
F-MNIST: To test the transferability in the different domains, we set EMNIST as the source task and F-MNIST as the target task which has images of fashion and clothing items. For imbalanced 40:1 ratio set-up in training, the number of majority class images is 6000 and that of minority class images is 150 in F-MNIST.
STL-10: STL-10 images are resized to 32 × 32 to match the image dimension of CIFAR-10. In this case, class imbalance is not applied, but a small training dataset where 500 images per class are used in STL-10.
CINIC-10: CINIC-10 dataset contains 9000 training images in each class and we sampled 900 images from each minor class for imbalance 10:1 ratio.
Caltech-256: Caltech 256 contains 257 object categories and 30607 images. In this study, we sampled 60 training samples for each category, referring to [30,13]. We resized the input images to 256 × 256, followed by data augmentation operations of random mirror and random crop to 224 × 224.
Food-101: Food-101 contains 101 food categories with 101000 images. A total of 750 training images and 250 test images were provided for each class. We sampled 250 training samples from minority classes for an imbalance of 5:1 ratio set-up during training.
All the experiments were implemented using Python scripts in the PyTorch framework and tested on a Linux system. Training procedures were performed on NVIDIA Tesla V100 GPUs. The training details can be found in Appendix A.

Results and comparisons
To establish a baseline for comparison, we denote "Original" as the performance of a classification model trained on the target dataset. "Original" is trained under the classimbalanced condition in the case of SVHN, F-MNIST, CINIC-10, and Food-101. In the case of STL-10 and Caltech-256, "Original" is trained under the class-balanced condition that contains a small number of train data. As mentioned, "Original" in Table 2 is trained without transfer learning and shows accuracies tested on the target test dataset achieved with classifiers trained on the target training dataset. For the class imbalance problem, in which classes are selected as majority classes, is an important factor in determining the performance of the neural network, and the deviation of result accuracies is large. Thus, accuracies under the class imbalanced condition show the mean and standard deviation.
The proposed method is compared with previous fine-tuning methods.
• Fine-tuning: Traditional transfer learning method by fixing weights before the last convolution block and fine-tuning the rest of them.
For a fair comparison with the other methods, we conducted experiments on the same neural network architecture for classification, such as LeNet5, VGGNet-16, and ResNet-50. Table 2 shows the results of the proposed DFG method compared to the finetuning, DELTA, and DIFA+cMWGAN methods. The results are the top-1 classification accuracies. The results in Table 2 show that our method based on DFG leads to accuracies higher than other methods for three different classifiers and six different benchmark datasets.

Qualitative results and ablation study
Although the proposed DFG method shows higher accuracy than the other methods, we would like to better understand the benefits brought by DFG and visualize how closely the proposed DFG generated the distribution of features resembled that of the original in two-dimensional spaces. We apply PCA [47] and t-SNE [48] analyses of real 11 and generated features. Figure 2 shows comparisons between the real and generated features. Different colors indicate different classes in the left part, and two different colors indicate real and generated features in the right part. From a qualitative point of view, real and generated features are distributed similarly, and the inter-class structure is preserved. Additionally, we analyzed the activation maps from the proposed DFG method using class activation maps [49]. From Figures 3 and 4, we observe that the regions of the target object have higher activation than that of the original classifier and fine-tuning. Compared with the activation maps from the DELTA, the proposed method improved the concentration and activation degrees.
We carried out an ablation study to better understand how the filter weight changes during the training procedure. We perform an experiment by changing the ρ value in (7). Figure 5 presents the distribution of the weights of two representative filters in the case of the SVHN dataset for two different hyperparameters: ρ = 0 and 0.75. The first row in Figure 5 represents the distribution of the weights of two representative filters with ρ = 0.75, and the second row represents the distribution of the weights of the same filters with ρ = 0. The red straight line is the line of equality for all filters. For instance, (a) if a weight value is placed above the line of equality, it means that the importance of the filter for the class is higher than the average of all filters. The case with ρ = 0 involves the training of the feature generator with the weights of the filters by updating from the feature extractor trained on the target dataset, and the case with ρ = 0.75 involves the training of the feature generator with the weights of the filters by two feature extractors: one is trained on the source dataset and the other is trained on the target dataset in the training procedure. We observed a significant difference between the two distributions of the weights. The distribution of the weights of the filters with ρ = 0 converges to the equality baseline, whereas the distribution of the weights of the filters with ρ = 0.75 maintains the characteristics of the weight for each class label. Our hypothesis of a possible cause of the difference is that the feature extractor trained on the target dataset tends to be trained to activate all filters and be overfitted to the target dataset. The feature extractor trained on the source dataset inactivates several convolution filters, and a small number of filters are activated. The average accuracy with ρ = 0.75 is 80.81% and that with ρ = 0 is 79.98%. The weights of the filters with ρ = 0.75, using the weight values from the feature extractor trained on the source dataset, prevent overfitting and help to generate discriminative features for each class label. Figure 6 shows the effect of hyperparameter ρ on the classification accuracy on the SVHN dataset using DFG. In this case, DFG with hyperparameter ρ = 0.75 outperformed that with other hyperparameters including the cases that the weights of the filters by the feature extractor only trained on the source or the target dataset. The test accuracy varies smoothly according to hyperparameter ρ, and combination of the two different weights of a filter yields a better accuracy. 15  Figure 6: Classification accuracy on SVHN using DFG by changing hyperparameter ρ.

Conclusions
In this paper, we proposed a novel DFG method using attention maps in the feature space. The structure of the feature classifier network of the independent network was employed in the proposed GAN structure. To generate discriminative features, we adopted supervised attention mechanisms for each class label and applied the mechanisms to the feature generator network for the filter weights and feature activation. The experimental results showed that the proposed DFG method improved the accuracy of the classification models of LeNet-5, VGG-16, and ResNet-50 by 4.24%, 5.52%, 14.73%, 13.95%, 42.99%, and 45.83% on six different benchmark datasets, respectively. Comparing to the transfer learning method, DELTA, and the other adversarial feature augmentation methods, the proposed DFG method provided the largest improvement in classification performance. Furthermore, we visualized the distribution of generated features by applying PCA and t-SNE and analyzed the activation maps from the proposed method using class activation maps. The qualitative results showed that the features generated by the proposed method are distributed similarly to the real features and the proposed method improved the concentration and activation degrees in the class activation maps.
In the future, we will extend the proposed method to real-world problems and contemplate further accuracy improvement in the feature generation method. Although the proposed method improved the performance of the classification under the classimbalanced conditions, we believe there is room for improvement as a gap still exists between our method and the balanced condition.