On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition

The Fine-Grained Visual Categorization (FGVC) is challenging because the subtle inter-class variations are difficult to be captured. One notable research line uses the Global Covariance Pooling (GCP) layer to learn powerful representations with second-order statistics, which can effectively model inter-class differences. In our previous conference paper, we show that truncating small eigenvalues of the GCP covariance can attain smoother gradient and improve the performance on large-scale benchmarks. However, on fine-grained datasets, truncating the small eigenvalues would make the model fail to converge. This observation contradicts the common assumption that the small eigenvalues merely correspond to the noisy and unimportant information. Consequently, ignoring them should have little influence on the performance. To diagnose this peculiar behavior, we propose two attribution methods whose visualizations demonstrate that the seemingly unimportant small eigenvalues are crucial as they are in charge of extracting the discriminative class-specific features. Inspired by this observation, we propose a network branch dedicated to magnifying the importance of small eigenvalues. Without introducing any additional parameters, this branch simply amplifies the small eigenvalues and achieves state-of-the-art performances of GCP methods on three fine-grained benchmarks. Furthermore, the performance is also competitive against other FGVC approaches on larger datasets. Code is available at \href{https://github.com/KingJamesSong/DifferentiableSVD}{https://github.com/KingJamesSong/DifferentiableSVD}.


INTRODUCTION
T HE Fine-Grained Visual Categorization (FGVC) aims to classify the subordinate categories from a given supercategory (e.g., birds [1], [2] or dogs [3]). Compared with ordinary classification tasks, the FGVC is very challenging due to the subtle inter-class variations but significant intraclass differences. It has a multitude of real-world applications, such as image captioning [4], food recommendation [5], and image retrieval [6]. Among the approaches for the FGVC task, the Global Covariance Pooling (GCP) methods, which exploit the second-order feature statistics, have achieved impressive performances on common FGVC benchmarks and have attracted research interests from the computer vision community [7], [8], [9], [10], [11], [12]. The GCP method is a spectral meta-layer that computes the sample covariance from convolutional features and conducts the matrix normalization to obtain more powerful representations. In general, a standard GCP meta-layer first uses Singular Value Decomposition (SVD) to factorize the sample covariance into the eigenvalue and eigenvector matrices. Afterwards, the matrix logarithm [13] or matrix square root [14] is performed on the eigenvalues for normalization. Finally, the normalized covariance is used as the global representation and fed into the FC layer for exploitation of second-order statistics.
In our previous conference paper [12], we conduct an investigation into the gradient smoothness of the SVD in the GCP layer. The SVD gradient involves the term K ij = 1 λi−λj where λ i and λ j are eigenvalues. Since the covariance size of GCP is very large (i.e., 256×256), it is very likely to have • Yue Song, Nicu Sebe, and Wei Wang   many similar and small eigenvalues, i.e., λ i ≈λ j . This will cause the gradient term moves towards infinity (i.e., K ij →∞) and trigger the numerical instability. To avoid this issue, a common practice is to truncate the small eigenvalues [12], [14], which could attain smoother gradients. We observe that when the small eigenvalues (i.e., last 50 out of 256) of the global covariance are truncated, the performance of GCP method on ImageNet [15] could get improvements and the training will be more stable. However, as can be seen from Fig. 1, on fine-grained recognition datasets truncating the small eigenvalues would make the model fail to converge. This intriguing phenomenon contradicts the common belief on the insignificant eigenvalues: as the eigenvalues decomposed by SVD are exponentially decayed, the majority of the matrix energy has been preserved well by the first few large eigenvalues. Truncating the small ones should not harm the compact representation of the data. For a given matrix P and the truncated one P k that keeps top-k eigenvalues, according arXiv:2205.13282v1 [cs.CV] 26 May 2022 to Eckart-Young-Mirsky theorem [16], we have: where σ i is the i-th largest eigenvalue, and d denotes the dimensionality. This is also known as the best low-rank approximation property of SVD. The theorem implies that truncating the insignificant eigenvalues still provides a very close approximation of the data (i.e., ||P − P k || F ≈0). In many practical applications such as image processing [17] and data mining [18], the noise of the data is usually hidden in the small eigenvalues. Therefore, truncating the small eigenvalues of large matrices has become a common practice. An intuitive explanation is that the large eigenvalues capture statistics of principal directions along which the feature variances are large. Small eigenvalues, on the other hand, correspond to the features with smaller variance. However, for fine-grained recognition datasets, the interclass feature differences (e.g., color of the bird head) are often very subtle. These features containing the classification clues are more likely to be encoded by the small eigenvalues. Unfortunately, due to the highly non-linear structure of CNNs, this assumption cannot be directly validated.
We propose two visual explainability methods to diagnose the behavior of eigenvalues and associated eigenvectors. One methodology is to back-propagate the gradients of eigenvalues to the input image, which will highlight the distinct regions crucial to specific eigenvalues. The other technique is to perturb the image such that the final representation only embeds the large or small eigenvalues. The obtained images are in a deep-dreamed style [19] and characterize the learned feature patterns. Through both quantitative evaluation and visual observation, the two aforementioned methods demonstrate that the small eigenvalues can effectively encode the discriminative and class-specific features. On the contrary, the large eigenvalues usually correspond to the background regions and fail in activating discriminative class-specific features compared with the small ones. As shown in Fig. 2, for the small eigenvalues, the backward-based input responses consistently have salient points falling on the object, and the class-relevant features emerge repeatedly in the perturbationbased visualizations.
Our explainability methods attribute the decision cues of fine-grained classification to the small eigenvalues. Existing GCP methods usually use matrix square root normalization to reduce the magnitude of large eigenvalues [7], [8], [20]. However, the importance of small eigenvalues is not fully enhanced and remains less exploited. These observations naturally raise the following question: Can we increase the significance of small eigenvalues to make existing GCP models focus more on the semantically meaningful features and therefore to help improving their performance on fine-grained visual recognition?
To solve this problem, we propose Scaling Eigen Branch (SEB), a general plug-in component for existing GCP models. It computes the exponential inverse of the covariance matrix that amplifies the relative significance of small eigenvalues. Then the covariance exponential inverse is used to generate a dynamic scaling factor and compensate for eigenvalues of the matrix square root. The backward gradient of the scaling factor helps the model to generate better-conditioned covariance matrices. The importance of small eigenvalues is thus magnified. Without introducing any additional parameters, the SEB improves the performances of GCP methods by 1.6% on three popular fine-grained benchmarks, i.e., Caltech University Birds (Birds) [1], Stanford Cars (Cars) [21], and FGVC Aircrafts (Aircrafts) [22]. Moreover, the GCP method equipped with our proposed SEB also has very competitive performances against recent transformer-based FGVC approaches on some larger benchmarks including Stanford Dogs (Dogs) [3] and iNaturalist (iNats) [23].
We summarize our contributions as follows: • We propose two visual explainability methods to diagnose the behavior of GCP eigenvalues, both of which demonstrate that small eigenvalues encode more class-specific features and make larger contributions to the decisionmaking process of GCP networks. • We propose Scaling Eigen Branch (SEB), an add-on network branch to amplify the importance of small eigenvalues. Without bringing any extra parameters, the proposed SEB achieves the state-of-the-art performance of GCP methods on three fine-grained benchmarks. On large fine-grained datasets, our method also has comparable performance against other FGVC approaches. This paper and our previous conference paper [12] are connected but different. In the experiments of [12], we identify the peculiar behavior of GCP models on fine-grained recognition datasets, i.e., the model cannot converge without the last few eigenvalues. In this paper, we perform additional substantial work to explain the behavior and propose a plugin network branch to improve the performances of GCP methods on the FGVC task.
The rest of the paper is organized as follows: Sec. 2 describes the related work in fine-grained classification, global covariance pooling, and visual explainability methods. Sec. 3 presents our proposed methods to attribute the eigenvalues and Sec. 4 introduces our SEB module that amplifies the small eigenvalues. Sec. 5 provides the experimental results and indepth analysis. Finally, Sec. 6 summarizes the conclusions.

Fine-grained Visual Categorization
The task of FGVC aims at distinguishing the subordinate categories of a given object category, e.g., birds [1], [2], cars [21], and dogs [3]. Different from common classification scenarios, the FGVC is more challenging because the subtle inter-class variation needs to be captured. Existing works can be roughly divided into two groups: localization-based methods [24], [25], [26], [27], [28], [29] and representationbased approaches [7], [8], [9], [10], [12], [14], [20], [30], [31], [32], [33], [34]. The former group exploits the part annotations to localize the semantically-relevant regions and assist the classification, whereas the latter category targets learning more powerful representations with either attentive information or high-order statistics to model the subtle inter-class details. Recently, several vision transformerbased works have been proposed to tackle the challenge of FGVC [35], [36], [37]. In [35], the authors design a feature fusion strategy to increase the representation power of the ordinary vision transformer [38] for the FGVC task. Thus, this work belongs to the representation-based category. The works of [36], [37] both design dedicated mechanisms to select the discriminative visual tokens or images patches to assist the FGVC, which can be considered as localizationbased methods. The GCP methods that explore the secondorder statistics are a particular kind of approach belonging to the representation-based category, and we will give a more detailed illustration in the following paragraph.

Global Covariance Pooling
In deep neural networks, Global Covariance Pooling (GCP) aims to explore the second-order statistics of convolutional features. DeepO 2 P [13] is the first end-to-end global covariance pooling network. It formulates the theory of matrix back-propagation and demonstrates its effectiveness in visual recognition and segmentation tasks. Another pioneering work is B-CNN [30] which proposes to aggregate the outer product of global features and perform element-wise power normalization. However, there exist two caveats in the two methods. First, the dimensionality of the covariance feature is too high, which significantly increases the parameters of the fully-connected layer and introduces the risk of overfitting. Secondly, the matrix logarithm normalization over-stretches the small eigenvalues and may not be effective enough. Based on the two pioneering works, the follow-up researches mainly proceed in three directions: • i) increase the representation power of global covariance by exploiting the distribution manifold or considering feature interactions in the convolutional layers [7], [10], [39]; • ii) reduce the dimensionality of the covariance feature [31], [32], [40]; • iii) seek for more efficient or effective matrix normalization schemes [8], [10], [14], [20]. The second direction is closely related to the robustness of eigendecomposition. As pointed in [41], large matrices are more likely to have gradient explosion problems, but this has been solved by [12]. It is also revealed that preserving the full dimension of covariance usually brings better performances. The second direction is therefore out of the scope of this paper. In this work, we mainly focus on the first and the last directions. To be more specific, we propose a plugin component to improve the effectiveness of the global covariance feature. Also, our proposed component, together with the matrix square root normalization, can be viewed as a special matrix normalization scheme.

Visual Explainability Methods
Although the CNN has achieved remarkable success in many computer vision tasks, it is often considered a black-box model and suffers from weak interpretability. There have been many attempts to improve CNN's explainability by visualizing the learned feature patterns or by highlighting the activation neurons. Existing visualization approaches can be roughly categorized into two families: backpropagationbased methods [42], [43], [44], [45], [46] and perturbationbased techniques [47], [48], [49], [50], [51], [52]. The former category designs dedicated back-propagation rules to selectively highlight the input activations, whereas the latter perturbs or occludes the input image to maximally activate or confuse the classification predictions. Our work is closely related to both categories. More specifically, we use both backpropagation-based and perturbation-based techniques to provide the hint of the impact of eigenvalues. To the best of our knowledge, no similar work has been done in interpreting the impact of eigenvalues of GCP networks.

ATTRIBUTING THE EIGENVALUES
In this section, we first revisit the procedure of global covariance pooling. Then, we introduce the two proposed visualization techniques in detail.

Global Covariance Pooling Recap
Consider the reshaped convolutional feature X∈R d×N , where d denotes the feature dimensionality (i.e., the number of channels) and N represents the number of features (i.e., the product of spatial dimensions of features), a GCP meta-layer first computes the sample covariance matrix as: whereĪ represents the centering matrix, I denotes the identity matrix, and 1 is a column vector whose values are all ones, respectively. The sample covariance matrices are always symmetric positive semi-definite. Such matrices do not have where U is the orthogonal eigenvector matrix, and Λ is the diagonal matrix in which the eigenvalues are sorted in a non-increasing order i.e., λ i ≥λ i+1 . Afterwards, the matrix square root is conducted for normalization: where the normalized covariance matrix Q will be fed to the fully-connected layer. Besides the matrix square root normalization, there exist some other normalization schemes such as matrix logarithm [13] and rank-1 update [20]. As the matrix square root is proved to amount to robust covariance estimation under the regularized MLE framework [7], this normalization is often preferred over other techniques. Existing state-of-the-art GCP methods are MPN-COV [7] and iSQRT-COV [8]. The MPN-COV [7] applies the above formulations for the covariance matrix computation. However, the standard SVD used in the MPN-COV is known to suffer from the gradient explosion problem as it involves the computation of 1 λi−λj where i =j. When λ i and λ j are very close or equal, the gradient will go to infinity. To avoid using SVD, the iSQRT-COV [8] proposes using Newton-Schulz iteration to directly derive the approximate matrix square root. Recently, the gradient explosion problem of SVD has been solved by [12], [41] and a faster scheme of computing matrix square root is proposed in [53]. Throughout the experiments in this paper, we mainly investigate MPN-COV [7] which is a standard GCP method that conducts explicit eigendecomposition, and we use the techniques of [12] to compute the gradients.
We take ResNet-50 models of MPN-COV [7] trained on the fine-grained benchmarks to diagnose the peculiar behavior of the eigenvalues.

Selective Back-propagation of Eigenvalues
The core idea of our backward method is to manipulate the eigenvalue matrix defined in eq. (3) and project the gradient back to the image. To visualize input activations to different eigenvalues, we could either abandon the small ones or the large ones from the eigenvalue matrix Λ: where t denotes the number of kept eigenvalues (i.e., top 206 out of 256), Λ L represents the matrix that only preserves the large eigenvalues, and Λ S is the matrix consisting of the small eigenvalues. The covariance matrix is usually of the size 256×256 and has 256 non-negative eigenvalues in total. According to our observation on fine-grained datasets, for most covariance matrices, the last 50 eigenvalues take less than 0.1% energy of the matrix (i.e., . Therefore, we categorize the last 50 eigenvalues as the small eigenvalue Λ S and the rest as the large eigenvalue Λ L .
Substituting the eigenvalue matrix Λ with Λ L or Λ S and back-propagating the gradients to the input could localize the salient regions that are crucial to the specific eigenvalues. To project the gradient back to the input, we need to consider the back-propagation rules of the GCP meta-layer and the standard CNN layers. For the sake of conciseness, we omit the back-propagation algorithm of the GCP meta-layer, and the readers are kindly referred to [7], [12], [54] for a detailed review. As for the gradient propagation of CNN, the main difficulty lies in the ReLU non-linearity. The vanilla backward gradient of ReLU [42] is calculated by: where p is the pixel-wise gradient back-propagated to ReLU. For the positive gradients, ReLU passes a constant to the next node in the computational graph, which loses the magnitude information and may not generate appealing visualizations.
To resolve this issue, DeConv [43] proposes an imputed back-propagation scheme: where the gradient magnitude is kept and more dynamic gradients are allowed. Compared with the vanilla gradient, DeConv gradient usually yields larger activated regions and more salient points, which could improve the visual appeal. We use both back-propagation rules to generate the input activations and they yield similar results.

Quantitative Evaluation
Correlation Coefficient. To quantitatively evaluate the importance of eigenvalues, we compute the average correlation coefficients between the input responses to the specific eigenvalues and those to all eigenvalues. The correlation coefficient is defined as: where A and B are images to be compared,Ā andB are the mean intensity of the images, and the correlation coefficient Corr(·, ·), which is normalized into the range of [0, 1], is used to assess their similarity. By measuring the similarity between the visualizations, we could understand what eigenvalues contribute more to the decision-making process. As the large eigenvalues using DeConv back-propagation rule have responses nearly everywhere on the image, the activated regions highly overlap with those using all the eigenvalues. Their correlation will be inevitably large. Mean Absolute Error. Alternatively, we could compute their Mean Absolute Error (MAE) to measure the statistical distance: where M and N denote the image width and height, respectively. The joint use of these two metrics can provide a comprehensive assessment on the similarities between the visualizations. Table 1 compares the similarity between the input activations of specific eigenvalues and those of all the eigenvalues. For both metrics, the activated regions of small eigenvalues have a substantially larger similarity with the activated regions of all the eigenvalues. This demonstrates that the small eigenvalues preserve more semantically meaningful information and have greater contributions to the decisions of the GCP networks. Validation Accuracy. Besides the analysis in the lens of visualization similarity, we also evaluate the impact of eigenvalues on the classification accuracy. Table 2 presents the validation accuracy using different subsets of the eigenvalues in the inference stage. Since the last eigenvalues are very small in magnitude, merely using the small ones might poorly reconstruct the covariance representation and cannot output reasonable class predictions. Therefore, we also add the first eigenvalue λ 1 to keep the dominant vector space when testing the performance of small eigenvalues. As can be seen, even using the last 30 eigenvalues has a higher validation accuracy than using the large ones. When the number of used eigenvalues is increased from 30 to 50, we can observe a steady performance improvement.
This demonstrates that the small eigenvalues have more predictive power and a larger contribution to the decisionmaking process.

Perturbation-based Methodology
Instead of seeking for input responses to eigenvalues, another visualization method is to freeze the classifier weights and perturb the input images such that only specific eigenvalues and the associated eigenvectors are maximally activated. After keeping updating the image values for a certain number of iterations, the perturbed image will highlight the learned feature patterns of specific eigenvalues. For an input image, we first decompose the covariance matrix into two submatrices that embed only the large or small eigenvalues:

Visualization of Large Eigenvalues
To visualize the feature patterns that correspond to the large eigenvalues, we could iteratively use the following loss function to perturb the image: where M denotes the covariance matrix generated by the perturbed image, l 1 is the loss to optimize the image, and || · || F represents the Frobenius norm. The loss can push the perturbed image to generate the covariance composed only by large eigenvalues. Notice that the loss l 1 can be regarded as the Mean Square Error (MSE) loss between P L and M.

Visualization of Small Eigenvalues
Deriving the perturbation loss for the small eigenvalues is more complex than the large ones. We cannot adopt the ||M − P S || 2 F loss as in eq. (11). Since the eigenvalues Λ S are sparse and exponentially smaller than the large ones, the reconstructed matrix P S contains little energy of the original matrix P (often <0.1%). It is infeasible for a covariance matrix M to have the identical eigenvalues as P S . Instead of pushing them to have the same eigenvalues, we enforce the eigenvalues of M and P S to be highly correlated. To achieve this goal, we introduce the following loss function: where the first term pushes M far from P L , and the second term makes sure that M stays close with P S . The composition of the two losses can guarantee that M maximally activates the eigenvectors of P S and has a highly correlated representation. Formally, we have the proposition as follows: Proof. Relying on Frobenius inner product, the loss l 2 is decomposed by: Since both ||P S || 2 F and ||P L || 2 F are independent of M, we can use a constant C to represent the sum of these two terms: For the inner product of a given matrix pair A and B, Von Neumann's trace inequality [55], [56] tells: where σ i (·) denotes the i-th eigenvalue. Injecting eq. (15) into eq. (14), the loss l 2 can be re-expressed as: Recall that P L is composed by the large eigenvalues Λ L ={λ 1 , . . . , λ t , 0, . . . , 0} diag and P S only embeds the small eigenvalues Λ S ={0, . . . , 0, λ t+1 , . . . , λ d } diag . Eq. 16 can be further formulated as: Since M is positive semi-definitive, there will not exist any negative eigenvalues σ i (M). Therefore, the minimum of the first term is zero. It happens when the eigenvalues of M have a zero correlation with Λ L , i.e., the large eigenvalues of M are zero.
The second term actually measures the correlation between the last (d−t) eigenvalues of M and P. When the r.h.s. of eq. (17) attains the minimum, the correlation will reach the maximum and M will maximally activate the eigenvectors associated with the small eigenvalues. The equality is taken when M also has the eigenvector matrix U.
This proposition shows that after the image is perturbed for certain iterations, i.e., the loss attains the minimum, M will only activate the learned feature patterns of the eigenvectors associated with small eigenvalues. But the eigenvectors that correspond to the large eigenvalues will not be encoded by M. Fig. 2 right visualizes the learned feature patterns that correspond to the specific eigenvalues. Although mainly mid-level and low-level features are activated in the visualizations, the semantically meaningful patterns can be observed. For the small eigenvalues, the visualizations exhibit distinct class-relevant features. Take the 2 nd row as an example, the bird head, which characterizes the bird species, emerges repeatedly in the visualizations. On the contrary, the large eigenvalues do not activate obviously classrelevant and human-interpretable features. Unfortunately, this visualization technique can only be assessed through visual observation but cannot be evaluated by any reasonable quantitative metrics. We provide more visualization results in Supplementary Material.

SCALING EIGEN BRANCH
The visualization results and quantitative analysis demonstrate that the small eigenvalues and associated eigenvectors are actually crucial to the network decisions as they capture rich semantic information and class-relevant feature patterns. This observation inspires us to propose SEB, a plug-in component dedicated to magnify the importance of small eigenvalues.

Amplifying Small Eigenvalues by Scaling
To magnify the importance of small eigenvalues, one straightforward approach is to increase their numerical values. However, the eigenvalues are sorted according to the varying extent of the feature along the eigenvector directions. Directly increasing the value of small eigenvalues could reverse the significance order and disturb the statistical information of covariance, which might make the model fail to converge. For the covariance matrix, it is important to maintain the significance order of the eigenvalues. Another intuitively possible method is to increase the relative importance of small eigenvalues (i.e., increase λ d λ1 ) but maintain the significance order (i.e. λi+1 λi ≤1 is always satisfied). This could be easily done by computing the matrix p-th root P 1 p instead of the square root P 1 2 . A small 1 p would punish the magnitude of large eigenvalues and magnify the significance of small ones. However, as pointed out in [7], only the matrix square root can amount to robust covariance estimation under the regularized MLE framework and approximately exploits Riemannian geometry under the Power-Euclidean metric [57]. Besides the theoretical analysis, the matrix square root also achieves the best experimental performances among different power normalization schemes (e.g., P 0.1 and P 0.3 ) [7]. All of these suggest that computing the covariance square root is indispensable for robust covariance pooling.
So far, the only practical approach seems to be properly transforming the covariance matrix such that the eigenvalues are amplified but the form of matrix square root still manifests. Since our covariance matrix is symmetric positive semi-definite (SPSD), it is appropriate to consider the geometry of SPSD manifold. In the space of SPSD manifold, the distance between covariance matrices is not measured by the traditional Euclidean metrics (i.e., d(X, Y )=||X−Y || 2 F ). Instead, a non-Euclidean metric should be adopted. One commonly used metric is the Log-Euclidean metric [58] defined as: We can easily find that this metric is scale-invariant, i.e., scaling two covariance matrices keeps their distance unchanged. Therefore, it is appropriate to consider scaling the covariance square root by a factor (>1) such that the eigenvalues are amplified and the distance between two scaled covariance matrices does not change in the SPSD manifold. Notice that the scale-invariance property generally applies for the non-Euclidean metrics used in the SPSD manifold (e.g., affine-invariant Riemannian metric [59] and Stein divergence [60]). In the following paragraphs, we will illustrate how the covariance matrices get appropriately scaled in our proposed SEB (see also Fig. 3).

Dynamic Scaling Factor
To scale the covariance square root Q, one may consider multiplying a constant a: a · Q = U{aλ 1 2 1 , . . . , aλ where {·} diag means transforming a vector to a diagonal matrix. It is non-trivial to choose a suitable constant a, as each covariance matrix has a different eigenvalue distribution. The scaling factor should accordingly be different and preferably dependent on the eigenvalues. An intuitive choice is using ||Q|| F or ||P|| F as the factor. However, both Q and P have very imbalanced eigenvalue distributions and are likely to be ill-conditioned (i.e., λ1 λ d is too large). In that case, the small eigenvalues will contribute little and the first eigenvalue λ 1 will dominate the scaling factor (i.e., ||P|| F ≈λ 1 and ||Q|| F ≈λ 1 2 1 ). To avoid this issue, we propose to first balance the eigenvalue distribution by calculating the exponential inverse of the covariance: Unfortunately, differentiating this step is not supported by the AutoGrad package of the deep learning frameworks. The gradient has to be manually derived. Given the partial derivatives of the loss l w.r.t S, we have dS=dUe −Λ U T +Ude −Λ U T +Ue −Λ dU T and de −Λ =−{e −λ1 ,. . .,e −λ d } diag dΛ. After some arrangements using the chain rule and matrix back-propagation rule [54], the partial derivatives of the loss l w.r.t the eigenvalue and eigenvector are calculated as: where (·) diag denotes the operation of setting off-diagonal elements to zero. The covariance exponential inverse S normalizes the eigenvalue distribution on (0, 1] but reverses the order of importance (see also Fig. 4). We then combine S and Q to compute their cross-covariance: Eq. (22) actually measures the similarity between Q and S. Compared with the ordinary covariance square root Q, it has a more balanced eigenvalue distribution. Thus we use its Frobenius norm ||QS|| F as the scaling factor to compensate for the covariance square root.

Covariance Square Root Compensation
After generating the scaling factor, we multiply it to the covariance square root plus the ordinary one. This process can be denoted by: where A is the final representation that will be fed to the fullyconnected layer. The value of the Frobenius norm ||QS T || F takes the following form: It is easy to find that the numerical value is in the range of (0, ||Q|| F ). We add the ordinary matrix square root to ensure that A can always amplify the eigenvalues of Q (i.e., ||QS T || F +1>1). Note that the norm ||QS T || F is differentiable and the gradient could help to increase the significance of small eigenvalues. Its derivatives w.r.t. Q and S are given by: For both matrices, the gradients are healthy as the use of S balances the eigenvalue distribution and reduces the 3: Comparison with the state-of-the-art GCP approaches. For a fair comparison, we report our training results of MPN-COV [7] and iSQRT-COV [8] using the same deep learning platform (PYTORCH). Notice that the values reported in these two papers are not based on the commonly used backbones with the first-order pooling layers (see Supplementary Material for details). The values of other methods are taken directly from their papers as they are less related to our approach.

Backbone
Method Birds [1] Aircrafts [22] Cars [21] VGG-16 [ risk of ill-conditioned derivative matrices. This could help the network to generate the better-conditioned covariance matrix P and the covariance square root Q, which implicitly amplifies the importance of the small eigenvalues.

Implementation Details
The code is implemented in PYTORCH. We run all the experiment on the workstation equipped with a GeForce GTX 1080 Ti GPU and a 6-core Intel Core i7-7800X@3.50GHz CPU. For MPN-COV [7] and our proposed SEB, the forward eigendecomposition is conducted on the CPU for a faster speed. The other operations are performed on the GPU.

Dataset Specifications
The Birds dataset includes 11, 788 images belonging to 200 bird species. The Aircrafts dataset contains 10, 000 images of 100 classes of airplanes, and the Cars dataset consists of 16, 185 images from 196 classes. Besides the three commonly used datasets, we also evaluate our method on two large datasets, namely Dogs and iNats. The Dogs dataset is comprised of 20, 580 images of 120 dog categories, and the INats dataset have 675, 170 images from 5, 089 natural finegrained categories that belong to 13 super-categories.

Perturbation-based Explainability Settings
The image is resized to 448×448 before being fed into the network. We keep the weights of the model frozen and only update the image pixel values. The learning rate to perturb the image is set to 0.1 and the perturbation lasts for 1, 000 iterations for each image. For the training recipe of different models on fine-grained benchmarks, please refer to Supplementary Material for details. Table 3 compares the validation accuracy on three benchmarks with different backbones against the standard GCP method (i.e., MPN-COV [7]), as well as iSQRT-COV [8] and other covariance pooling methods. We can observe that the standard GCP method equipped with our proposed SEB improves the performance of its original version [7] by 1.6% on average across different datasets regardless of the backbones. To be more specific, on Birds dataset, the performance is improved by 1.5% across backbones. On Aircrafts dataset, our SEB improves MPN-COV [7] by 1.5% on average. On Cars dataset, the average performance gain brought by SEB is 1.7%. Based on ResNet-152 [63] and EfficientNet-b5 [64], our method achieves the state-of-the-art performance of GCP methods on all datasets.

Results Compared with GCP Methods
The consistent performance gain demonstrates that our proposed SEB could be a powerful add-on for the GCP methods on the fine-grained recognition task. With the VGG backbone, our method is slightly inferior to RUN [20] and MoNet [33] on Aircrafts and Birds, respectively. That is mainly because the covariance of VGG architecture is of the size 512×512, which is twice larger than ResNet [63] and more prone to be ill-conditioned. However, there are two sides to reducing the number of channels. Projecting the channel dimension into 256 could reduce the covariance size but might weaken the representation power and might not necessarily lead to performance improvements. Besides, these methods do not report the experimental results on other backbones and their applicability on other deep architectures cannot be confirmed. It is also worth mentioning that our SEB does not introduce any additional parameters to MPN-COV [7] and only slightly increases the speed and memory consumption (See Sec. 5.6).
To sum up, the small eigenvalues play a vital role on the fine-grained classification task as they capture the semantic class-specific features, and amplifying their importance does bring consistent improvements. However, the small eigenvalues seem less important on the large-scale classification dataset (i.e., ImageNet [15]) in which the inter-class feature differences are no longer subtle and are more likely to be encoded by the large eigenvalues. The small eigenvalues might only capture the data noise and the network predictions would be largely dependent on large eigenvalues. For the detailed analysis, please refer to Supplementary Material which illustrates this point by a series of experiments including the explainability visualizations. Besides the evaluation within the scope of GCP methods, we also compare our method with other state-of-theart FGVC approaches on some larger datasets. Table 4 presents the validation accuracy of our approach and recent transformer-based FGVC methods on the three larger datasets, i.e., Cars [21], Dogs [3], and INats [23]. Our method outperforms other baselines by 0.7% on Dogs [3] and by 0.6% on INats [23], while the performance slight falls behind TransFG [37] by 0.4% on Cars [21]. This observation indicates that our method can also have very competitive performances against other FGVC approaches on large benchmarks.

Ablation Studies
Using ResNet-50 as the backbone, we conduct two ablation studies on the impact of using only SEB path and the impact of different scaling factors.  Instead of combining the matrix square root and the compensated one ||QS T || F ·Q+Q, one may consider only passing the amplified covariance square root ||QS T || F ·Q to the fully-connected layer. Table 6 compares the performances of using different paths. Solely using the SEB path ||QS T || F ·Q brings about 1.2% increase on the performances over MPN-COV [7]. The combination of the two paths can assure that the final representation always amplifies the covariance square root Q, thus outperforming using only SEB by 0.4% on average. To magnify the eigenvalues of Q, one may think about using other scaling factors instead of our used ||QS T || F . We take some possible factors to evaluate the impact, including the fixed constant, ||Q|| F , ||S|| F , and factors learned by an MLP from Λ 1 2 and Q. As shown in Table 7, none of the alternative factors bring obvious improvements and some even have side effects on the performances. We conjecture that this is related to the backward gradient of the scaling factor. Using a constant factor and ||S|| F cannot propagate valid gradients to Q. For the scaling factor ||Q|| F , its derivative w.r.t. Q is Q/||Q|| F , which is prone to be illconditioned. For the learned scaling factor, the value is   [7] and our SEB on three fine-grained benchmarks. The proposed SEB effectively narrows the eigenvalue range and magnify the small ones. (Bottom) The condition number κ(P) of MPN-COV [7] and our proposed SEB on the fine-grained benchmarks. Our covariance matrices are consistently better-conditioned than MPN-COV [7] and the relative importance of small eigenvalues are amplified.

Impact of Scaling Factors.
unbounded and the gradient can not be constrained in good conditions. By contrast, our ||QS T || F consistently has well-conditioned gradient matrices and outperforms other schemes significantly. Table 8 compares the time and memory consumption of the GCP models for each mini-batch. Compared with the ordinary MPN-COV [7], our proposed SEB only brings marginal overhead on the computation complexity and memory cost. The extra memory usage is due to the calculation and storage of the scaling factor in the proposed SEB path.  As can be observed, our proposed SEB consistently brings the performance gain about 2% over MPN-COV [7] and iSQRT-COV [8] during the training process. Besides, the original GCP methods show some signs of overfitting, i.e., their validation accuracy seems to slowly decrease as the training goes in the later stage. In contrast, our SEB appears to continuously improve throughout the training.

Eigenvalue Distribution Comparison
In Sec. 4.2, we show that the backward gradient of ||QS T || F are healthy and can help the network to generate betterconditioned covariance matrices. To validate the concrete impact, we compare the eigenvalue distributions of our proposed SEB method and the ordinary GCP method MPN-COV [7]. The comparison is made by visualizing the histogram of the eigenvalues and by evaluating the matrix condition number. The matrix condition number κ(·) is defined as: where X is the matrix to be measured. The condition number computes the ratio between the maximum and minimum eigenvalues. The lower the ratio is, the higher the relative importance of small eigenvalues will be. The histogram displays the overall eigenvalue distribution for all covariance matrices, whereas the condition number measures the conditioning of each individual matrix. The joint use of these two evaluation methods can provide a comprehensive picture of the eigenvalue information. Fig. 6 top shows the histogram of eigenvalues distribution for the global covariance P. As can be observed, our SEB effectively narrows the eigenvalue range. The large eigenvalues are reduced and the small ones are increased. Interestingly, on all datasets, a large portion of small eigenvalues group together to form a peak around 2 −18 . We expect that these eigenvalues and associated eigenvectors are the determinant factors that capture the rich semantic information and boost the performances of the GCP methods. The condition number of all the covariance matrices κ(P) are plotted in the bottom of Fig. 6. For MPN-COV [7], the value varies from 10 8 to 10 15 , whereas the condition number of our SEB is on [10 7 , 10 8 ]. This demonstrates that our covariance matrices are consistently much better-conditioned than MPN-COV [7].

CONCLUSION
In this paper, we propose two explainability methods to diagnose the eigenvalue behaviors of GCP layers. Both quantitative evaluation and visual observation demonstrate that the small eigenvalues capture more class-relevant and semantically meaningful feature patterns. Based on this finding, we further propose a plug-in network branch to magnify the importance of small eigenvalues and the associated eigenvectors. Without bringing extra parameters, the existing GCP method integrated with our proposed component achieves state-of-the-art performances of GCP methods on fine-grained benchmarks. Moreover, on larger datasets, our method also has competitive performances against other FGVC approaches.

A.1 Training Recipe for Fine-grained Benchmarks
The ResNet and VGG pre-trained on ImageNet [15] are finetuned on each fine-grained dataset. The images are first resized to the resolution of 512×512, then centered cropped to 448×448, and finally fed into the network. The 1000-d fully-connected layer of the original network is changed to fit the number of classes. The model is trained using SGD with momentum 0.9. The batch size is set as 8 for Aircrafts [22] and set as 10 for Birds [1] and Cars [21]. We make the inference also on the 448×448 centered crop of the test image. Since the covariance matrix is symmetric, we only pass the upper triangular part to the fully-connected layer.

A.1.1 ResNet Settings
For the ResNet architectures, we squeeze the channels of the final convolutional feature from 2048 to 256 for computation efficiency of covariance matrices. The resultant spatial dimension of the covariance is 256 × 256. The initial learning rate is set as 3×10 −3 for the convolutional layers. The learning rate of the fully-connected layer is set 5, 10, and 20 times larger than the convolutional layers for Aircrafts [22], Cars [21], and Birds [1] respectively. The weight decay of the optimizer is set as 1×10 −3 for Aircrafts [22] and as 1×10 −4 for Birds [1] and Cars [21]. For Aircrafts [22] dataset, the training lasts for 50 epochs with dividing the learning rate by 10 at epoch 20. For Cars [21] and Birds [1] datasets, the training lasts for 100 epochs and the learning rate dividing happens at epoch 50.

A.1.2 VGG Settings
For the VGG architecture, the final representation has 512 channels, which is much smaller than ResNet and will lead to the covariance matrix of a acceptable size. Therefore, we do not perform any channel reduction and the resultant covariance matrix is of size 512 × 512. For Cars [21] dataset, the initial learning rate is set as 6×10 −3 for the fullyconnected layer and 1.2×10 −3 for the other layers. We set the initial learning rate to 3×10 −3 for Aircrafts [1] and Birds [1]. Batch normalization is used to stabilize the training process. The training lasts 150 epochs for all the datasets with periodically learning rate decay by 10 at epoch 50 and epoch 100.
For the concern of comparison fairness, the same training protocol is applied to MPN-COV [7] and iSQRT-COV [8] methods. Fig. 7 displays randomly selected backward-based visualizations on the three fine-grained benchmarks. For all the datasets, the small eigenvalues display much more similar input activations with those of all the eigenvalues. This implies that the classification decisions of the GCP networks are more dependent on the small eigenvalues and the associated eigenvectors. Fig. 8 shows the randomly selected perturbation-based visualization. For Birds datatset, the visualization of small eigenvalues are interpretable, where the head, beak, and feather of the bird species emerge repeatedly in the images. However, the images on Cars and Aircrafts are less explainable. It might be difficult to tell which visualization characterizes more class-relevant feature patterns. We believe this is largely because the subtle differences between cars or airplanes are not easily interpretable by human beings. For example, classes of Cars datasets are typically at the level of Make, Model, and Year. One can not really tell the differences between the classes 'Audi S4 Sedan 2012' and 'Audi S4 Sedan 2007'. Thus the visualized feature patterns can be vague. The fine-grained datasets of natural species, on the other hand, might generate more human-friendly feature patterns. Fig. 9 displays the perturbation-based visualizations of the ResNet-50 trained on Dogs [68] and Flowers [69] dataset. Similar with the images on Birds, the visualization is interpretable and the feature patterns of small eigenvalues are more class-relevant.

A.3 Random Perturbation-based Visualization
All in all, for the fine-grained datasets we have tested, the visualizations of the small eigenvalues generally have more semantically meaningful and more structured patterns. The large eigenvalues mainly generate not obviously classrelevant and hard-to-interpret features.

APPENDIX B EXPERIMENTS ON IMAGENET
Since some GCP methods (i.e., MPN-COV [7] and iSQRT-COV [8]) can work on the generic visual recognition task (e.g., ImageNet), we also apply our proposed SEB on this dataset to evaluate the performances.

B.1 Training Recipe for ImageNet
We take AlexNet and ResNet-50 as the model architectures and use the same experimental settings in our conference paper [12]. We set the batch size as 128 for AlexNet and 256 for ResNet. The AlexNet is trained for 30 epochs with an initial learning rate set as 10 −1.1 . The learning rate decays by 10 every 10 epochs. For training ResNet, we use the same learning rate to train for 60 epochs but decays by 10 at epoch 30 and epoch 45. We use SGD for optimization, with momentum of 0.9 and weight decay of 0.0001 for ResNet and 0.0005 for AlexNet. The network parameters are randomly initialized for both architectures. During training, the images are resized to 256×256 and then cropped to 224×224, with random horizontal flip augmentation. The inference is conducted on the 224×224 centered crop from the validation set. Table 9 shows the validation top-1 and top-5 accuracy on ImageNet. For AlexNet, our proposed SEB significantly improves the performances of MPN-COV [7] by about 5 %. However, it brings 1 % performance drop on ResNet-50 though the training accuracy still surpass others by a large margin (see Fig. 10). This phenomenon implies that the ResNet-50 equipped with our SEB has the risks of over-fitting  Zoom in for a better view. Fig. 9: Extra visualizations on Dogs and Flowers datasets. Obviously, small eigenvalues capture more semantically-meaningful and class-relevant feature patterns. and the poorer generalization performances. We observe that the eigenvalue distributions of the GCP methods on ImageNet are quite different from those on fine-grained benchmarks. For ImageNet, both the overall eigenvalue range and the eigenvalue differences are much smaller. Consider the large class complexity and the considerable amount of noise of ImageNet [70], [71], the small eigenvalues do not necessarily capture the class-specific features and the subtle classification clues. On the contrary, the small eigenvalues are more likely to encode only the data noise. Therefore, amplifying the insignificant eigenvalues might bring side effects on the generalization ability. To validate this guess, we also apply our proposed explainability methods on the ResNet-50 of MPN-COV [7] trained on ImageNet to evaluate the behavior of eigenvalues, which will be illustrated in the following sections. Despite the analyses from the perspective of visual explainability, we think this problem is worth further research in our future work.  Fig. 11 shows some examples of input responses to different eigenvalues. Unlike the case of fine-grained benchmarks, the large eigenvalues almost generate the identical input activations with all the eigenvalues, while the visualizations of small eigenvalues are less similar. Table 10 compares the quantitative evaluation results. For the large eigenvalues, the MAE is nearly negligible and the correlation coefficient achieves the maximum. This demonstrate that on ImageNet the decisions of the GCP networks are almost entirely dependent on the large eigenvalues and the associated eigenvectors. Fig. 12 shows several examples of perturbation-based visualizations on ImageNet. Different from the situation on finegrained benchmarks, the learnt feature patterns of the small eigenvalues are not obviously class-relevant. In contrast, the large eigenvalues show some signs of the class-specific feature patterns.  [42]. The evaluation is conducted on the subset of randomly selected 2, 000 images from the ImageNet validation set.  Both MPN-COV [7] and iSQRT-COV [8] first train ResNet-50 models with each GCP meta-layer on ImageNet from scratch and then use the same model to fine-tune on the fine-grained benchmarks. Although their reported results on fine-grained benchmarks are claimed to be based on the backbone ResNet-50, strictly speaking their backbone is actually ResNet-50 with the GCP meta-layer rather than the ordinary widely used ResNet-50. Table 11 shows their performances on these two backbones. As can be observed, only when the backbone equipped with the GCP meta-layer is trained on ImageNet from scratch, the GCP methods MPN-COV [7] and iSQRT-COV [8] can achieve comparable performances on fine-grained benchmarks. This may limit their practical usage as a dedicated backbone with the GCP layer is needed. Our proposed SEB is free of this constraint. It does not pose any backbone requirement and can achieve impressive performances on the commonly used backbones.