Identifying and Mitigating Flaws of Deep Perceptual Similarity Metrics

Measuring the similarity of images is a fundamental problem to computer vision for which no universal solution exists. While simple metrics such as the pixel-wise L2-norm have been shown to have significant flaws, they remain popular. One group of recent state-of-the-art metrics that mitigates some of those flaws are Deep Perceptual Similarity (DPS) metrics, where the similarity is evaluated as the distance in the deep features of neural networks. However, DPS metrics themselves have been less thoroughly examined for their benefits and, especially, their flaws. This work investigates the most common DPS metric, where deep features are compared by spatial position, along with metrics comparing the averaged and sorted deep features. The metrics are analyzed in-depth to understand the strengths and weaknesses of the metrics by using images designed specifically to challenge them. This work contributes with new insights into the flaws of DPS, and further suggests improvements to the metrics. An implementation of this work is available online: https://github.com/guspih/deep_perceptual_similarity_analysis/


I. INTRODUCTION
Similarity metrics are a fundamental part of many machine learning processes. Every time two or more objects are compared, a similarity metric is used. In computer vision, widely used metrics, such as the pixel-wise L2-norm, have been carefully studied and their benefits and flaws are wellknown which lets users make an informed decision when using them.
Many improvements to pixel-wise metrics have been, with a common goal being to mimic human perception with a so-called perceptual similarity. One popular perceptual similarity metric is the Structural Similarity Index Measure [1]. A more recent approach is to utilize deep features learned by machine learning models for measuring perceptual similarity. This practice, called Deep Perceptual Similarity (DPS) measures the similarity of two images by comparing their respective activations in the deep layers of Convolutional Neural Networks (CNNs), instead of using the pixel values directly.
DPS metrics have outperformed previous models on perceptual similarity [2]. Additionally, such metrics have been used as part of the loss function for training models, which have achieved impressive results on a host of tasks. These tasks include, image generation [3], style transfer and superresolution [4], object detection [5], and image segmentation [6].
While there are clear benefits of DPS, its flaws are not as well studied. While it has been shown that deep perceptual similarity 1 https://github.com/guspih/deep_perceptual_similarity_analysis/ is vulnerable to adversarial examples, this is expected from any method depending on deep networks and existing methods for protecting from adversarial attacks such as ensembles may be utilized [7]. Additionally, adversarial examples are quite complex compared to the known flaws of other metrics. For example, pixel-wise metrics would consider a black-and-white image to be as dissimilar as possible from its inverted version.
This work aims to analyze if and how DPS can successfully handle the flaws of the pixel-wise L2-norm, and investigate if there are any similar unexplored flaws of DPS and how those may be mitigated. Additionally, several different DPS metrics are analyzed for flaws and then evaluated on the BAPPS dataset [2], to check if those flaws translate into performance on an actual dataset.
The investigation of DPS is performed by creating image pairs that are similar to each other compared to some reference images and checking in which cases the DPS metrics succeed or fail in identifying the image pairs as more similar than the reference. The feature maps of the CNNs used for calculating similarity are analyzed to gain insight as to what underlies the successes and failures.

II. RELATED WORK
Commonly used image similarity metrics are pixel-wise metrics where each pixel of one image is compared directly against the corresponding pixel of the other. These metrics have long been known to be poor similarity metrics as they disregard high-level image structures [8,9,10]. Instead many different perceptual similarity metrics have been proposed including Dynamic Partial Function [11], the Structural Similarity Index Measure [1], and Structural Texture Similarity [12]. Despite known flaws and suitable alternatives, per-pixel metrics have consistently been used for image comparison within computer vision in general, and to calculate the loss for machine learning models specifically.
One powerful attribute of deep learning is that the deep features learned by the networks typically contain information useful for other tasks than the one the network was trained for. This attribute was used to great effect with the introduction of neural style transfer, where the content and style of images were compared using different sets of deep features within a neural network [13]. This practice of training models to minimize the difference between the activations of a deep network in order arXiv:2207.02512v1 [cs.CV] 6 Jul 2022 to get visually similar images is known as deep perceptual loss.
Deep perceptual loss has since its introduction been successfully applied to a large number of computer vision tasks such as improving the performance of variational autoencoders [14,15,16], Generative Adversarial Networks [3], Super-Resolution [17,18], and style transfer [4]. The method has been proven effective at the task of perceptual similarity where it significantly outperformed previous methods [2]. This method of calculating perceptual similarity using the deep features of neural networks is referred to as deep perceptual similarity (DPS).
One potential problem with DPS is that it relies on deep neural networks, which are known to be vulnerable to adversarial examples. Adversarial examples are almost imperceptible perturbations to images or other input data that induce significant changes or errors to the prediction model [19]. While no perfect protection from adversarial examples is currently known, there is a wide array of defenses that can be used, including using ensembles [7]. Additionally, outside of malicious attacks, this is rarely a problem.
Another paradigm for creating similarity metrics is to optimize a machine learning model for the task [20]. This has been applied to DPS with the LPIPS method, though it notably only performed marginally better than using methods that had only been pretrained [2]. Like with many other machine learning methods the results can be improved somewhat with the use of ensemble methods, though still comparable to pretrained models [7].
Where this work analyzes DPS through deep analysis of cases where it fails, another recent work investigates how different network architectures and pretraining procedures affect performance [21]. That work found, among other things, that better pretraining performance on ImageNet [22], does not necessarily lead to better perceptual similarity, It additionally showed that a good pretrained model can outperform models trained specifically for the similarity task.
As DPS metrics inherently rely on the deep activations of neural networks, most commonly CNNs, analyzing these activations is inherently interesting. Many methods for such analysis exist and one of the most common is to visualize the feature maps of the CNNs [23], which is utilized in this work.

III. DEEP PERCEPTUAL SIMILARITY
Most uses of deep perceptual similarity and deep perceptual loss have directly compared the corresponding activations of the two images. This method, referred to as spatial DPS, is formalized as the distance measure between x and x 0 in Eq. 1, where f is a norm such as L1 or L2 and p is a convolutional feature extractor with extraction layers l ∈ L each with C l channels with height H l , and width W l .
This work evaluates two additional methods of calculating deep perceptual similarity besides the spatial method. These two are the mean method tested in [21] and a sort method which is introduced in this work. The two methods are formalized in Eq. 2 Eq. 3 where x and x ↓ are the average and descending reordering of x respectively.
Both of these methods ignore the spatial positions of the features. The mean method compares the average of the features in each channel and the sort method pairs the features of each channel with one another in such a way as to minimize the norm. In the sort method the norm is minimized for any convex function f , compared to any other ordering of the features. This follows from In the case of infinitely large input images and translationinvariant CNNs the two presented methods are translationinvariant as no matter how much the image is translated the same features will appear. In the case with bounded images, as long as regions with the strongest feature activations aren't shifted off the image or too close to the boundaries, this should still likely result in a metric that is robust to translations. Even though many CNNs aren't strictly translation-invariant, in general translations have very little effect on the methods. The reasoning behind comparing average and sorted channels is that a strong activation in one channel often represent different concepts than a similar activation in another.
A problem with the mean and sort methods on their own is that humans would likely say that a lower translation is more similar to the original than a greater one. As such complete translation-invariance is not desirable. Thus, this work also investigates metrics that uses the sum of the spatial method with one of the two non-spatial methods.

A. Experimental Setup
DPS relies on neural networks which deep features contain useful information for image comparison. While networks can be trained specifically for the task, the most common use of DPS and deep perceptual loss is pretrained networks.
This work uses mostly the same feature extraction and comparison setup as [2]. The methods are analyzed and evaluated with the L2-norm as the comparison function (f ) using three models (p) pretrained on the ImageNet dataset [22]. The architectures for the three models are SqueezeNet [25], AlexNet [26,27], and VGG-16 [28]. The deep features are extracted from the same multiple layers for each network as in [2]. The features extracted in the original work were channelwise unit-normalized, and this work analyzes and evaluates both using and ignoring this practice. However, for brevity the analysis in Section IV concerns only the case without unit-normalization and the use of unit-normalization is later discussed in Section VII.

IV. QUALITATIVE ANALYSIS OF DPS ON DISTORTIONS
This work carries out a qualitative analysis of deep perceptual similarity metrics over images specifically designed to test for its strengths and potential flaws. The analysis is carried out by distorting images in ways for which DPS is previously known to work well or speculated to perform poorly. The similarity of the distorted image with the original is then compared to the similarities of a set of reference images and the original, where the reference images are intended to be notably less similar than the distortion. The feature maps at various layers of the DPS networks are then analyzed for each case to gain a deeper understanding of why the metric performed the way it did in this case. Such insight was then used to create further image pairs to test against. Finally, for one category of images, specific reference images were created for each image pair. These reference images, like the others, were created to be perceived by humans as less similar than the distorted versions but intended to fool some DPS metrics.
The images used in the tests are 96 × 96 pixels and have been designed and distorted by hand. The distortion tested are divided into four categories; color inversion, translation, rotation, and color stain. Seven reference images were created; mono-colored images of black, white, gray, red, green, and blue, as well as one with randomly colored pixels. One image pair each from the inversion, translation, and rotation categories is shown in Fig. 1

A. Black-and-White Color Inversion
Color inversion of black-and-white images is typically used as an example of when pixel-wise metrics break down. This is because each pixel in the inverted image is as different from each other as they could be, which means every other possible image would be regarded as more similar, which is obviously not the case for perceptual similarity. Despite this being used as an example of why to abandon pixel-wise metrics in favor of DPS metrics, there has been little investigation of how well DPS performs in these scenarios. For these reasons the first set of images created for analyzing DPS were simple black-and-white patterns that were distorted by inverting the colors.
While pixel-wise metrics fail by definition on this category of images, all tested DPS metrics correctly identify each image pair as more similar than any of the reference images. Analysis of the feature maps reveals that many channels are activated by contrasts or higher-level structures like lines or shapes. These activations are often completely agnostic to inversion and identify the structures regardless of color. This makes the black-and-white inversion pairs almost exactly the same for many channels in the feature space, which leads to the good performance of DPS on color inversion.

B. Translation and Rotation
It is also clear from the feature maps that all activations are strongly spatially correlated to where those features appear in the input image, which can be seen in Fig. 3. This is obvious as CNN architectures in general are built around each activation depending only on a small region of the input or previous layer. While, in theory, activations in the later layers depend on information aggregated from a large swath of the image, in practice, strong activations in the feature maps at any layer are correlated with features in the spatially corresponding region of the input image. This has been previously suggested as a potential flaw of spatial DPS [21].
To investigate whether this would have a significant impact on spatial DPS and whether other DPS metrics could handle these cases, the categories of translation and rotation have been tested. The translation images have a region containing much structure in otherwise plain images which have been distorted by translating that region. The rotation images are simply images that have been distorted by rotation in steps of 22.5 up to 90 degrees, as well as one rotated 180 degrees. The purpose of the incremental steps was to see if and further how sensitive DPS is to rotation.
Both the pixel-wise metric and spatial DPS fail to identify any translated image as more similar than the reference images, while the other DPS metrics succeed in each case. For rotation, both pixel-wise and spatial DPS metrics fail on about the same amount of cases, slightly less than half, while the other DPS metrics almost succeed on each image pair.
This clearly shows that the spatial DPS metric on its own is not suitable for these types of scenarios, while translationinvariant DPS metrics can handle them very well. It is also interesting that the translation-invariant DPS metrics handle rotation so well since it is well known that early channels in CNNs often learn to identify specific orientations in lines and other structures. Likely, the later layers of CNNs combine orientation-specific features into higher-level orientationindependent ones.

C. Color Stain
Another revelation from feature map analysis is that many channels tend to activate strongly from specific colors, textures, or random noisy structures. This might be challenging for nonspatial methods as ignoring the spatial position of activations might lead to confusing noise for interesting structures. For example, in the case of mean DPS, an image with a small interesting region might be seen as dissimilar from the same with added stains since the average activation in the noisy one will be larger.
To test for this the color stain category is used. The image pair for the color stain category consists of a plain image with a structurally interesting region, and a distorted version with a similar or same interesting region but the plain color is changed, and added noisy features for some images. The color stain category does not use the same reference images as the other categories, and instead, each image pair has a specific reference image designed to be less perceptually similar. These reference images have the same plain color without stains as the non-distorted image, but their interesting region is significantly different compared to the distorted version. Examples of image pairs and their specific reference image are shown in Fig. 2.
For the color stain category, the pixel-wise metric again fails for each image pair. Notably, both the mean and sum of spatial and mean DPS fails almost all image pairs. The remaining DPS metrics tested perform well, with spatial DPS being the best.
One specific image in this category was a white image with a red, green, and blue irregular circle in one corner. The distorted image retained the circles but the plain white background was colored a darker shade of green with random yellow stains. By observing the feature maps of these images it is clear that the color change and stains add significant activations to the otherwise sparse feature maps, especially in later layers. This is shown in Fig. 3, where the image and its distortion are displayed together with feature maps from the second and fourth SqueezeNet ReLU layer.

V. EVALUATION
In order to investigate how the insights gained through the qualitative analysis translate to performance on a perceptual similarity dataset, the DPS metrics are evaluated on the BAPPS dataset, using the same procedure as in the original work [2]. However, the evaluation is only carried out for the pretrained networks SqueezeNet, AlexNet, and VGG-16 without additional fitting to the training data. This is equivalent to what the original work refers to as "Net (Supervised)", with the addition of testing non-spatial DPS metrics. This evaluation follows the original work which means f is the L2 norm and the features extracted from p have been channel-wise unit-normalized.
2AFC consists of image patches and two distorted versions of each patch, as well as human annotations as to which distorted patch is most similar to the original. The aim of 2AFC is to train and evaluate models for perceptual similarity judgment by evaluating if those models give higher similarity to the distortion that most human annotators agreed was more similar. In addition to evaluating on the complete 2AFC part, results are also given for a number of subdivisions defined by the type of distortions that are applied: (1) Traditional augmentation methods, outputs from (2) CNN-based autoencoders, (3) superresolution, (4) frame interpolation, (5) video deblurring, and (6) colorization.
JND consists of an image patch as well as a barely distorted version along with human annotations of whether the two patches are the same. The human annotators were shown the two images only briefly and were also shown pairs of the same and very different images. The aim of JND is to test models for perceptual similarity by evaluating if those models give a higher similarity to those samples that human annotators had difficulty telling apart.

VI. RESULTS
An aggregation of the outcome of the tests described in Section IV is shown in Table I. The performance is presented as the number of images, where the metric did not find any of the reference images to be more similar than the distorted version.
The results of the evaluation on the BAPPS dataset are shown in Table II for each subdivision of the 2AFC part, for the entire 2AFC part given by the average over the subdivisions, and for the JND set. The results for the evaluated metrics are presented along with the LPIPS metrics from [2] and human performance for reference.

VII. DISCUSSION
The purpose of this work has been to evaluate if and how DPS metrics can handle the typical cases where pixel-wise metrics fail, and to investigate whether similar flaws exist in current DPS implementations. All tested DPS metrics handle the color inversion tests for which pixel-wise metrics break down, and additionally the clear preference for contrasts and structures in feature maps indicates that DPS metrics are well-suited to handle other similar color-changing operations. By far most common form of DPS metrics used is spatial DPS, which seems to perform as poorly as the pixel-wise metric on the rotation and translation test cases. While the non-spatial DPS metrics perform well on these weaknesses, they do not perform as well as spatial metrics on the color stain category of tests. This is especially true for mean DPS which failed most of the color stain tests. The spatial and non-spatial combined metrics perform similar to spatial DPS, indicating that perhaps combining metrics using unweighted summation gives a preference for spatial DPS. Though the results of the combined metrics improved somewhat for rotation, indicating that there are at least some benefits to this strategy.
Analyzing the BAPPS scores for the different DPS metrics shows that flaws of spatial DPS also affect performance on a perceptual similarity dataset. While spatial DPS, in general, performs worse than the other DPS metrics, notably, this is especially true for the traditional augmentations subdivision. Traditional augmentations include operations such as rotation, translation, and skewing which indicates that the weaker performance of spatial DPS is due to the flaws identified in this work.
Another notable result is that mean DPS, on average, performs best on BAPPS, even though it was vulnerable to the color stain category of distortions to a much larger degree than sort DPS. However, both mean and sort DPS metrics perform similarly and are both better choices than spatial DPS. It is possible that color stain and related distortions are not so common to be a problem in a real-world scenario, or that the BAPPS dataset does not include such cases. Additionally, the image pairs used for analysis in this work have often been very simple and plain compared to the images typically included in datasets.

A. The effects of unit-normalization
As mentioned in Subsection III-A, the qualitative analysis described in Section IV was also performed with channel-wise unit-normalization of the extracted features. This had three notable effects. First, the success rate in the rotate category rose for all DPS metrics, especially for the metrics that use spatial DPS. In fact, those metrics became almost competitive with mean and sort DPS. Second, while spatial still fails on each image pair in the translate category, the combined metrics are somewhat improved. Likely due to normalizing making the spatial distances lower which gives more weight to the non-spatial distances. Finally, using normalization made each DPS metric perform poorly in the color stain category.
When evaluating with the BAPPS dataset unit-normalization has only a small positive effect on performance. Likely translation and rotation and similar augmentation are more common in that dataset than augmentations similar to the color stain procedures.

VIII. FUTURE WORK
From the results and analysis presented in this work there are some notable directions of research to explore.
Both this and a prior work [21] has shown that spatial DPS does not perform as well as on perceptual similarity tasks as other implementations of DPS. One future possibility is to investigate if this translates to related field such as deep perceptual loss and content-based image retrieval. If it does, simply changing the way perceptual loss is calculated could improve the results on many different tasks.
While most DPS metrics outperform previous perceptual similarity metrics, the discrepancy in performance of DPS metrics indicates that exploring how to calculate DPS metrics is an open problem. For example, a DPS metric that make use of both spatial and non-spatial comparisons could perhaps gain the benefit of both. Additionally, the upsides and downsides of unit-normalization remain inconclusive.