Distributed Learning and Inference with Compressed Images

Modern computer vision requires processing large amounts of data, both while training the model and/or during inference, once the model is deployed. Scenarios where images are captured and processed in physically separated locations are increasingly common (e.g. autonomous vehicles, cloud computing). In addition, many devices suffer from limited resources to store or transmit data (e.g. storage space, channel capacity). In these scenarios, lossy image compression plays a crucial role to effectively increase the number of images collected under such constraints. However, lossy compression entails some undesired degradation of the data that may harm the performance of the downstream analysis task at hand, since important semantic information may be lost in the process. Moreover, we may only have compressed images at training time but are able to use original images at inference time, or vice versa, and in such a case, the downstream model suffers from covariate shift. In this paper, we analyze this phenomenon, with a special focus on vision-based perception for autonomous driving as a paradigmatic scenario. We see that loss of semantic information and covariate shift do indeed exist, resulting in a drop in performance that depends on the compression rate. In order to address the problem, we propose dataset restoration, based on image restoration with generative adversarial networks (GANs). Our method is agnostic to both the particular image compression method and the downstream task; and has the advantage of not adding additional cost to the deployed models, which is particularly important in resource-limited devices. The presented experiments focus on semantic segmentation as a challenging use case, cover a broad range of compression rates and diverse datasets, and show how our method is able to significantly alleviate the negative effects of compression on the downstream visual task.


I. INTRODUCTION
M ODERN intelligent devices such as smartphones, autonomous vehicles and robots are equipped with highquality cameras and powerful deep neural networks that enable advanced on-board visual analysis and understanding. These large models are trained with a large amount of data and require powerful hardware resources (e.g. GPUs). These models also require days or even weeks to train, which is not possible in resource-limited devices. Thus, training is often performed in a centralized server, which also allows using data captured by multiple devices to train better models (e.g. a fleet of autonomous cars). In this case, training and test take place in two physically separated locations, i.e. server and device, respectively. In other cases, such as in mobile cloud computing, the data is captured by the device, while the inference takes place in a server.
One important requirement in these scenarios is that, at some point, the visual data needs to be transmitted from the device to the server. Fig. 1a shows an archetypal scenario of autonomous driving, where each vehicle of the fleet captures and encodes data and transmits it to the server. The server decodes the data and uses it for training the analysis models. The trained models are then deployed to autonomous vehicles, where they perform inference. Captured data often requires to be annotated by humans in order to train supervised models, which adds to the reasons to process the data in a server.
The captured data can be stored on-board in a storage device and physically delivered to the server, or directly transmitted through a communication channel. In either case, storage space or channel capacity are constraints that condition the amount of collected samples in practice, and effective collection requires data compression to exploit the limited storage and communication resources efficiently.
The amount of data captured (possibly from multiple cameras) can be enormous, requiring high compression rates with lossy compression. However, this entails a certain degradation in the images, which depends on the bitstream rate (lower the rate, higher the degradation). In this paper, we study the impact of such degradation on the performance of the downstream analysis task. At times, the degradation affects only one of the training and test data. For instance, in Fig 1a, training data is degraded while test data (on-board) can be accessed without degradation.
When training data is compressed and test is not (or vice versa), a first effect we observe is covariate shift (i.e. train and test data are sampled from different distributions). For instance, the first column of Fig. 1b represents the original captured images, while the second represents the compressed images (i.e. reconstructed 1 ). A clear difference in terms of lack of details and blurred textures is observed, which causes covariate shift (e.g. original for test, compressed for training in the example of Fig 1a). A possible solution to this problem is compressing both training and test data at the same rate. For the autonomous driving scenario of Fig. 1a, this would mean deploying an image compressor in the car (including encoder and decoder) and performing inference on the reconstructed images. While this approach alleviates the covariate shift, it is not always effective and also increases the computational cost in the on-board system. The degradation caused by lossy compression not only induces covariate shift, but can also harm the performance of the downstream task through the means of semantic information loss. Here semantic information refers to the information that is relevant to solve a particular downstream task and it can be lost during the process of compression. Semantic information is task-dependent and its loss is typically irreversible. For example, the actual plate number WAF BA 747 in the second column of Fig. 1b is lost in the process of compression, and cannot be recovered. However, if the task is car detection, the actual plate number is not necessarily relevant semantic information.
In this paper, we study the effect of compression on downstream analysis tasks (focusing on semantic segmentation) under different configurations, which in turn can be related to real scenarios. We observe that both covariate shift and semantic information loss indeed result in a performance drop (see Fig. 1c) compared to training and test with original images (configuration OO). The performance depends on the compression rate and the particular training/test configuration. For instance, in the configuration of the autonomous driving scenario of Fig. 1a compressing the test data prior to inference (we refer to this approach as compression before inference, and corresponds to the training/test configuration compressedcompressed, or CC for short) degrades the performance more than not using the original data (configuration CO), showing that it is preferable to keep the test data more semantically informative than correcting the covariate shift.
The previous result also motivates us to explore whether there exists a solution that improves over the baseline CO configuration. As a result, we propose dataset restoration, an effective approach based on image restoration using generative adversarial networks (GANs) [1]. Dataset restoration is applied to the images in the training set without modifying the test images, effectively alleviating the covariate shift while keeping the test data semantically informative. In this case, we show that the configuration restored-original (RO) does improve performance over the baseline (see Fig. 1c). An additional advantage is that there is no computational cost penalty nor additional hardware or software requirements in the deployed on-board system (in contrast to compressing the test data). Note also that our approach is generic and independent of the particular compression (deep or conventional) used to compress the images.
Adversarial restoration decreases the covariate shift by hallucinating texture patterns that resemble those lost during compression while removing compression artifacts, both of which contribute to the covariate shift. The distribution of restored images is closer to the distribution of original images and thus the covariate shift is lower. Fig. 1b shows an example where the trees have lost their texture and appear essentially as blurred green areas. A segmentator trained with these images will expect trees to have this appearance, but during test they appear with the original texture and details of leaves and branches, which leads to poor performance. The restored image has textures that resemble real trees and contains less compression artifacts, which makes its distribution closer to that of the actual test images, contributing to a significant improvement in downstream performance (see Fig 1c). Note that adversarial restoration cannot recover certain semantic information. This example also illustrates the effect on semantic information. The license plate appears completely blurred due to compression. Note that adversarial restoration can recover the texture of digits (or even hallucinate random digits), which can be useful to improve car segmentation, but the original plate number is lost (i.e. semantic information), which makes it impossible to perform license plate recognition at that compression rate.
In summary, our contributions are as follows: • Systematic analysis of training/test configurations with compression and relation of downstream performance with rate, semantic information loss and covariate shift.
• Dataset restoration, a principled method based on our theoretical analysis, to improve downstream performance in on-board analysis scenarios. This method is task-agnostic and can be used alongside multiple image compression methods. It also does not increase the inference time and memory requirements of the downstream model.

A. Lossy compression
A fundamental problem in digital communication is the transmission of data as binary streams (bitstreams) under limited capacity channels [2], [3], a problem addressed by data compression. Often, practical compression ratios are achievable only with lossy compression, i.e. a certain loss with respect to the original data is tolerated. Traditional lossy compression algorithms for images typically use a DCT or a wavelet transform to transform the image into a compact representation, which is simplified further to achieve the desired bitrate. Examples of lossy image compression algorithms are JPEG [4], JPEG 2000 [5], [6], and BPG [7]. BPG is the current state-of-the-art and is based on tools from the HEVC video coding standard [8].
Recently, deep image compression [9]- [13] has emerged as a powerful alternative to the traditional algorithms. These methods also use a transformation based approach like the traditional methods, but use deep neural networks to parameterize the transformation [9]. The parameters of the networks are learned by optimizing for a particular rate-distortion tradeoff on a chosen dataset. Mean Scale Hyperprior (MSH) [13], a deep image compression method based on variational autoencoders and BPG are used as representative methods of deep learning based and traditional image compression respectively.

B. Visual degradation and deep learning
A loss in the quality of images can occur through many factors including blur, noise, downsampling and compression. Researchers have reported a drop in task performance of convolutional neural networks (CNN) models when such degradations are present in the test images [14]- [16]. Further, numerous methods have been proposed to make these CNN models robust to degradations [15], [17], [18]. These approaches include forcing adversarial robustness during training [15], modifying and retraining the network [17], and using an additional network altogether [18].
While the aforementioned works target robustness across degradations, there have been studies focusing exclusively on compression as well. These include [19] (on the deep compression method [12]), [20] (JPEG) and [21] (both deep [22] and JPEG). Unlike the previous methods, these works use the compressed images (in some form) for training the deep models and thus obtain a better performance on compressed images. Moreover, [19] and [20] encode the images using the compressors and the deep networks are trained to predict the task output using the encoded representation directly, resulting in faster inference.
A compressed image can be processed using a restoration method before using it for inference to improve its performance; although our analysis reveals that this is a sub-optimal approach. Galteri et al. [39] propose a GAN-based restoration network to correct JPEG compression artifacts. They also evaluate different restoration algorithms on the basis of the performance of restored images on a trained object detection network. They show that their GAN-based algorithm performs better than other methods compared in the paper. Our analysis provides an explanation for this observation.

D. Domain adaptation.
Domain adaptation [42] is a problem motivated by the lack of sufficient annotations. Typically, domain adaptation methods leverage the abundant annotated data available from a different yet related domain (called as the source domain) to improve performance on the domain of interest (target domain), where there is a lack of annotated data. Examples of source and target domains include synthetic images vs real images, images in the wild vs images on a webpage, etc. Domain adaptation methods can be divided into unsupervised [43], [44], semi-supervised [45] and supervised [46]- [50] depending on the quantity of available data (and annotations) in the target domain. Approaches for domain adaptation can be categorized into latent feature alignment using autoencoders [46], [47], adversarial latent feature alignment [48]- [51] and pixel-level adversarial alignment [44], [50].
The scenario when only compressed images are available at training time, with original images available at test time is related to domain adaptation. Dataset restoration, our proposed method for this scenario, corrects the covariate shift and domain adaptation algorithms account for domain shift in some form. Probably, the closest domain adaptation method to dataset restoration is a unsupervised method that addresses alignment only at pixel-level [44]. However, an important distinction is that domain adaptation tackles problem arising due to lack of annotations for the images in the target domain, while for us the concern lies in the non-availability of images themselves. Thus, we study the effectiveness of dataset restoration, using a varying number of original images for training the restoration model.

A. Problem definition
We are concerned with downstream understanding tasks where we want to infer from an input image x ∼ p X (x) the corresponding semantic information y ∼ p Y |X (y). In the rest of the paper, we will assume that y is a semantic segmentation map, but our approach can be applied to other semantic inference tasks, such as image classification or object detection. The objective is to find a parametric mapping φ : x → y by supervised learning from a training dataset , where each image x (i) has a corresponding ground truth annotation y (i) . The mapping is typically implemented as a deep neural network. The performance of the resulting model is evaluated on a test set X ts = x . Under conventional machine learning assumptions, X tr ∼ p X (x) and X ts ∼ p X (x), i.e. both training and test sets are sampled from the same underlying distribution p X .
In our setting, we consider that X tr and/or X ts undergo a certain degradation ψ : x →x. In our case, the degradation is related with the lossy compression process necessary to transmit the image to the remote location where the actual training or inference takes place; and so we havex is the image decoder 2 and z is the compressed bitstream. The resultx is the reconstructed image, which follows a new distribution pX of degraded images, i.e.x ∼ pX (x). Note that parallels can be drawn from the arguments in this section for other image degradations such as blur, downsampling, noise, color and illumination changes, etc.
Lossy compression is characterized by the distortion D (x,x) of the reconstructed image and the rate R (z) of the compressed bitstream. The encoder and decoder are designed to operate around a particular rate-distortion (R-D) tradeoff λ, either by expert crafting in conventional image compression, or by directly optimizing parameters of a deep neural network.

B. Covariate shift
The covariate shift problem precisely occurs when the underlying distributions of training and test data differ, i.e. X tr ∼ p Xtr and X ts ∼ p Xts with p Xtr = p Xts . This leads to sub-optimal performance because the model is evaluated on a data distribution different from the one it was optimized for. While covariate shift is often found in machine learning (e.g. training with synthetic data and evaluating on real), in our case this problem is a consequence of lossy compression and it increases severely as the rate decreases. The drop in performance is related to the degree of covariate shift, which could be seen as the divergence between distributions d (p Xtr , p Xts ). In the conventional machine learning setting without compression there is no covariate shift, since X tr ∼ p X and X ts ∼ p X , nor when both training and test set are compressed with the same method and at the same rate, since X tr ∼ pX and X ts ∼ pX . 2 We only consider lossy compression, since in lossless compressionx = x.
However, covariate shift exists in the other two configurations, namely CO and OC (see Table I).
The degradation due to lossy compression can be observed clearly in Fig. 1b, when comparing the original captured image and the image after compression. This also gives an idea of the difference between the original domain and the domain induced by compression. It has images with lesser details which also suffer from blurring and coding artifacts. More examples are shown in Fig. 2 for the two compression methods (MSH and BPG), with the images compressed at a similar rate. It can be seen that degradations are consistent yet with some differences (e.g. blocky artifacts for BPG, more blurred in MSH).

C. Semantic information loss
Covariate shift explains how compression impacts the downstream task when data at training and test time are compressed unequally. Another factor that impacts task performance arises from compression resulting in semantic information loss. By the semantic information present in an image we refer only to what is relevant to the downstream task. Thus, by definition, semantic information loss is task dependent. Continuing with our example from the introduction (Fig. 1b), the letters in the license plate of the car plays little to no role in establishing the presence of a car in the image. Thus, the exact letters are not relevant semantic information for the task of car detection. However, if the task is license plate recognition, the letters are an integral part of semantic information. Compression causes semantic information loss as it makes the compressed image devoid of some semantic attributes present in the original image. The loss of letters on the plate in the compressed image ( Fig. 1b) is evidence of semantic information loss (when the task is license plate detection).
Further evidence of semantic information loss can be found in Fig. 2, since the degradation often removes details and textures, blends small objects together via blur and lack of contrast, and introduces confusing artifacts, preventing us from recognizing small objects at all (e.g. individual pedestrians), and making larger objects more difficult to recognize due to the loss of discriminative details and textures (e.g. tree leaves). Only in retrospective, after observing the original undistorted crop, we can infer the small objects in the distorted image. Similarly, a semantic segmentation model will struggle to recognize them, or directly fail when the semantic information has disappeared completely (e.g. license plate number).
Let Y be a random variable that represents the semantic information in the original image, X. For instance, if the task is semantic segmentation, Y would take values from the set of semantic maps of images. Mathematically, we formulate semantic information loss, S, in the compressed images, X, using mutual information, I, as follows: S Y (X,X) = I(X, Y ) − I(X, Y ). Predictably, S Y (X,X) is non-negative, sinceX is produced from X via the map ψ and thus, we have I(X, Y ) ≥ I(X, Y ) as a consequence of the data processing inequality.   Table I). A configuration is defined by the pair (X tr , X ts ), with X i ∼ p X represented as O and X i ∼ pX represented as C. Thus, the conventional machine learning setting (i.e. without compression) corresponds to OO, and the configuration of Fig. 1a is CO, since X tr ∼ p X and X ts ∼ pX . The former does not suffer from semantic information loss nor covariate shift, while the latter does suffer from both. The CO configuration can also be generalized to other scenarios involving on-board analysis 3 where data capture and inference takes place in the device and the training in a server (e.g. autonomous cars, unmanned aerial vehicles and other robotic devices).
The configuration OC involves training performed in the server with the original images, while the device is resourcelimited (e.g. a smartphone) but can send the image to the server, and the result of the analysis (e.g. predicted class, bounding box, segmentation map) is used in the server side or sent back to the device. In this case, the test images are compressed, which implies semantic information loss, and also covariate shift. Fig. 3 illustrates the paradigmatic scenario 3 Often called on-board perception, but we prefer on-board analysis to avoid confusion later. of (mobile) cloud computing [52]- [54]. Another example of OC configuration is distributed automotive perception [21], where the sensor module compresses the captured image and transmits it through the automotive bus system to the perception module where the downstream tasks are performed.
Similarly, the configuration CC appears in the previous scenario when training images are also compressed, and at the same rate as test images. In this case, both training and test images suffer from semantic information loss, but there is no covariate shift since both are sampled from the same pX .
Compression before training. We can remove the covariate shift from a configuration OC by transforming it into CC. This can be achieved by compressing the training data at the same rate and we refer to this adaptation approach as compression before training (see Fig. 4a). However, through this process, semantic information loss is additionally introduced to the training set. Generally, the presence or absence of semantic information loss in the test images is a major factor, while it is not always the case with the training images. While some class information may be lost in a particular training image, its presence in other training images can compensate for it. As such we expect the model with configuration CC to outperform configuration OC.
Compression before inference. Similarly, we can also transform a configuration CO into CC by compressing the test images. We refer to this process as compression before inference (see Fig. 4b). While this process allows us to correct the covariate shift due to compression, it also introduces semantic information loss at test time. The introduction of semantic information loss in the test is critical and can cause the performance of configuration CC to be even worse than the configuration CO at times (as shown in Fig. 1c). Moreover, compression before inference requires installing a full compression encoder and decoder module on-board prior to the downstream task, resulting in a significant computational penalty in the deployed system.

A. Proposed approach
Motivated by the two limitations mentioned above, we propose dataset restoration as an alternative approach that alleviates covariate shift without inducing semantic information loss in the test data (in contrast to compression before inference). The key idea is to adapt the training dataset using adversarial image restoration, and use the adapted dataset as actual training data for the downstream task (see Fig. 4c). In this way, the on-board analysis module can exploit all the information available in the captured image. Another important advantage is that adaptation takes place only in the server, and the resulting model can be readily and seamlessly deployed in the car with the same hardware, therefore without requiring to install any additional hardware nor increasing the inference cost.
We now recall that a great deal of degradation is related to the loss of texture in the decoded image and the appearance of compression artifacts (these two factors are clearly apparent in Figs. 1b and 2). Our goal was to find an appropriate image restoration technique that could learn from a given set of examples and provide us a way to remove the artifacts and recover texture in the images.
Our restoration module is based on adversarial image restoration, where a generative adversarial network (GAN) [1] conditioned on the degraded image is employed to improve the image quality. A GAN is based on two networks competing in an adversarial setting. The generator takes the input image and outputs the restored image. The discriminator observes real and restored images and it is optimized to classify between real and restored images. The generator, in contrast is optimized to fool the discriminator, and indirectly improves the quality of the restored images. Through the process, the generator learns to remove compression artifacts and replace unrealistic textures by realistic ones that could be used by the discriminator to identify the restored images. The architecture of GAN is based on the one proposed in [55] (for image-toimage translation [56]), which has a generator and multiple discriminators (see Appendix B for details).
During the process of dataset restoration, we use our trained generator to restore individually every image in the training dataset for the downstream task. Examples of some of the restored images can be found in Figs. 1b and 2. While not being able to restore lost semantic information (e.g. the same individual pedestrians), the restored images look sharper, with less artifacts and blurred regions are enhanced with hallucinated textures that resemble the real images. As such, the shift with respect to the distribution of original images, on which the trained model will be evaluated, is reduced. Table I includes two new configurations OR and RO, where R refers to restored images.

B. Adversarial restoration, covariate shift and perceptual index
There is an interesting relation between the perceptual index described in [57] and the covariate shift, which explains why adversarial image restoration is the appropriate approach (compared to non-adversarial). In the CO configuration, the training images X tr are compressed and therefore follow pX , while the test images X ts follow p X . Thus, the covariate shift is equal to d(p X , pX ), where d denotes a probabilistic divergence, like the Kullback-Leibler divergence or the Jenson-Shannon divergence [3]. On the other hand, the perceptual index in [57] is defined as the divergence between real images p X and compressed (and decoded) images pX (note that there is not training/test distinction). Therefore, in the case of CO configuration, the covariate shift corresponds to the perceptual index of the test set (in both cases, the lower the better). An important conclusion from [57] is that perception and distortion are at odds with each other, and that there exists a limit beyond which perception and distortion cannot be reduced simultaneously (see Fig. 5). Thus, the effect of dataset restoration (i.e. moving from CO to RO) is to lower the covariate shift (and perceptual index) at the expense of increasing distortion, provided we are close to the perceptiondistortion limit.
We are ultimately interested in the implications on the performance of the downstream task, semantic segmentation in particular. Our analysis in the previous section reveals that the task performance is greatly dependent on covariate Attainable at r 1 and r 2 Fig. 5. Illustration of the perception-distortion tradeoff [57], [58]. Restoration process shifts the compressed images from a point of low distortion and low perceptual quality to a point of higher distortion but higher perceptual quality in the perception-distortion plane.
shift and semantic information loss. Reducing the perception index is therefore more crucial than training with images of low distortion, as a lower perception index corresponds to a lower covariate shift. Hence, we use adversarial restoration and decrease the perceptual index at the cost of increased distortion. Are all image restoration approaches helpful? We argue that only adversarial image restoration are suitable, since they explicitly minimize the perceptual index through the discriminator and consequently the covariate shift with respect to the captured images. In contrast, non-adversarial image restoration methods do not necessarily reduce the perceptual index. Typically, these methods try to further decrease the distortion and this can be counter-productive as perception and distortion are at odds near the limit.

A. Experimental settings
Datasets. We evaluate our methods on three datasets: Cityscapes [59] is a popular dataset in autonomous driving, and contains 5000 street images (2975/500/1525 for training/validation/test sets) of which training and validation have pixel-level segmentation maps annotated with 19 different concepts, including objects and "stuff". We use the annotated sets to train (training set) and evaluate semantic segmentation (validation set). It also contains another 20000 images with coarse annotation. We ignore these annotations and use a subset of 2000 images to train the deep image compression model (i.e. MSH [10]) and the image restoration methods.
INRIA Aerial Images Dataset [60] contains aerial images of diverse urban settlements with segmentation maps with two classes (building and background). The dataset consists of aerial images from 10 cities with 36 images per city. Annotations are provided for 5 of these cities and the segmentation models were trained on 4 cities and evaluated on 1 from these. The images from the other 5 cities were used for compression and restoration. Semantic Drone Dataset [61] contains 400 high resolution images captured with an autonomous drone at an altitude of 5 to 30 meters above ground, and their corresponding annotated segmentation maps (20 classes). The 400 publicly released images were resized from a resolution of 6000x4000 to 3000x2000. The segmentation models were trained on 265 images and evaluated on 70 images while the remaining 65 images were used for the compression and restoration models. Each image was further split into 12 patches each with dimension of a 1200x800. All metrics are calculated on these patched images.
Compression methods. We use two state-of-the-art image compression methods. The Better Portable Graphics (BPG) format [7] is based on a subset of the video compression standard HEVC/H.265 [8] and is the state-of-the-art in non-deep image compression. The Mean Scale Hyperprior (MSH) [10], [13] is a state-of-the-art deep image compression method, based on an autoencoder whose parameters are learned to jointly minimize rate and distortion at a particular tradeoff λ, i.e. min R + λD. MSH models were pretrained for 600k iterations on the CLIC Professional Dataset 4 with MSE loss. Appendix A contains details of the model architecture.
Segmentation. For the downstream task we use the state-ofthe-art semantic segmentation method DeepLabv3+ [62]. The model is trained using the same procedure mentioned in the paper. We use an output stride of 16 and perform single scale evaluation.
Metrics. The quality of the inferred semantic segmentation map is evaluated using the mean intersection over union (mIoU, the higher the better). For image compression we measure rate in bits per pixel (bpp, the lower the better) and the distortion in PSNR (in dB, the higher the better).

B. Cityscapes
Rate-distortion curves. We first characterize the ratedistortion performance and the tradeoff for the two compression methods in our experiments, and the impact of the proposed restoration approach on them (see Fig. 6). The curves sweep the whole range, from low to high quality images. As expected, the distortion decreases (PSNR increases) with rate. We observe that MSH performs significantly better than BPG on Cityscapes, i.e. it produces images with lower average distortion at similar rate. Interestingly, once the images are restored, images compressed with MSH have marginally higher distortion than those compressed with BPG. Segmentation performance. We evaluate the segmentation performance under six different configurations (i.e. OO, CO, RO, CC, OC, OR). The results are shown in Fig. 7.
For the Cityscapes dataset, we observe that the model with configuration CO outperforms the model with configuration CC which shows that correcting covariate shift by compression before inference can potentially result in lowering the performance. Table II shows the performance per class. We see that the mIoU of classes representing small objects 5 is significantly lower in the configuration CC when compared to CO. Since smaller objects are relatively easier to lose by compression, the observation confirms that the introduction of semantic information loss in the test set from the process of compression before inference is responsible for the decrease in performance.
The proposed dataset restoration approach is able to improve 1.4-4.8% on the configuration CO, and we achieve close to optimal performance (77% mIoU) requiring only 0.13 bits per pixel. Thus, lossy compression can result in huge storage savings during data collection when compared to lossless image compression. For the same budget required to collect the 2975 training images with MSH at 0.13 bpp, using lossless compression (PNG in our experiments, resulting in 9 bpp) we would have collected only 42 images, which is clearly insufficient to train the segmentation network. Furthermore, for the same performance, dataset restoration effectively reduces the required budget (for example, configuration CO achieves 75.64% with 0.128 bpp, while RO approximately requires around 0.07 bpp). Amount of samples for training the restoration network. In order to learn the restoration network it is necessary to collect high quality images (uncompressed or compressed losslessly, ideally). Thus, the amount of images to train the restoration network is also an important factor. Fig. 8 shows the result for configuration CO (no restoration) and RO with a restoration network train with different amounts of training samples. It suggests that restoring images compressed with MSH can be learned with fewer images (or at least the performance saturates) than in the case of BPG.
Adversarial vs non-adversarial restoration. In order to show the connection between perceptual index and adversarial image restoration, we compute the perceptual index of the original, compressed and restored images using the blind quality assessment method HOSA [63] (as described in [57]). We use Residual Dense Networks [29] as a representative method of non-adversarial image restoration. RDN is trained  While the previous experiment shows that adversarial restoration does improve the perceptual quality of images, we are ultimately interested in the segmentation task. We use the model trained with the original images to extract features from a shallow layer. Then we extract features of original images, compressed images and restored images. Since we only use one model, all features are aligned in the channel dimension. We want to measure the level of alignment in activation, so we plot the average activation (over the validation set of Cityscapes) value for each channel of original, compressed or restored versus the original one (see Fig. 10). Obviously, the points corresponding to original images lie on the identity line, while many channels from compressed images are clearly not aligned. Adversarial restoration manages to bring back the features to the identity line, while non-adversarial restoration has little effect. This shows that adversarially restored images are not only perceptually closer to the real images, but also semantically more correlated.
Further, Table III compares the performance of different restoration methods in the configuration RO. Adversarial restoration results in a far better segmentation performance when compared to RDN. Efficiency. Table IV reports the inference times for different configurations. The time for encode-decode times for MSH and the segmentation time for DeepLabv3+ were measured on a Quadro RTX6000 GPU, while the encode-decode times for BPG was measured on a Intel Xeon(R) E5-1620 v4 CPU. Inference using dataset restoration (configuration RO) is faster than compression before inference (configuration CC) by 26% and 66%, when the compression method used is MSH and BPG respectively.

C. INRIA Aerial Images Dataset
Segmentation performance. Fig. 11 depicts the segmentation results obtained on the INRIA Aerial Images Dataset (AID) for the same six configurations mentioned in Section B. We observe that the model with configuration CO performs better than CC when the compression method used is BPG. However, the same cannot be said for MSH. We hypothesize that the smoothing artifacts caused when MSH is used destroy the discriminative features and a segmentation model capable of taking advantage of these features in the original image, cannot be learnt. This is especially critical with the AID since there are only a few features that discriminate a building from the background.
The proposed approach of RO performs consistently better than both these configurations with gains up to 3.9 % mIoU. Fig. 12 shows a portion of an image from the dataset along with the segmentation maps predicted by various models.
Interestingly, contrary to the results on the Cityscapes dataset, the performance of configuration OR does not improve over OC. This suggests that the process of restoration causes further damage (over compression) to the discriminative features used by the model trained on the original images for its prediction.

D. Semantic Drone Dataset
Segmentation performance. The segmentation results obtained on the Semantic Drones Dataset (SDD) are shown in Fig. 13. We observe that the models with configuration CC outperform CO consistently for both MSH and BPG. We attribute this result to the lack of semantic information loss in this dataset. A typical object from the SDD covers a significant portion of the image and compression does not significantly affect its recognizability. When we assume little to no semantic information loss, the effects of covariate shift are dominant and as such, the configuration CC performs better. This result shows that the performance of various configurations are dependent on the properties of the dataset.
The proposed approach of RO performs similarly to configuration CC and lies within ±1.5% mIoU of the configuration CC. The lack of significant semantic information loss in SDD affects the effectiveness of dataset restoration.

VI. CONCLUSIONS
The rapid development in sensor quality and increasing data collection rate makes lossy compression necessary to reduce transmission and storage costs. By means of dataset restoration, we enable the incorporation of lossy compression for on-board analysis, greatly mitigating the drop in performance. Dataset restoration is a principled approach, based on our analysis of the various scenarios involving learning and inference with compressed images. Our analysis framework involving covariate shift and semantic information loss can be further extended to other degradations like blur, noise, color and illumination changes, etc.

APPENDIX B ADVERSARIAL IMAGE RESTORATION
We use the FineNet from Akbari et. al [65] (adapted from [55]) with slight changes for our image restoration module.
Following the same notation in [65], the generator architecture is written as c 64 , d 128 , d 256 , d 512 , 9 × r 512 , u 256 , u 128 , u 64 , o 3 where • c k : Conv: 7 7 x k, Instance Normalization, ReLU • d k : Conv: 3 x 3 x k / ↓ 2, Instance Normalization, ReLU • r k : Conv: 3 x 3 x k, Reflection padding, Instance Normalization, ReLU • u k : Conv: 3 x 3 x k / ↑ 2, Instance Normalization, ReLU • o 3 : Conv: 7 x 7 x 3, Instance Normalization, Tanh We use two discriminators, as in [65], operating at two different scales. Akbari et al. rescale the image to half the  Let VGG denote a VGG-Net trained on the ImageNet dataset and M j denote the size of the output of the j th layer of VGG. The output of each of the 5 convolution blocks are considered for the VGG feature distillation loss, which is given by Similarly, the features of the discriminators are also distilled for stable GAN training.
The discriminators are trained using the standard GAN loss.
We use a batchsize of 1 and train the GAN for around 135k iterations. Adam optimizer with β 1 = 0.1 and β 2 = 0.9 is employed. Initially, the learning rate is set to 0.0002 and is reduced by a factor of 10 after 80k iterations.

APPENDIX C RESIDUAL DENSE NETWORK
We use the RDN architecture from [29]. We ask the reader to refer to the paper for the architecture. The following hyperparameters are used: Global layers = 16, Local layers = 6, Growth rate = 32.
We train the CAR model with the objective of maximising MS-SSIM or PSNR. The models are trained using 256x256 patches. A mini-batchsize of 1 is used and the model is trained for around 200k iterations. Adam optimizer is used with initial learning rate of 0.001 which is reduced by a factor 10 at 80k and 150k iterations.