Evaluating the Robustness of Semantic Segmentation for Autonomous Driving against Real-World Adversarial Patch Attacks

Deep learning and convolutional neural networks allow achieving impressive performance in computer vision tasks, such as object detection and semantic segmentation (SS). However, recent studies have shown evident weaknesses of such models against adversarial perturbations. In a real-world scenario instead, like autonomous driving, more attention should be devoted to real-world adversarial examples (RWAEs), which are physical objects (e.g., billboards and printable patches) optimized to be adversarial to the entire perception pipeline. This paper presents an in-depth evaluation of the robustness of popular SS models by testing the effects of both digital and real-world adversarial patches. These patches are crafted with powerful attacks enriched with a novel loss function. Firstly, an investigation on the Cityscapes dataset is conducted by extending the Expectation Over Transformation (EOT) paradigm to cope with SS. Then, a novel attack optimization, called scene-specific attack, is proposed. Such an attack leverages the CARLA driving simulator to improve the transferability of the proposed EOT-based attack to a real 3D environment. Finally, a printed physical billboard containing an adversarial patch was tested in an outdoor driving scenario to assess the feasibility of the studied attacks in the real world. Exhaustive experiments revealed that the proposed attack formulations outperform previous work to craft both digital and real-world adversarial patches for SS. At the same time, the experimental results showed how these attacks are notably less effective in the real world, hence questioning the practical relevance of adversarial attacks to SS models for autonomous/assisted driving.


Introduction
The rise of deep learning unlocked unprecedented performance in several scientific areas [24]. Convolutional (a) (b) (c) (d) (e) (f) Figure 1: Proposed adversarial patches on Cityscapes [5] (b) and CARLA Simulator [6] (e); (c/f) show the corresponding SS predicted by BiSeNet [40]; (a/d) show the corresponding predictions obtained using random patches instead of adversarial ones.
neural networks [16] (CNNs) yielded super-human performance for many different computer vision tasks, such as image recognition [9], object detection [27] [26], and image segmentation [20]. Image segmentation, and semantic segmentation (SS) in particular, is used in autonomous driving perception pipelines [30], mainly for object detection [20]. Despite their high performance, CNNs are prone to adversarial attacks [31]. Most of the literature on adversarial attacks focuses on directly manipulating the pixels of the whole image, hence making the assumption that the attacker has control over the digital representation of the environment obtained by the on-board cameras. This kind of unsafe inputs are called digital adversarial examples.
Although such digital attacks do not transfer well into the real world, they continue to be used to evaluate the robustness of models in safety-critical systems [12,3,19]. Realworld adversarial examples (RWAEs), on the other hand, are physical objects that can be placed in the field of view of a camera, such that the resulting image acts as an adversarial example for the neural network under attack [17]. Thus, RWAEs can induce errors in neural networks without requiring the attacker to access the digital representation of the image, thereby making them a more realistic and dangerous threat to safety-critical systems. This paper. This work focuses on RWAEs, as they repre-sent a potential threat to tasks in autonomous driving today. Although the effects of RWAEs have been studied extensively in the literature for classification and object detection, those on SS remain relatively unexplored. However, SS is an integral part of autonomous driving pipelines [30]. Thus, this paper examines various state-of-the-art models for real-time SS aiming at benchmarking their robustness to RWAEs in autonomous driving scenarios.
Of the several types of RWAEs proposed in the literature [32], the form of attack used in this paper is adversarial patches [4]. This is because attacks that perturb the whole image are not practically feasible in the real world. Conversely, such patches can be easily printed and attached to any visible 2D surface in the driving environment, such as billboards and road signs, thus making them a simple, yet effective attack strategy.
The paper starts by recognizing the shortcomings of the standard cross-entropy loss for optimizing adversarial patches for SS. Thus, an extension to the cross-entropy loss is proposed and integrated in all the performed attacks. This extension forces the optimization to focus on pixels that are not yet misclassified, thus obtaining patches that are more powerful compared to those generated with the standard cross-entropy-based setting [21].
Following this rationale, the robustness of real-time SS models to RWAEs attacks is benchmarked. The paper starts by first examining the case of driving images, crafting adversarial patches on the Cityscapes dataset [5], a popular benchmark of high-resolution images of urban driving. Robust real-world patches are crafted by following the Expectation Over Transformation (EOT) [2] paradigm, which has been extended in this work to attack SS models. Furthermore, a comparison against non-robust patches (without EOT) is presented to question their effectiveness on driving scenes.
Another set of experiments targeted a virtual 3D scenario, for which a stronger adversarial attack is presented and tested. The proposed scene-specific attack, defined in Section 3.4, is a more practical tool for crafting adversarial patches in a realistic autonomous driving scenario. It assumes that the attacker is interested in targeting an autonomous driving scene at a particular corner of a specific town, where information about the position of the attackable 2D surface (in our case, a billboard) is available. To satisfy such requirements we developed and tested this attack using the CARLA simulator, which provides all the needed geometric information. These experiments include a comparison with the EOT-based and non-robust patches, performed by importing them into the CARLA world and placing them on billboards to simulate a realistic study. Figure 1 provides some examples of the effect of our patches on Cityscapes and CARLA.
The last set of experiments were conducted on a real-world driving scenario, which required collecting a dataset within the city, optimizing a patch on it, physically printing said patch on a billboard, and finally evaluating SS models on images containing the printed patch.
To the best of our knowledge, this work represents the first exhaustive evaluation of the robustness of SS models against RWAEs for autonomous driving systems. The results of the experiments state important observations that should be taken into consideration while evaluating the trustworthiness of SS models in autonomous driving. First, they demonstrate that non-robust patches are not good candidates for assessing the practical robustness of an SS model to adversarial examples. Indeed, while they proved to be effective in attacking images related to driving scenes (from Cityscapes), they do not induce any real-world adversarial effect when crafted and tested in a virtual 3D world (based on CARLA). Conversely, robust patches, crafted with EOT or the proposed scene-specific approach, resulted to be less effective than non-robust ones on Cityscapes images, but were capable to accomplish the attack in both virtual 3D world and the real world. Nevertheless, their effectiveness in the latter two cases still resulted to be quite limited, hence questioning the practical relevance of RWAEs.
In summary, the paper makes the following contributions: • It proposes an extension to the pixel-wise crossentropy loss to enable crafting strong patches for the semantic segmentation setting. • It proposes a novel technique for crafting adversarial patches for autonomous driving scenarios that utilize geometric information of the 3D world. • It finally reports an extensive evaluation of RWAEbased attacks on a set of real-time semantic segmentation models using data from the Cityscapes dataset, CARLA, and the real world. The remainder of this paper is organized as follows: Section 2 provides a brief overview of related work existing in the literature, Section 3 formalizes the proposed loss function, pipeline, and attack strategy, Section 4 reports the experimental results, and Section 5 states the conclusions and proposes ideas for future work.

Related Work
Szegedy et al. [33] showed that small well-crafted perturbations when added to the input image were sufficient to fool strong classification networks. [3,19,37,14,13,1,21,29] have studied such attacks for the specific use case of fooling SS models. However, these attacks directly manipulate the pixels of the image. Although such digital perturbations provide a convenient way to provide benchmarks in research, they do no extend well to real-world applications.
A more realistic threat model led to the introduction of RWAEs by Kurakin et. al. [17]. The attacker here is as-sumed to be able to craft adversarial pictures in the physical world, without the ability to manipulate the digital representation of inputs to the neural network. However, this work did not account for factors that affect images of objects in the real-world (e.g., varying viewpoints from which input images could be captured, changes in lighting conditions and so on). Athalye et. al. [2] address this issue by introducing the EOT algorithm. EOT accounts for such factors in the optimisation by modeling them as a distribution of transformation functions applied to the adversarial input. These transformations can be in the form of rotation, scaling, noise, brightening and so on. Then, the idea is to optimize the loss function in expectation across the range of selected transformation functions.
The EOT formulation led to the development of adversarial patches, introduced by Brown et al. [4] to fool image classifiers. They are robust, localized, image-agnostic perturbations, crafted with the EOT paradigm, capable of fooling neural networks when placed within the input scene or added digitally on images.
Although extensive prior work exists to construct such physical attacks for classification [4,28,8], object detection [36,35,18,41], optical flow [25], LiDAR object detection [34], and depth estimation [39], only a few focus on autonomous driving tasks, since testing the adversarial robustness is more challenging, as it requires controlling the 3D outdoor environments. Other works [41,35] have shown CARLA to be a viable solution in alleviating this issue by crafting and evaluating adversarial situations in virtual 3D environments. This paper also heavily relies on CARLA to evaluate how the optimized adversarial patches translate to a 3D world.
The work closest to ours is the one by Nakka et. al. [21], who attempted to fool a variety of SS models via local attacks (i.e., creating pixel perturbation in a specific area of the image). Despite the attacks being local, the objective of their study was not to evaluate the robustness to real-world attacks, which is instead the main focus of this paper. To the best of our knowledge, such a study is missing in the literature for the case of SS models, which represent essential components in an autonomous driving perception pipeline [30].
Additionally, this paper also improves the loss function used for generating patches. Section 3.5 presents a more general formulation of the cross-entropy loss for the SS setting, designed to optimize more powerful and effective adversarial examples, while all the previously mentioned papers use the standard pixel-wise cross-entropy loss.

Attack Formulation
This section presents the design of adversarial patches for semantic segmentation (SS), starting with a short recap of the basic notions behind SS. The patch optimization scheme for both the EOT-based and the scene-specific attacks is then presented. Finally, the proposed loss function is introduced.

Background on SS
An image with height H and width W can be represented as x ∈ [0, 1] H×W ×C , where C is the number of channels. An SS model returns f (x) ∈ [0, 1] H×W ×Nc , where N c is the number of classes. This output represents the predicted class-probability scores associated to each image pixel. In particular, f j i (x) ∈ [0, 1] denotes the predicted probability score for the i-th pixel of the image corresponding to the class j. Consequently, the predicted semantic segmentation SS(x) ∈ N H×W is computed by extracting those classes with the highest probability score in each pixel: The ground truth for the SS of x is defined as y ∈ N H×W , and assigns the correct class (in {1, ..., N c }) to each pixel. The performance of the SS models is evaluated by computing the cross-entropy loss L CE (f i (x), y i ) = −log(f yi i (x)). Thus, for each pixel i, the model's prediction f i is compared against the ground truth class y i .

Patch-based attack pipeline
Both the EOT-based and the scene-specific attacks share a similar pipeline, which is explained in the following paragraph and illustrated in Figure 2.
An adversarial patch of heightH and widthW is denoted as δ ∈ [0, 1]H ×W ×C , whereH < H andW < W . This patch is then added to the original image x to obtain a patched imagex. Thus, the output of the SS model on this patched image would now be f (x).
The attacks considered are both untargeted. This means that the objective is to maximize a certain loss function L, without forcing the classification of pixels towards any specific class.
Inspired by the EOT method [2], the idea is to find an optimal patch δ * (starting from a random patch) that maximizes the loss L for all the patched images in expectation according to the distribution of transformations used to apply the patch δ on the image set X.
Formally, we need to define: • A set of appearance-changing transformations Γ a , for instance changes in illumination (brightness, contrast) and noise (uniform or gaussian). These transformations are directly applied to the patch, so obtaining a transformed patch ζ a (δ), where ζ a ∈ Γ a . They are used to make the patch robust to illumination changes and acquisition noise. • A patch placement function η that defines which portion of the original image x is occupied by the patch. This is the only part of the pipeline that differs between the two proposed attacks, and is discussed further in the following subsections. • A patch application function g(x, ζ a (δ), η) that replaces a certain area of x with ζ a (δ) according to η and returns the patched imagex. These functions are sufficient to define both the EOTbased attack, which uses randomized spatial transformations to place the patch onto the image, and the scenespecific attack, which uses a precise projective transformation to enhance the accuracy of the patch placement.

EOT-based patch attack
The classic EOT-based attack, as in previous work [4] [38], uses a set of combinations of spatial transformations Γ, including translation and scaling, from which the patch placement function η is selected.
The parameters for each transformation are randomized within a pre-defined range. Section 4 provides a more detailed explanation of the set of transformations used. The optimal patch is then defined as In practice, the optimal patch is computed via an iterative optimization process. At each iteration t, the pixels values of the patch are modified in the direction of the gradient of the loss function computed with respect to the patch: where represents the step size. L consists of a weighted sum of multiple loss functions. The adversarial patch effect is obtained through the optimization of the adversarial loss L adv (discussed further in subsection 3.5). Additionally, to ensure that the patch transfers well to the real world, two losses are added to account for the physical realizability of the patch (see supplementary material 1 ): smoothness loss L S and non-printability score L N .

Scene-specific patch attack
To provide a more realistic approach for autonomous driving environments, this work proposes an alternative attack methodology that exploits the geometrical information provided by the CARLA Simulator [6].
Here, the key assumption is the availability of an attackable 2D surface, e.g., a billboard, with a fixed location in close proximity to the road. The CARLA simulator features the possibility to extract camera extrinsic and intrinsic matrices (details in supplementary material), and the pose of the attackable surface. The billboard-to-image transformations can be computed using a 3D roto-translation composition, which allows the patch to be warped accordingly, thus obtaining higher precision in applying the patch to the attack surface.
This attack uses the same optimization pipeline as before, with one major difference: instead of placing the patch randomly, as in the previous attack, correct projective transformations are used to determine the placement of the patch on the attackable surface. Hence, η is no longer randomized, but is computed for each image in the dataset.
This method allows crafting precise attacks that are optimised for the region of the town that the attacker is interested in. The attacker would need to collect several images, from different viewpoints, of the desired attackable surface, along with the corresponding intrinsic and extrinsic matrices. This approach to image collection also implies that EOT is no longer needed for patch placement, thereby simplifying the optimization process.
The downside of this approach is that a digital representation of the target scene is required to accurately capture the required matrices. Although CARLA provides the possibility to import cities via OpenStreetMaps (https:// www.openstreetmap.org/), it requires some amount of manual effort to properly model 3D meshes to include objects in this simulated world. These objects need to be properly designed to ensure that the patches optimised in simulation transfer well to the real-world. Although this paper does not investigate CARLA-to-real-world transfer issues, future work will address this problem to improve the proposed methodology and adapt it for real-world attacks. Section 4 provides a comparison of this method against the EOT-based attack.

Proposed loss function
Cross-entropy (CE) is a popular choice as adversarial loss. Pixel-wise CE has been shown to work well when crafting an untargeted digital attack (i.e., by directly adding a perturbation r to the pixels of a digital image) [21] [3]. This is formulated as: L adv (f (x + r), y) = 1 |N | · i∈N L CE (f i (x+r), y i ), where N = {1, ..., H ×W } denotes the whole set of pixels inx. However, modifications can be introduced to this formulation to allow crafting stronger attacks for fooling SS models.
Following previous notation, letÑ = {1, ...,H ×W } ⊆ N denote only the pixels that correspond to the patch δ. Then, Υ defines a subset of image pixels that do not belong to the patch and are still predicted correctly by the model with respect to the trusted ground truth label y: Using Υ, the previous pixel-wise CE loss computed on N \Ñ can be split into two distinct terms: (4) Lx M describes the cumulative CE for those pixels that have been misclassified with respect to the ground truth y, while Lx M refers to all the others.
Note that both Lx M and Lx M do not consider pixels of the patch, which have been discarded to focus the optimization on attacking portions of the image away from the patch. By computing these separate contribution to the total loss, we avoid that the contribution of the non-misclassified pixels gets obscured by the other term, which is a problem we found during preliminary tests. Hence, the adversarial loss function gradient is redefined as follows: where γ ∈ [0, 1] is a parameter that determines whether the optimization should focus on decreasing the number of correctly classified pixels or improving the adversarial strength for the currently misclassified pixels. The rationale of γ is to provide an empirical balancing between the importance of L M and L M at each iteration t depending on the number of pixels not yet misclassified.
Moreover, an adaptive value of γ = |Υ| |N \Ñ | has been proposed to provide an automatic tuning of γ at each iteration. The idea is to initially focus on boosting the number of misclassified points. Over time, as this number increases, the focus of the loss function gradually shifts toward improving the adversarial strength of the patch on these wrongly classified pixels.
Section 4 provides an extensive analysis of the proposed loss function by comparing multiple values of γ with the standard pixel wise CE measured both on N \Ñ and N (which is used by [21]), suggesting that our formulation is indeed more general and effective for this kind of attack.

Experimental results
This section describes the experimental setup and the results achieved with the proposed attacks. First, the proposed loss function is evaluated for different values of γ comparing its effectiveness against the standard pixel-wise CE, showing that it is a better alternative for this kind of problems. Following this, the results of patches crafted with and without EOT are presented on the Cityscapes dataset.
Subsequently, the results obtained with the scenespecific attack on three CARLA-generated datasets are presented. These results are compared against the EOT-based attack to show the improved effectiveness of this formulation. Finally, some preliminary results of real-world adversarial patches are presented. A more detailed analysis of all models tested against these attacks can be found in the supplementary material.

Experimental setup
The experiments were performed on a set of 8 NVIDIA-A100 GPUs, while the CARLA simulator was run on a system powered by an Intel Core i7 with 12GB RAM and a GeForce GTX 1080 Ti GPU.
All experiments were performed in PyTorch [23]. The optimizer of choice was Adam [15], with learning rate set to 0.5 empirically. The effect of the adversarial patches on the SS models was evaluated using the mean Intersectionover-Union (mIoU) and mean Accuracy (mAcc) [20].
The repository link with the code used for all the experiments was not included for double-blind requirements, but it will be inserted upon request or acceptance.
Datasets The experiments in the case of driving images were carried out using the Cityscapes dataset [5], a popular benchmark for urban scene SS. The dataset consists of 2975 and 500 high resolution images (1024 × 2048) for training and validation, respectively. The experiments reported in this paper were conducted on 250 images randomly sampled from the training set. Conversely, the entire validation set was used to evaluate the effectiveness of the patches.
The CARLA simulator was used to provide a 3D virtual scenario. This set of experiments was performed in Town01, one of the built-in maps provided with the simulator, with 'CloudyNoon' as the preset weather. To mimic the settings of the Cityscapes dataset, RGB images of size 1024 × 2048, along with their corresponding SS tags, were collected by placing a camera on-board the ego vehicle.
The SS models trained on Cityscapes had to be finetuned to ensure good performance on CARLA images. 600 images for fine-tuning, 100 for validation, and 100 for testing were collected by spawning the ego vehicle at random positions in Town01. Additional details about the finetuning can be found in the supplementary material.
To study the effectiveness of the patches in CARLA, the map of Town01 was manually edited to include three billboards. Thus, without-EOT, EOT-based, and the scenespecific attacks were carried out at three different locations within Town01. The datasets for the attacks were collected by spawning the ego vehicle at random locations within the proximity of each billboard to emulate varying viewpoints  Table 1: mIoU and mAcc of the tested models on Cityscapes (pre-trained) and our CARLA dataset (finetuned).
from which the patch might be captured in the real-world. For each of the three billboards, 150 training images, 100 validation, and 100 test images were gathered. Details about the position and orientation of the billboard and the camera were stored to compute the roto-translations used in the scene-specific attack. Lastly, to study the effects of adversarial patches in the real world, an additional dataset of 1000 images, hereafter referred to as Patches-scapes, was collected by mounting an action camera on the dashboard of a vehicle using a setup similar that of the Cityscapes dataset, and then driving the vehicle within the streets of our city. Models The attacks studied in this paper were evaluated using DDRNet [11], BiSeNet [40], and ICNet [42], which represent the state-of-the-art in real-time SS, making them preferable for the use case of autonomous driving. Additionally, PSPNet [43] was included in the study for the EOT-based attack on the CityScapes dataset, but not for the scene-specific attack on CARLA, since we are interested in real-time performance.
All the models were loaded with the pre-trained weights provided by the authors (further specifications are provided in the supplementary material. Table 1 summarizes the performance of these models on both the Cityscapes and CARLA validation sets.

EOT-based patches on Cityscapes
The Cityscapes dataset is used to optimize two types of patches on the same training images, one with EOT and the other without EOT (non-robust). Three different patch sizes are studied: 150 × 300, 200 × 400, and 300 × 600 pixels.
The non-robust patches (without EOT) were optimized by placing them at the center of the image at each training iteration (i.e., η(·) = fixed position) and applying no appearance transformations (i.e., Γ a = ∅).
Conversely, the robust optimizations with EOT apply multiple digital transformations. Γ a includes only Gaussian noise with standard deviation 5% of the image range. Γ includes random scaling (80% − 120% of the initial patch size) and random translation defined as follows: if (c x , c y ) is the center of the image, the position of the patch is randomized within the range (c x ±r ·W /2 , c y ±r ·H/2), wherer ∈ [0, 1] is a uniform random value. The translation range was kept limited, rather than considering the full image space, to ensure a greater stability and faster convergence. The patches were optimized over 200 epochs. Figure 3 reports the mIoU obtained by training patches with the pixel-wise CE computed on N (used by [21]) and N \Ñ compared to the extended CE loss proposed in this paper, with γ ∈ {0.5, 0.6, 0.7, 0.8, 0.95, 1.0, |Υ| |N \Ñ | }. Among the models evaluated in the paper, ICNet [42] appears to be most robust on the Cityscapes dataset. Thus, the loss functions are studied by optimizing a 200 × 400 patch with and without EOT on ICNet.

Loss functions analysis
For all the tested values of γ, our formulation converges to a higher attack effect (i.e., smaller mIoU) with lesser number of epochs than the one based on the pixel-wise CE. Experiments without EOT show that all the compared implementations converge after only 10 epochs. In the EOT case, the advantages are even more evident: our proposed formulation converges at almost 25 epochs, while the CE cases still reduce slowly at 200 epochs (nearly 6 hours of optimization time).
The same study was performed for the scene-specific attack in the CARLA virtual world, and produced similar results, reported in the supplementary material.
Adversarial patch effects. Table 2 reports how varying the patch size affects each of the SS models. We used the adaptive γ (i.e., γ = |Υ| |N \Ñ | ) that has shown the best overall effect among multiple experimental tests. Figure 4 illustrates the effects of the optimized patches on the BiSeNet model. As expected, the non-robust patches (without EOT) obtain better attack performance with respect to the ones optimized with EOT. This is because the optimization process is simpler when not considering the randomized transformations. However, it is important to note that these patches would not be transferrable to the real world, and are not robust even to simple transformations [10,22].

Scene-specific patches on CARLA
The scene-specific attack was performed on the same set of models as defined earlier. Each of these models were first fine-tuned on images generated via CARLA. The performance of these fine-tuned models on the CARLA datasets is summarized in Table 1. Please note that the mIoU score is computed as an average of the per-class IoU scores, which, for CARLA, might get to 0 for some classes due to the presence of a few pixels belonging to non-common objects.
As described in Section 3.4, the patch is optimized to be adversarial for a specific urban scene by reprojecting it on the attackable 2D surface, which, in this work, is a billboard placed in three different spots in the Town01 map of CARLA. This section reports the effect of the scene-specific attack compared against the non-robust (without-EOT) and   Since the objective of this work is to craft RWAEs, the performance of the attacks is evaluated by measuring the mIoU and mAcc scores on additional scene-specific datasets. These additional datasets are produced by collecting the same images of the validation set of each scenespecific dataset, but with a single major modification: the billboard object is modified in the Unreal Editor [7] by applying the optimized patch as a decal object, which is a way to stick an image on a surface in the virtual environment. This method should provide a simulated real-world application of the patches, since they are no more applied directly on the image, but the image itself includes the patched billboard. Table 3 summarizes the results obtained on these three additional scene-specific datasets, with a random patch, a non-robust patch, an EOT-based patch, and a scene-specific patch. Figure 5 shows a comparison of all the discussed attacks on DDRNet.
For almost all the combinations of scene and network, the scene-specific attack outperforms the EOT-based attack, confirming that the scene-specific attack, for this kind of problems, is a better alternative to the EOT formulation for the placement of the patch within the image. The only case where the the two attacks show comparable performance is for scene 3, where the billboard is almost perpendicular to the camera plane, allowing the EOT method to cover realistic patch placing functions.
It is also worth noting that SS models are rather robust to adversarial patch attacks in general. Although it is possible to craft adversarial patches that cause a section of the image to be wrongly segmented, it tends to be more difficult than fooling models for tasks such as classification. Additional details are reported in the supplementary material.

Real-world patches
In order to prove that the proposed pipeline can be used for a real-world attack, we use the Patches-scapes dataset (described in Section 4.1) to craft an adversarial real-world  Table 3: Adversarial patch results on the three scene CARLA datasets. The Table reports the mIoU and mAcc obtained with random, non-robust (without EOT), EOT-based and scene-specific patches.  patch using the EOT-based patch attack. This section presents the results of an attack in Figure 6. Although the presence of the optimized patch does alter a significant area of the predicted SS (while the random patch does not), portions of the image far from its position are not affected. Furthermore, the attack performance decreases as we move the patch away from the camera (details provided in the supplementary material.The patch is optimized for 200 epochs on the pre-trained version of ICnet (since it showed good performance on the Patches-scapes dataset) and printed as a 1m × 2m poster. Testing adversarial patches for autonomous driving in the real world poses a series of difficulties that heavily limited the tests. First, it is not easy to find a urban corner with good prediction accuracy, and which is not crowded with moving vehicles (which might be dangerous). Second, the patch must be printed in the highest resolution possible on a large rigid surface, which might get expensive. Furthermore, since weather conditions are not controllable and change throughout the day, results can diverge from what is expected.
The scene-specific attack, which requires additional geometric information, could not be implemented at the time of writing, but will be considered in future work.

Conclusions and future work
This paper presented an extensive study of the adversarial robustness of semantic segmentation models. This was accomplished by extensively evaluating the effect introduced by adversarial patches, to investigate the limits of real-world attacks for segmentation neural networks in an autonomous driving scenario. Carrying out the investigations with increasingly "real-world" benchmarks, we studied the effect of non-robust and EOT-based patches on the Cityscapes dataset, on a virtual 3D scenario, and in a realworld setting. We also introduced a new method called scene-specific attack, which improves the EOT formulation for a more realistic and effective patch placement.
The novel loss function introduced in the paper enabled to advance the state-of-the-art for adversarial patches optimization methods, as it proved to be a more general and efficient alternative to the classic cross-entropy function for this kind of problems.
This exhaustive set of experiments practically opens to a new point of view for studying SS models in autonomous driving. Although the proposed attacks were able to reduce the baseline model accuracy, the SS models proved to be somehow robust to real-world patch-based attacks. This was especially noticeable when the tests were performed in more realistic settings using CARLA and the real world, where, in most cases, the patch only affected the proximity of the attacked surface.
Nevertheless, this is a promising result, since it shows how the prediction provided by these models is not easily corruptible, especially in real-world scenarios. This is in contrast with previous work on patch-based adversarial attacks against classification and object detection models.
Future work will further investigate the robustness properties of these models, introducing defense mechanisms and trying to enhance the robustness of SS model by adding a temporal dimension.