This work is supported in part by the US National Science Foundation under grants 2213700, 2217071, 2008208, 1955151. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Appendix
Dataset Description
We test the proposed attack approaches on the following three different datasets of varying domains and levels of security and trust required. For a standard classification task such as digit or bird identification, a wrong concept set is not a very concerning outcome - however for a medical diagnosis - a wrong concept set can be catastrophic.

ConceptMNIST: We augment the original MNIST dataset which already consists of images of various handwritten numbers from 0-9 along with their labels with carefully constructed, human understandable concepts to emulate real-world image recognition tasks. We propose to augment the dataset with a combination of overlapping and non-overlapping concepts as detailed below:

Non-Overlapping Concepts: Certain real-world concepts are shared by samples belonging to only a specific class instance - for example, in medical diagnosis of cancer, the presence of a specific malicious cell. We attempt to emulate such concepts by creating a vector of concepts which are specific to only a class of digits in MNIST. The easiest way to perform this is to one-hot encode labels for each handwritten digit to form a vector of size 10. For example for the number 4, the vector would be represented by [0,0,0,0,1,0,0,0,0,0] where each entry corresponds to a binary concept we call- “isNum\(_i\) ”. Note that no 2 digit classes can share the same vector representation.

Overlapping Concepts: Next, we emulate real-world concepts which are shared by samples from multiple classes - for example, in the CUB bird identification dataset, the color of the bird being black is shared by multiple bird classes. For MNIST, we consider 2 physical characteristics of numbers in the image - presence of a curved line and a presence of a straight line in their visualizations (the way they are printed in typeset formatting). We call these concepts as [“CurvedLine:present” , “StraightLine:present”] and are respectively set to 1 if the number can be constructed by only straight lines or curved lines in the standard typeset format. For example, the number 6 has no straight lines but only curved lines - hence is represented as [1,0] while the number 5 has both straight and curved lines and hence represented as [1,1]. Meanwhile, the number 7 has only straight lines so can be represented as [0,1]. Note that multiple digits can share similar representations, for example both numbers 0 and 6 will be represented by [1,0].

Both the “isNum\(_i\) ” and [“CurvedLine:present” , “StraightLine:present”] as described above are concatenated - resulting in a size 12 concept vector for each image. Note that this construction of concepts is completely up to user's subjective convenience and can be modified according to what the user thinks best interprets a prediction result. For a visual description of concept annotations, refer to Figure REF .

CUB: The Caltech-UCSD Birds-200-2011 dataset [1]} consists of 11,788 photos of 200 different classes of birds. The task is to classify each photo in one of the categories. The original dataset consists of 312 binary concepts representing physical traits. The dataset is processed exactly as described in [2]} - final dataset consists of 112 annotated concepts for each class of birds with each concept representing various physical traits of birds like wing color, beak size, etc. It is ensured all birds of same class contain same concept traits.

OAI: Osteoarthritis Initiative (OAI) dataset [3]} consists of X-ray images and clinical data for about 36,000 patients over 4 years of study who pose a risk of knee osteoarthritis. The task is to perform X-ray grading and classify each image into 4 different risk categories corresponding to Kellgren-Lawrence grade (KLG) which measures the severity of osteoarthritis, with higher numbers implying higher severity. Each image is annotated with 10 concepts which include medical diagnosis from X-ray images such as bone spacing. For a more comprehensive dataset details and processing description, refer to [2]} and [5]}. Note that OAI dataset is not publicly available and requires special permissions as detailed in [5]}.

<FIGURE>
Model Training Objectives
[2]} proposes 3 different training strategies for Concept Bottleneck Models (CBMs), we only consider joint and sequential model training strategy for all 3 datasets. As [2]} have demonstrated, joint model training mostly outperforms sequential training approach on task errors and has comparable performance on the concept errors. However, as already discussed previously, both sequential and joint model training have respective advantages with joint offering higher prediction accuracies while sequential offering more robust concept learning. In addition, joint model training is more time-efficient due to being end-to-end in nature and is more flexible to downstream tasks.
Although [2]} utilizes 3 different training objectives (Section 3, [2]}), we formulate them in a unified, general purpose objective function for all types of model training.
\(\mathcal {L}_{f,g} = \Sigma _{i}[\gamma *l(f(g(x_{i});y_{i}) + \lambda *\Sigma ^T_{j}l(g_j(x_{i});c^j_{i}) ]\) 

Sequential Objective: We use Equation REF  with \(\gamma =0\)  to first train the model \(g\)  and subsequently train set \(\gamma =1\)  and \(\lambda =0\)  for training the model \(f\) .

Joint Objective: Equation REF  can be directly used with appropriate weights for \(\gamma \)  and \(\lambda \)  for training both models \(f\)  and \(g\)  in tandem.

Hybrid Objective: We use Equation REF  with \(\gamma =0\) , to first train the model \(g\)  and subsequently train set \(\gamma =1\)  and \(\lambda \)  to appropriate concept weight value for training both models \(f\)  and \(g\)  in tandem.

As discussed before, for datasets CUB and OAI, we set the concept weights \(\gamma \)  to 0.001 and 1 respectively (as reported in [2]}), while for ConceptMNIST we set \(\gamma \)  to 0.5 for all subsequent experiments. The learning rates \(\omega \)  for C-MNIST is 0.01 and \(\omega ^{^{\prime }}\)  is set as 0.005. For CUB, we follow [12]} with \(\omega \)  at 0.01 and \(\omega ^{^{\prime }}\)  0.005 while for OAI \(\omega \)  and \(\omega ^{^{\prime }}\)  is set as 0.0008 and 0.0005 respectively. We use SGD momentum as the optimizer and follow tuning rules in [13]} to set the hyperparameters of the optimizer.
<TABLE>
Model Architectures
All CBMs consist of two distinct networks - \(f\) and \(g\). We utilize similar model architectures proposed in [2]} for CUB and OAI datasets. The network \(g\)  maps inputs in \(\mathbb {R}\)  to concept space while the network \(f\)  maps concepts to the output space.

ConceptMNIST: The network \(g\)  consists of 2 convolutional layers with 32 channels each, along with a maxpool layer in between followed by a fully connected layer. The network \(f\)  consists of a fully connected layer. The fine-tuning is done end-to-end with the weight 0.1 given to the concept loss. We train the model with batch size set to 64, learning rate set to 1e-4 for a total of 20 epochs.

CUB: The network \(g\)  consists of a Inception V3 pre-trained on Imagenet and the network \(f\)  consists of one layer MLP. The fine-tuning is done end-to-end with the weight 0.001 given to the concept loss. The rest of the hyperparameters are unchanged from [2]}.

OAI: For the OAI dataset, we utilize the architecture provided in [2]} which consists of a ResNet-18 with last 12 layers unfrozen. The prediction model consists of a 3 layer MLP with 50 neurons each in the first 2 layers while the last layer provides a regression score. The training hyperparameters are used unchanged.

The final benchmarking task and concept losses are reported in the main paper. As can be seen both models trained using standard objective and with robust concept learning with adversarial training obtain similar task and concept errors.

Attack Formulation
Recall that our attack framework's objective follows the form detailed in Equation REF .
\(&\max _{||\delta ||_{\infty } \le \epsilon _{thresh}} \quad G(\mathcal {U}(x;f,g),\mathcal {U}(x+\delta ;f,g)) \quad \\ &\text{s.t.}\quad argmax f(x+\delta )=argmaxf(x), \quad x + \delta \in [0,1]^{D},\) 

The goal of the objective function is to maximize the difference of the generated concept-based explanations before and after the performed adversarial attacks. However importantly, the objective function follows 2 constraints - the first which ensures that model predictions for both the original and attacked samples remain identical, and second that the generated adversarial perturbation is imperceptible.
The difference between original and attacked sample's concepts which needs to be maximized, denoted as \(\mathcal {D}\)  differs for various attacks which are detailed below.

For erasure attack, we attempt to minimize the concept score of a selected target concept, hence the selected set contains only the target concept score denoted by \(C_{S_{target}}\)  such that:
\(C_{S_{target}} = &\lbrace s_{target} | s_{target} \in \mathbb {I}(\sigma (\hat{g}(x_i)){\rbrace }\\ & \mathcal {D}(x_i) = \Sigma ~(C_{S_{target}})\) 

For introduction attack, we attempt to maximize the concept scores of concepts which are not-relevant to the initial model predictions. Therefore, we include all non-relevant concept scores in the selected set denoted by \(C_{S_{non-rel}}\)  such that:
\(C_{S_{non-rel}}& = \lbrace s_i | s_i \notin \mathbb {I}(\sigma (\hat{g}(x_i)) \rbrace \\ &\mathcal {D}(x_i) = \Sigma (C_{S_{non-rel}})\) 

For confounding attack, we simultaneously maximize concept scores of all relevant concepts and minimize concept scores of non-relevant concepts. The final objective is a weighted sum of relevant and non-relevant concept sets. Therefore, we include all relevant and non-relevant concept scores in the selected set denoted by \(C_{S_{rel}}\)  and \(C_{S_{non-rel}}\)  such that:
\(&C_{S_{rel}} = \lbrace s_i | s_i \in \mathbb {I}(\sigma (\hat{g}(x_i)) \rbrace \\ &C_{S_{non-rel}} = \lbrace s_i | s_i \notin \mathbb {I}(\sigma (\hat{g}(x_i)) \rbrace \\ &\mathcal {D}(x_i) = \Sigma (C_{S_{non-rel}}) + \frac{\gamma }{\beta }*\Sigma (C_{S_{rel}})\) 

However, it is not enough to maximize only \(\mathcal {D}\) , without considering its effect on model prediction. Note that the first metric specifically ensures model predictions should remain identical after attack. To this effect, we propose to maximize the prediction score of the original sample as output by the prediction model \(\hat{f}\) . We capture this information in \(\mathcal {P}\)  which would also be maximized in conjunction with \(\mathcal {D}\)  to ensure model predictions remain identical.
We have 2 different types of model outputs - standard softmax label-wise outputs for CUB and ConceptMNIST datasets, and a single regressed value for OAI dataset. The different forms of \(\mathcal {P}\)  are detailed below:

Label-wise outputs (like CUB, ConceptMNIST):
\(\mathcal {P}(x_i) = -\Vert \hat{f}(\hat{g}(x_i))_{y_{gt}}\Vert _{2}\,\, \\ \text{where $y_{gt}$ is index of predicted label}.\) 

Continuous outputs (like OAI):
\(\mathcal {P}(x_i) = -|(f(g(x)) - y_{gt})|\,\, \\\text{where $y_{gt}$ is predicted value}. \) 

The third constraint is ensured by stopping optimization before a pre-defined \(\epsilon _{thresh}\)  value which is kept small enough to ensure visual imperceptibility.
The final objective function \(\mathcal {L}\)  is thus a weighted sum of \(\mathcal {D}\)  and \(\mathcal {P}\)  to maximize the difference in concept values as well as ensure identical model predictions, which is given as follows:
\(\mathcal {L}(x_i) = \alpha *\mathcal {P}(x_i)+\beta *\mathcal {D}(x_i).\) 
Algorithm implementation details can be found in Section REF  for erasure, introduction and confounding attacks.

Attack Settings
Erasure.
We set the \(\epsilon \)  value to \(10^{-4}\) . We limit our optimization process to 1000 steps and maximum \(L_{\infty }\)  value to \(\epsilon _{thresh}\)  so as to ensure visual imperceptibility of added noise. The \(\epsilon _{thresh}\)  for CUB, ConceptMNIST and OAI datasets is set to 0.2, 30 and 10 respectively for all attacks. We also report results for increasing \(L_{\infty }\)  values in Table REF . For all datasets - \(\alpha \)  and \(\beta \)  are both set to 1.
Introduction.
As before, we set the \(\epsilon \)  value to \(10^{-4}\) . We limit our optimization process to 5000 steps and maximum \(L_{\infty }\)  value to 0.2, 30, 10 as respectively for CUB, ConceptMNIST and OAI datasets as before. We set \(\beta \)  as 5 to give higher weight to non-relevant concepts and set \(\alpha \)  to 1.
Confounding. Confounding attack is a combination of both erasure and introduction attacks. Again we set the \(\epsilon \)  value to \(10^{-4}\) , limit our optimization process to 5000 steps and maximum \(L_{\infty }\)  value to 0.2, 30 and 10 for CUB, ConceptMNIST and OAI datasets respectively. For CUB and ConceptMNIST datasets, the value for \(\beta \)  and \(\gamma \)  is set to 10 and 5 respectively. For OAI dataset, the value for \(\beta \)  is set at 0.5.

Attack Algorithms
This section details the specific algorithms to perform general purpose erasure, introduction and confounding attacks. Algorithm REF  details general form of the Erasure attacks as discussed. Algorithm REF  details a general form of confounding attacks. Note that confounding attack with \(\gamma \)  set to 0 is identical to introduction attack.
[h]
Algorithm to perform concept erasure attack

Adversarially perturbed image with same prediction label and concept set with targeted index absent
For each image \(x_i\)  in test set
\(C^{orig}\leftarrow \sigma (\hat{g}(x_i))\) 
\(y_{gt}\leftarrow argmax\; \hat{f}(\hat{g}(x_i))\) 
\(x^{\prime }_i \leftarrow x_i\)  
\(step\leftarrow 0\)  
\(flips\leftarrow 0\) 
\(argmax\; \hat{f}(\hat{g}(x^{\prime }_i)) == y_{gt}\)  and
step \(\le \)  N

\(C_{S_{target}}\leftarrow |\sigma (\hat{g}(x^{\prime }_i))|_{y_{gt}}\) 
\(\mathcal {L}_{x_i}\)  = \(\alpha *\Vert \hat{f}(\hat{g}(x^{\prime }_i))_{y_{gt}}\Vert _{2}\) 
\(+\beta *\Sigma (C_{S_{target}})\) 
\(x^{\prime }_i \leftarrow x^{\prime }_i + \epsilon *sign(\nabla _{x^{\prime }_i}\mathcal {L}_{x_i})\) 
\(x^{\prime }_i \leftarrow clamp(x^{\prime }_i,0,1)\) 
\(L_{\infty }(x^{\prime }_i)\ge \)  \(\epsilon _{thresh}\) 
break
\(\sigma (C_{idx}^{pred})<0.5\)\(flips\leftarrow flips+1\)continue
\(step\leftarrow step+1\) 

[h]
A standard algorithm to perform Introduction and Confounding attacks

Adversarially perturbed image with same prediction label and different concept set
For each image \(x_i\)  in test set 
\(x^{\prime }_i\leftarrow x_i\) 
\(C^{orig}\leftarrow \sigma (\hat{g}(x_i))\) 
\(S_{non-rel}\leftarrow \mathbb {I}(C^{orig}<0.5)\) 
\(S_{rel}\leftarrow \mathbb {I}(C^{orig}\ge 0.5)\) 
\(y_{gt}\leftarrow argmax\; \hat{f}(\hat{g}(x_i))\) 
\(step\leftarrow 0\)
\(argmax\; \hat{f}(\hat{g}(x^{\prime }_i)) == y_{gt}\)  and step \(\le \)  N

\(C_{pred}\leftarrow \sigma (\hat{g}(x^{\prime }_i))\) 
\(\mathcal {L}_{x_i}\)  = \(\alpha *\Vert \hat{f}(\hat{g}(x^{\prime }_i))_{gt}\Vert _{2}\) 
\(+\beta *\Sigma (C_{S_{non-rel}})\) 
\(+\gamma *\Sigma (C_{S_{rel}})\) 
\(x^{\prime }_i \leftarrow x^{\prime }_i + \epsilon *sign(\nabla _{x^{\prime }_i}\mathcal {L}_{x_i}))\) 
\(x^{\prime }_i \leftarrow clamp(x^{\prime }_i,0,1)\) 
\(L_{\infty }(x^{\prime }_i)\ge \epsilon _{thresh}\) 
break
\(step\leftarrow step+1\)

Evaluation Metrics
In the following, we describe the evaluation metrics in greater detail.

Percentage of concepts flipped: Erasure and confounding attacks for binary valued concepts attempt to “flip” concepts already present in a sample, i.e., the concept is no longer predicted as being relevant to prediction after the attack. Equation REF  details the calculation of percentage of flipped concepts for each attacked sample. The terms \(C^{orig}\)  and \(C^{pert}\)  denote the concept sets of the original sample and the sample after attack respectively. The indicator function \(\mathbb {I}\)  returns 1 if concept \(i\)  is present in the set, else returns 0. Note that \(C^{pert}\)  is generated separately for each target concept.
\(\text{\% Flipped} = \frac{1}{|C^{orig}|}\Sigma _{i}^{|C^{orig}|}\Vert \mathbb {I}(C^{orig}_{i}) - \mathbb {I}(C^{pert}_{i})\Vert \) 
However, the notion of concepts “present” does not make sense in datasets with real-valued continuous concept annotations. This is because each value of each concept has a certain degree of physical manifestation in the sample - implying a concept is always present. Hence, we consider a concept removed if the absolute difference between the original concept set and the concept set after attack is greater than a threshold value \(\delta _{thresh}\) . Equation REF  details the calculation of percentage of flipped concepts for each attacked sample. The indicator function \(\mathbb {I}\)  returns 1 if the condition is true, else returns 0. Note that \(C^{pert}\)  is generated separately for each target concept.
\(\text{\%Flipped} = \frac{1}{|C^{orig}|}\Sigma _{i}^{|C^{orig}|}\Vert \mathbb {I}(|C^{orig}_{i} - C^{pert}_{i}|>\delta _{thresh}))\Vert \) 

Percentage of concepts introduced and retained: Introduction attacks attempt to introduce concepts which are not already present in the sample's concept prediction and are non-relevant to sample's prediction. To this effect, we report two relevant metrics - percentage of concept introduction which refers to the number of new concepts introduced as compared to the original concept set detailed in Equation REF . As we are only interested in new concept introductions, we also report the concept retention percentage refer to the number of concepts which are shared with the original concept set as detailed in Equation REF .
\(\text{\%Concepts Introduced} = \frac{|C^{orig} \cap C^{pert}|}{|C^{orig}|}\) 
\(\text{\%Concepts Retention} = \frac{|C^{orig} \cup C^{pert}|}{|C^{orig}|}\) 

Jaccard Similarity Index (JSI): JSI measures the simmilarity (or dissimilarity) between two different concept sets. We utilize the JSI value to understand the dissimilarity of concepts sets predicted before and after the confounding attacks. Mathematically, it can be thought of the percentage of shared set elements with respect to combined total set elements also called Intersection over Union. In general, the lower the JSI value, the more different the input sets are. The exact mathematical formulation is detailed in Eqution REF .
\(\text{JSI} = \frac{|C^{orig} \cup C^{pert}|}{|C^{orig} \cap C^{pert}|}\) 

[h]

Robust Concept Learning (RCL)
OutputOutput
Trained models \(\hat{f},\hat{g}\)  with increased concept robustness
InputInput

\(epoch\leftarrow 0\) 
\(epoch\le N\)  

Sample a batch of training samples: \(Z\, = \, \lbrace (x_{i},c_{i},y_{i})| i = 1,2,...,B\rbrace \) 
\(\mathcal {L}_{C}\leftarrow ~\Sigma ^{T}_{i}{l}(g_j(x_{i}),c^{j}_{i})\)  [note: \(l\)  - Binary Cross Entropy/Mean Square Error Loss]
\({\mathcal {L}_{Y}}\leftarrow ~l(f(g(x_{i})),y_{i})\)  [note: \(l\)  - Cross Entropy Loss]
\(s\leftarrow 0\) 
\(\tilde{x_{i}}\leftarrow x_{i}\) 
\(s\le ~num\) 

\(\tilde{x_i}~\leftarrow ~\tilde{x_i}+\epsilon *sign(\nabla _{\tilde{x_{i}}}\Sigma ^{B}_{i=1}l(g(\tilde{x_i}),c_{i}))\)  [note: \(l\)  - Binary Cross Entropy/Mean Square Error Loss]

\(\mathcal {L}_{adv}\leftarrow ~l(f(g(\tilde{x_{i}})),y_{i})\) 
\(\mathcal {L}_{f,g}~ = \Sigma ^B_{i}[\gamma *{\mathcal {L}_{Y}} + \lambda *{\mathcal {L}_{C}} + \alpha *\mathcal {L}_{adv}\) ]
epoch \(\le \)  N/2
\(\theta _{g} \leftarrow \theta _{g} - \omega *\nabla _{\theta _{g}}~\mathcal {L}_{f,g}\) 
\(\theta _{f,g} \leftarrow \theta _{f,g} - \omega ^{\prime }*\nabla _{\theta _{f,g}}~\mathcal {L}_{f,g}\)

Robust Concept Learning (RCL)
We detail the exact implementation of the proposed defense method Robust Concept Learning (RCL) in this subsection. As discussed before, the essential reason behing the lack of robustness of CBMs is the lack of rich concept information during training. To alleviate this problem, we propose adversarial augmentation to further improve the smoothness of the concept space. Algorithm  iteratively generates increasingly diverse images by optimizing the adversarial loss \(\mathcal {L}_{adv}\) . The final model optimization occurs with a weighted sum of all 3 - prediction, concept and adversarial loss. The values of \(\alpha \)  are set to 1 for C-MNIST, CUB and OAI datasets.

Additional Results
We report additional results on standard joint models and hybrid adversarially concept trained with higher concept robustness over all 3 datasets and attacks for varying \(L_{\infty }\)  values. As previously discussed, the \(L_{\infty }\)  values are considered to be a direct measure of visual imperceptibility, i.e the lower the value, the more imperceptible adversarial perturbations introduced in an image. As we postulated, any attack is much more effective and malicious if maximum attack success can be obtained with small \(L_{\infty }\)  values. We demonstrate the evaluation metrics for all 3 attacks in Tables REF , REF  and REF  for all 3 datasets. As evident from all 3 tables, the models trained using the proposed Robust Concept Learning with Adversarial Training algorithm are much more robust to attacks for small \(L_{\infty }\)  values, thus improving trust and reliability in CBMs. The values for budget - \(\epsilon _{thresh}\)  are chosen according to standard benchmarks for adversarial attacks e.g. https://github.com/MadryLab/mnist_challenge. As MNIST has a smaller latent representation the values for \(\epsilon _{thresh}\)  are relatively high while CUB is a very high dimensional dataset, the relative \(\epsilon _{thresh}\)  values are low.
<TABLE>
Additional visual results
We also present additional visual results for Erasure attacks on the ConceptMNIST (top-left), CUB (top-right) and OAI datasets (bottom-center) in Figure REF . The bars in the first row of each image represent the initial concept scores as predicted by the model. The bars in color red are randomly chosen concepts we attempt to and successfully flip in rows 2 and 3 of each image. (Note: we attempt to flip all present concepts, but demonstrate only 2 random successfully flips).
Visual results for Introduction attacks on the ConceptMNIST (top-left), CUB (top-right) and OAI datasets (bottom-center) are reported in Figure REF . The first image and the bars below it represent the initial image and the concept scores as predicted by the model, while the second image and bars below it represents the image and concept predictions after the attack. The bars in color red are previously non-relevant concepts which were successfully introduced after attack. The images in the first and last row are for ConceptMNIST and CUB datasets, respectively.
We release code for data processing, attacks and results on the ConceptMNIST dataset. Upon acceptance, we will also release codes for CUB dataset. As OAI dataset requires special permissions - only limited code will be released for it.
<TABLE><TABLE><FIGURE><FIGURE>