We thank Panos Parpas and Emtiyaz Khan for their feedback on an earlier draft of this paper. We thank the anonymous reviewers for their thorough reviews and constructive comments. Pranava Madhyastha kindly acknowledges the support of Amazon AWS Cloud Credits for Research Award, hardware grant from NVIDIA, Anne O'Neill and the Imperial Corporate Partnership Programme.
<FIGURE><FIGURE><FIGURE><FIGURE>Jaccard Distance Experiments
In Figure REF  and Figure REF  we plot the Jaccard distance plots with CNN models and LSTM models and note that ASWA consistently improves the stability of the interpretations.
<FIGURE><FIGURE><FIGURE><FIGURE>
Results with 100 seeds
We perform attention stability based experiments as mentioned in the paper, but now with 100 seeds, i.e, 100 models initialized with different seeds instead of 10. Figures REF , REF , and REF  show the entropy of attention based interpretations for different datasets. Experimenting with 100 seeds helps strengthen our claims about the instability of the model and the effectiveness of our proposed algorithms like ASWA and NASWA.

Hyperparameter Settings
For training purposes, we use the same model settings for the models as mentioned in the paper [1]} (or the Github implementationhttps://github.com/successar/AttentionExplanation), our port of the code is made available at: https://github.com/rishj97/ModelStability. Additional hyper-parameters for replication studies are:

Number of epochs: 20

Optimizer: Adam

Learning rate: 0.001

The exact seeds used for running the experiments can be found in our code repository.

Binary Classification with LSTM based Models
For LSTM based models, we notice (in Figure REF ) similar trends as to the CNN models in terms of the instability of the attention based interpretations.
