On the amplification of security and privacy risks by post-hoc explanations in machine learning models
A variety of explanation methods have been proposed in recent years to help
users gain insights into the results returned by neural networks, which are
otherwise complex and opaque black-boxes. However, explanations give rise to
potential side-channels that can be leveraged by an adversary for mounting
attacks on the system. In particular, post-hoc explanation methods that
highlight input dimensions according to their importance or relevance to the
result also leak information that weakens security and privacy. In this work,
we perform the first systematic characterization of the privacy and security
risks arising from various popular explanation techniques. First, we propose
novel explanation-guided black-box evasion attacks that lead to 10 times
reduction in query count for the same success rate. We show that the
adversarial advantage from explanations can be quantified as a reduction in the
total variance of the estimated gradient. Second, we revisit the membership
information leaked by common explanations. Contrary to observations in prior
studies, via our modified attacks we show significant leakage of membership
information (above 100% improvement over prior results), even in a much
stricter black-box setting. Finally, we study explanation-guided model
extraction attacks and demonstrate adversarial gains through a large reduction
in query count.