In this paper, we aim to develop a novel mechanism to preserve differential privacy (DP) in adversarial learning for deep neural networks, with provable robustness to adversarial examples. We leverage the sequential composition theory in DP, to establish a new connection between DP preservation and provable robustness. To address the trade-off among model utility, privacy loss, and robustness, we design an original, differentially private, adversarial objective function, based on the post-processing property in DP, to tighten the sensitivity of our model. An end-to-end theoretical analysis and thorough evaluations show that our mechanism notably improves the robustness of DP deep neural networks. The pervasiveness of machine learning exposes new vulnerabilities in software systems, in which deployed machine learning models can be used (a) to reveal sensitive information in private training data (Fredrikson et al., 2015) , and/or (b) to make the models misclassify, such as adversarial examples (Carlini & Wagner, 2017) . Efforts to prevent such attacks typically seek one of three solutions: (1) Models which preserve differential privacy (DP) (Dwork et al., 2006) , a rigorous formulation of privacy in probabilistic terms; (2) Adversarial training algorithms, which augment training data to consist of benign examples and adversarial examples crafted during the training process, thereby empirically increasing the classification accuracy given adversarial examples (Kardan & Stanley, 2017; Matyasko & Chau, 2017) ; and (3) Provable robustness, in which the model classification given adversarial examples is theoretically guaranteed to be consistent, i.e., a small perturbation in the input does not change the predicted label (Cisse et al., 2017; Kolter & Wong, 2017) . On the one hand, private models, trained with existing privacy-preserving mechanisms (Abadi et al., 2016; Shokri & Shmatikov, 2015; Phan et al., 2016; 2017b; a; Yu et al., 2019; Lee & Kifer, 2018) , are unshielded under adversarial examples. On the other hand, robust models, trained with adversarial learning algorithms (with or without provable robustness to adversarial examples), do not offer privacy protections to the training data. That one-sided approach poses serious risks to machine learning-based systems; since adversaries can attack a deployed model by using both privacy inference attacks and adversarial examples. To be safe, a model must be i) private to protect the training data, and ii) robust to adversarial examples. Unfortunately, there has not yet been research on how to develop such a model, which thus remains a largely open challenge. Simply combining existing DP-preserving mechanisms and provable robustness conditions (Cisse et al., 2017; Kolter & Wong, 2017; Raghunathan et al., 2018) cannot solve the problem, for many reasons. (a) Existing sensitivity bounds (Phan et al., 2016; 2017b; a) and designs (Yu et al., 2019; Lee & Kifer, 2018) have not been developed to protect the training data in adversarial training. It is obvious that using adversarial examples crafted from the private training data to train our models introduces a previously unknown privacy risk, disclosing the participation of the benign examples (Song et al., 2019) . (b) There is an unrevealed interplay among DP preservation, adversarial learning, and robustness bounds. (c) Existing algorithms cannot be readily applied to address the trade-off among model utility, privacy loss, and robustness. Therefore, theoretically bounding the robustness of a model (which both protects the privacy and is robust against adversarial examples) is nontrivial. In this paper, we established a connection among DP preservation to protect the training data, adversarial learning, and provable robustness. A sequential composition robustness theory was introduced to generalize robustness given any sequential and bounded function of independent defensive mechanisms. An original DP-preserving mechanism was designed to address the trade-off among model utility, privacy loss, and robustness by tightening the global sensitivity bounds. A new Monte Carlo Estimation was proposed to improve and stabilize the estimation of the robustness bounds; thus improving the certified accuracy under adversarial example attacks. However, there are several limitations. First, the accuracy of our model under adversarial example attacks is still very low. Second, the mechanism scalability is dependent on the model structures. Third, further study is needed to address the threats from adversarial examples crafted by unseen attack algorithms. Fourth, in this study, our goal is to illustrate the difficulties in providing DP protections to the training data in adversarial learning with robustness bounds. The problem is more challenging when working with complex and large networks, such as ResNet (He et al., 2015) , VGG16 (Zhang et al., 2015) , LSTM (Hochreiter & Schmidhuber, 1997) , and GAN (Goodfellow et al., 2014a) . Fifth, there can be alternative approaches to draft and to use DP adversarial examples. Addressing these limitations needs significant efforts from both research and practice communities. A NOTATIONS AND TERMINOLOGIES Function/model f that maps inputs x to a vector of scores f (x) = {f1(x), . . . , fK (x)} yx ∈ y A single true class label of example x y(x) = max k∈K f k (x) Predicted label for the example x given the function f x adv = x + α Adversarial example where α is the perturbation lp(µ) = {α ∈ R d : α p ≤ µ} The lp-norm ball of attack radius µ ( r , δr) Robustness budget r and broken probability δr The expected value of f k (x) E lb andÊ ub Lower and upper bounds of the expected valueÊf (x) = Feature representation learning model with x and parameters θ1 Bt A batch of benign examples xi Data reconstruction function given Bt in a(x, θ1) The values of all hidden neurons in the hidden layer h1 of a(x, θ1) given the batch Bt RB t (θ1) and R B t (θ1) Approximated and perturbed functions of RB t (θ1) xi and xi Perturbed and reconstructed inputs xi Sensitivity of the approximated function RB t (θ1) h1B Sensitivities of x and h, given the perturbation α ∈ lp(1) Privacy budget to protect the training data D (κ + ϕ)max Robustness size guarantee given an input x at the inference time B PSEUDO-CODE OF ADVERSARIAL TRAINING (KURAKIN ET AL., 2016B) Given a loss function: where m 1 and m 2 correspondingly are the numbers of examples in B t and B adv t at each training step. Proof 1 Assume that B t and B t differ in the last tuple, x m (x m ). Then, Proof 2 Regarding the computation of h 1Bt = {θ The sensitivity of a function h is defined as the maximum change in output, that can be generated by a change in the input (Lecuyer et al., 2018) . Therefore, the global sensitivity of h 1 can be computed as follows: following matrix norms (Operator norm, 2018): θ T 1 1,1 is the maximum 1-norm of θ 1 's columns. By injecting Laplace noise Lap( , and χ 2 drawn as a Laplace noise [Lap( β , in our mechanism, the perturbed affine transformation h 1Bt is presented as: This results in an ( 1 /γ)-DP affine transformation h 1Bt = {θ Similarly, the perturbed inputs where ∆ x is the sensitivity measuring the maximum change in the input layer that can be generated by a change in the batch B t and γ x = ∆ R m∆x . Following (Lecuyer et al., 2018) , ∆ x can be computed as follows: Consequently, Lemma 3 does hold.
Preserving Differential Privacy in Adversarial Learning with Provable Robustness to Adversarial Examples
{"src": "scitldr-AIC_train", "paper_id": "Byg-An4tPr"}