The complexity of large-scale neural networks can lead to poor understanding of their internal  details. We show that this opaqueness provides an opportunity for adversaries to embed unintended functionalities into the network in the form of Trojan horse attacks. Our novel framework hides the existence of a malicious network within a benign transport network. Our attack is flexible, easy to execute, and difficult to detect. We prove theoretically that the malicious network's detection is computationally infeasible and demonstrate empirically that the transport network does not compromise its disguise. Our attack exposes an important, previously unknown loophole that unveils a new direction in machine learning security. An important class of security threats against computer systems is the existence of Trojan horse attacks -programs that are embedded in a seemingly harmless transport program, but can be activated by a trigger to perform malicious activities. This threat is common in software, where the malicious program may steal user information or modify the underlying system's behavior (Felt et al., 2011) . Similar attacks have also been studied in depth for hardware circuits (Chakraborty et al., 2009) . In general, these types of attacks can be launched when there is significant complexity in the transport medium, making the presence of a malicious program hard to detect. Due to the complex architecture of modern neural networks, both the model and their behavior are arguably obscure to humans (Ribeiro et al., 2016; Selvaraju et al., 2017; Koh & Liang, 2017) . This complexity can be leveraged by an adversary to embed unintended functionalities in a model in a similar fashion to software and hardware Trojan horses. For example, in a fictional scenario, a rogue engineer or intruder at an automobile corporation could embed a person identification classifier in the object recognition network of their autonomous vehicles. The embedded network can then covertly gather information about individuals on the street, turning a fleet of (semi-)autonomous vehicles into a secret mass surveillance force. Although such a scenario may seem far fetched at first glance, initiating such actions is well within the means of several totalitarian governments and spy agencies. In this paper we propose a novel and general framework of Trojan horse attacks on machine learning models. Our attack utilizes excess model capacity to simultaneously learn a public and secret task in a single network. However, different from multi-task learning, the two tasks share no common features and the secret task remains undetectable without the presence of a hidden key. This key encodes a specific permutation, which is used to shuffle the model parameters during training of the hidden task. The gradient updates for the concealed model act similar to benign additive noise with respect to the gradients of the public model (Abadi et al., 2016) , which behaves indistinguishable to a standard classifier on the public task. We demonstrate empirically and prove theoretically that the identity and presence of a secret task cannot be detected without knowledge of the secret permutation. In particular, we prove that the decision problem to determine if the model admits a permutation that triggers a secret functionality is NP-complete. We experimentally validate our method on a standard ResNet50 network (He et al., 2016) and show that, without any increase in parameters, the model can achieve the same performance on the intended and on the secret tasks as if it was trained exclusively on only one of them. Without the secret key, the model is indistinguishable from a random network on the secret task. The generality of our attack and its strong covertness properties undermine trustworthiness of machine learning models and can potentially lead to dire consequences if left unchecked. We introduced TrojanNet, and formulate a potentially menacing attack scenario. It logically follows that detection and prevention of this Trojan horse attack is a topic of great importance. However, this may be a daunting task, as we show theoretically that the detection problem can be formulated as an NP-complete decision problem, and is therefore computationally infeasible in its general form. While strategies such as Markov Chain Monte Carlo have been used in similar contexts to efficiently reduce the search space (Diaconis, 2009) , the number of candidate permutations may be too large in our case. In fact, the number of permutations for a single convolutional layer of ResNet50 can be upwards of (64 × 64 × 3 × 3)! ≈ 1.21 × 10 152336 ! While our paper focuses on malicious uses of the TrojanNet framework, it can potentially be utilized for improving the security of neural networks as well. Our framework has striking resemblance to symmetric key encryption in cryptography (Katz & Lindell, 2014) . This enables the sharing of neural networks across an insecure, monitored communication channel in a similar fashion as steganography (Petitcolas et al., 1999) -the hiding of structured signals in files such as images, audio or text. We hope to explore benevolent uses of TrojanNet in future work. A APPENDIX
Parameters of a trained neural network can be permuted to produce a completely separate model for a different task, enabling the embedding of Trojan horse networks inside another network.
{"src": "scitldr-AIC_train", "paper_id": "BJeGA6VtPS"}