No Peek: A Survey of private distributed deep learning <s> Computational Goals <s> Deep Learning has recently become hugely popular in machine learning for its ability to solve end-to-end learning systems, in which the features and the classifiers are learned simultaneously, providing significant improvements in classification accuracy in the presence of highly-structured and large databases. Its success is due to a combination of recent algorithmic breakthroughs, increasingly powerful computers, and access to significant amounts of data. Researchers have also considered privacy implications of deep learning. Models are typically trained in a centralized manner with all the data being processed by the same training algorithm. If the data is a collection of users' private data, including habits, personal pictures, geographical positions, interests, and more, the centralized server will have access to sensitive information that could potentially be mishandled. To tackle this problem, collaborative deep learning models have recently been proposed where parties locally train their deep learning structures and only share a subset of the parameters in the attempt to keep their respective training sets private. Parameters can also be obfuscated via differential privacy (DP) to make information extraction even more challenging, as proposed by Shokri and Shmatikov at CCS'15. Unfortunately, we show that any privacy-preserving collaborative deep learning is susceptible to a powerful attack that we devise in this paper. In particular, we show that a distributed, federated, or decentralized deep learning approach is fundamentally broken and does not protect the training sets of honest participants. The attack we developed exploits the real-time nature of the learning process that allows the adversary to train a Generative Adversarial Network (GAN) that generates prototypical samples of the targeted training set that was meant to be private (the samples generated by the GAN are intended to come from the same distribution as the training data). Interestingly, we show that record-level differential privacy applied to the shared parameters of the model, as suggested in previous work, is ineffective (i.e., record-level DP is not designed to address our attack). <s> BIB001 </s> No Peek: A Survey of private distributed deep learning <s> Computational Goals <s> In domains such as health care and finance, shortage of labeled data and computational resources is a critical issue while developing machine learning algorithms. To address the issue of labeled data scarcity in training and deployment of neural network-based systems, we propose a new technique to train deep neural networks over several data sources. Our method allows for deep neural networks to be trained using data from multiple entities in a distributed fashion. We evaluate our algorithm on existing datasets and show that it obtains performance which is similar to a regular neural network trained on a single machine. We further extend it to incorporate semi-supervised learning when training with few labeled samples, and analyze any security concerns that may arise. Our algorithm paves the way for distributed training of deep neural networks in data sensitive applications when raw data may not be shared directly. <s> BIB002 </s> No Peek: A Survey of private distributed deep learning <s> Computational Goals <s> We propose an approach for solving Visual Teach and Repeat tasks for routes that consist of discrete directions along path networks using deep learning. Visual paths are specified by a single monocular image sequence and our approach does not query frames or image features during inference, but instead is composed of classifiers trained on each path. Our method is efficient for both storing or following paths and enables sharing of visual path specifications between parties without sharing visual data explicitly. We evaluate our approach in a simulated environment, and present qualitative results on real data captured with a smartphone. <s> BIB003
It is also quite important that any mechanism that aims to protect these details also preserves utility of the model above an acceptable level. These goals are to be ideally achieved at a low cost with regards to 1. Memory 2. Computational time 3. Communications bandwidth 4. Synchronization As shown in Fig 1. below smaller hospitals or tele-healthcare screening centers do not acquire an enormous number of diagnostic images and they could also be limited by diagnostic manpower. A distributed machine learning method for diagnosis in this setting should ideally not share any raw data (no peek) and at same time achieve high accuracy while using significantly lower resources. This helps smaller hospitals to effectively serve those in need while benefiting from decentralized entities. Table 1 : This is a survey of distributed deep learning methods with decreasing levels of leakage from distributed NN to splitNN. Hybrid approaches of these techniques and differential privacy, homomorphic encryption and MPC are also included. The citations for these 9 groups have been grouped separately with subtitles in the references section for convenience. In federated learning and large batch synchronous SGD, the architecture and parameters are shared between the client and server along with intermediate representations of the model that include the gradients, activations and weight updates which are shared during the learning process. Although the data is not explicitly shared in raw form in these two techniques, works like BIB001 have shown that raw data can be reconstructed up to an approximation by adversaries, especially given the fact that the architecture and parameters are not completely protected. SplitNN BIB002 has an added advantage in this context in that it does not share the architecture and weights of the model. The protection offered by splitNN lies in the compact representations found in deeper layers of neural networks and the difficulty of recovering the underlying data from these representations without knowing the model weights used to produce them. Such representations form after passing the data through numerous activations whose inverse in the case of ReLU are nonlinear and ill-defined (the inverse of a zero-valued ReLU can map to any negative number). Such representations have been shown to preserve information important for certain tasks (path following BIB003 resources required such as computations, communication bandwidth, memory and synchronization. We categorize the techniques across these dimensions as having low, medium and high requirements. As shown, splitNN requires the lowest resources on the client side. This is because the architecture is cut (arbitrary shape and not necessarily vertical) at a layer where the computations are only performed up to that cut on the client side. The rest of the computations happen on the server side. The experimental results in BIB002 , quantify these comparisons.