Deep Learning has recently become hugely popular in machine learning, providing significant improvements in classification accuracy in the presence of highly-structured and large databases. Researchers have also considered privacy implications of deep learning. Models are typically trained in a centralized manner with all the data being processed by the same training algorithm. If the data is a collection of users' private data, including habits, personal pictures, geographical positions, interests, and more, the centralized server will have access to sensitive information that could potentially be mishandled. To tackle this problem, collaborative deep learning models have recently been proposed where parties locally train their deep learning structures and only share a subset of the parameters in the attempt to keep their respective training sets private. Parameters can also be obfuscated via differential privacy (DP) to make information extraction even more challenging, as proposed by Shokri and Shmatikov at CCS'15. Unfortunately, we show that any privacy-preserving collaborative deep learning is susceptible to a powerful attack that we devise in this paper. In particular, we show that a distributed, federated, or decentralized deep learning approach is fundamentally broken and does not protect the training sets of honest participants. The attack we developed exploits the real-time nature of the learning process that allows the adversary to train a Generative Adversarial Network (GAN) that generates prototypical samples of the targeted training set that was meant to be private (the samples generated by the GAN are intended to come from the same distribution as the training data). Interestingly, we show that record-level DP applied to the shared parameters of the model, as suggested in previous work, is ineffective (i.e., record-level DP is not designed to address our attack).
Covert channels, however, can be used to defeat DP-protected databases as shown in the work of @cite_3 . In general, privacy cannot be guaranteed if auxiliary information (outside the DP model) is accessible to the adversary @cite_96 . At NDSS'16, it was shown by @cite_26 that DP at a certain granularity is not effective in real-life scenarios where data such as social data, mobile data, or medical records have strong correlations with each other. Note that it's a matter of setting DP granularity right and DP is not being violated at all.
@cite_26 @cite_96 @cite_3
2017509273 2043868537 2129457795 1589091684 2109784850 2074701325 2058970046 1533145153 2124071994 2901937773 2070085078 2751501713 1494212869 2802314446 2119149536 2761138375 1473189865 2963175488 2165223967 1570343904
Existing differential privacy (DP) studies mainly consider aggregation on data sets where each entry corresponds to a particular participant to be protected. In many situations, a user may pose a relational algebra query on a database with sensitive data, and desire differentially private aggregation on the result of the query. However, no existing work is able to release such aggregation when the query contains unrestricted join operations. This severely limits the applications of existing DP techniques because many data analysis tasks require unrestricted joins. One example is subgraph counting on a graph. Furthermore, existing methods for differentially private subgraph counting support only edge DP and are subject to very simple subgraphs. Until recent, whether any nontrivial graph statistics can be released with reasonable accuracy for arbitrary kind of input graphs under node DP was still an open problem. In this paper, we propose a novel differentially private mechanism that supports unrestricted joins, to release an approximation of a linear statistic of the result of some positive relational algebra calculation over a sensitive database. The error bound of the approximate answer is roughly proportional to the empirical sensitivity of the query --- a new notion that measures the maximum possible change to the query answer when a participant withdraws its data from the sensitive database. For subgraph counting, our mechanism provides a solution to achieve node DP, for any kind of subgraphs. Consider two parties Alice and Bob, who hold private inputs x and y, and wish to compute a function f(x, y) privately in the information theoretic sense; that is, each party should learn nothing beyond f(x, y). However, the communication channel available to them is noisy. This means that the channel can introduce errors in the transmission between the two parties. Moreover, the channel is adversarial in the sense that it knows the protocol that Alice and Bob are running, and maliciously introduces errors to disrupt the communication, subject to some bound on the total number of errors. A fundamental question in this setting is to design a protocol that remains private in the presence of large number of errors. If Alice and Bob are only interested in computing f(x, y) correctly, and not privately, then quite robust protocols are known that can tolerate a constant fraction of errors. However, none of these solutions is applicable in the setting of privacy, as they inherently leak information about the parties' inputs. This leads to the question whether we can simultaneously achieve privacy and error-resilience against a constant fraction of errors. We show that privacy and error-resilience are contradictory goals. In particular, we show that for every constant c > 0, there exists a function f which is privately computable in the error-less setting, but for which no private and correct protocol is resilient against a c-fraction of errors. The same impossibility holds also for sub-constant noise rate, e.g., when c is exponentially small (as a function of the input size). The general Gaussian multiple-access wiretap channel (GGMAC-WT) and the Gaussian two-way wiretap channel (GTW-WT) are considered. In the GGMAC-WT, multiple users communicate with an intended receiver in the presence of an eavesdropper who receives their signals through another GMAC. In the GTW-WT, two users communicate with each other over a common Gaussian channel, with an eavesdropper listening through a GMAC. A secrecy measure that is suitable for this multiterminal environment is defined, and achievable secrecy rate regions are found for both channels. For both cases, the power allocations maximizing the achievable secrecy sum rate are determined. It is seen that the optimum policy may prevent some terminals from transmission in order to preserve the secrecy of the system. Inspired by this construct, a new scheme cooperative jamming is proposed, where users who are prevented from transmitting according to the secrecy sum rate maximizing power allocation policy ldquojamrdquo the eavesdropper, thereby helping the remaining users. This scheme is shown to increase the achievable secrecy sum rate. Overall, our results show that in multiple-access scenarios, users can help each other to collectively achieve positive secrecy rates. In other words, cooperation among users can be invaluable for achieving secrecy for the system. Investigating network covert channels in smartphones has become increasingly important as smartphones have recently replaced the role of traditional computers. Smartphones are subject to traditional computer network covert channel techniques. Smartphones also introduce new sets of covert channel techniques as they add more capabilities and multiple network connections. This work presents a new network covert channel in smartphones. The research studies the ability to leak information from the smartphones applications by reaching the cellular voice stream, and it examines the ability to employ the cellular voice channel to be a potential medium of information leakage through carrying modulated speech-like data covertly. To validate the theory, an Android software audio modem has been developed and it was able to leak data successfully through the cellular voice channel stream by carrying modulated data with a throughput of 13 bps with 0.018 BER. Moreover, Android security policies are investigated and broken in order to implement a user-mode rootkit that opens the voice channels by stealthily answering an incoming voice call. Multiple scenarios are conducted to verify the effectiveness of the proposed covert channel. This study identifies a new potential smartphone covert channel, and discusses some security vulnerabilities in Android OS that allow the use of this channel demonstrating the need to set countermeasures against this kind of breach. The ubiquity of mobile devices has brought forth the concept of participatory sensing, whereby ordinary citizens can now contribute and share information from the urban environment. However, such applications introduce a key research challenge: preserving the privacy of the individuals contributing data. In this paper, we study two different privacy concepts, k-anonymity and l-diversity, and demonstrate how their privacy models can be applied to protect users' spatial and temporal privacy in the context of participatory sensing. The first part of the paper focuses on schemes implementing k-anonymity. We propose the use of microaggregation, a technique used for facilitating disclosure control in databases, as an alternate to tessellation, which is the current state-of-the-art for location privacy in participatory sensing applications. We conduct a comparative study of the two techniques and demonstrate that each has its advantage in certain mutually exclusive situations. We then propose the Hybrid Variable size Maximum Distance to Average Vector (Hybrid-VMDAV) algorithm, which combines the positive aspects of microaggregation and tessellation. The second part of the paper addresses the limitations of the k-anonymity privacy model. We employ the principle of l-diversity and propose an l-diverse version of VMDAV (LD-VMDAV) as an improvement. In particular, LD-VMDAV is robust in situations where an adversary may have gained partial knowledge about certain attributes of the victim. We evaluate the performances of our proposed techniques using real-world traces. Our results show that Hybrid-VMDAV improves the percentage of positive identifications made by an application server by up to 100 and decreases the amount of information loss by about 40 . We empirically show that LD-VMDAV always outperforms its k-anonymity counterpart. In particular, it improves the ability of the applications to accurately interpret the anonymized location and time included in user reports. Our studies also confirm that perturbing the true locations of the users with random Gaussian noise can provide an extra layer of protection, while causing little impact on the application performance. Privacy and information security in general are major concerns that impede enterprise adaptation of shared or public cloud computing. Specifically, the concern of virtual machine (VM) physical co-residency stems from the threat that hostile tenants can leverage various forms of side channels (such as cache covert channels) to exfiltrate sensitive information of victims on the same physical system. However, on virtualized x86 systems, covert channel attacks have not yet proven to be practical, and thus the threat is widely considered a "potential risk." In this paper, we present a novel covert channel attack that is capable of high-bandwidth and reliable data transmission in the cloud. We first study the application of existing cache channel techniques in a virtualized environment and uncover their major insufficiency and difficulties. We then overcome these obstacles by: 1) redesigning a pure timing-based data transmission scheme, and 2) exploiting the memory bus as a high-bandwidth covert channel medium. We further design and implement a robust communication protocol and demonstrate realistic covert channel attacks on various virtualized x86 systems. Our experimental results show that covert channels do pose serious threats to information security in the cloud. Finally, we discuss our insights on covert channel mitigation in virtualized environments. Access control models protect sensitive data from unauthorized disclosure via direct accesses, however, they fail to prevent indirect accesses. Indirect data disclosure via inference channels occurs when sensitive information can be inferred from non-sensitive data and metadata. Inference channels are often low-bandwidth and complex; nevertheless, detection and removal of inference channels is necessary to guarantee data security. This paper presents a survey of the current and emerging research in data inference control and emphasizes the importance of targeting this so often overlooked problem during database security design. Information security and privacy in general are major concerns that impede enterprise adaptation of shared or public cloud computing. Specifically, the concern of virtual machine (VM) physical co-residency stems from the threat that hostile tenants can leverage various forms of side channels (such as cache covert channels) to exfiltrate sensitive information of victims on the same physical system. However, on virtualized ×86 systems, covert channel attacks have not yet proven to be practical, and thus the threat is widely considered a "potential risk". In this paper, we present a novel covert channel attack that is capable of high-bandwidth and reliable data transmission in the cloud. We first study the application of existing cache channel techniques in a virtualized environment, and uncover their major insufficiency and difficulties. We then overcome these obstacles by (1) redesigning a pure timing-based data transmission scheme, and (2) exploiting the memory bus as a high-bandwidth covert channel medium. We further design and implement a robust communication protocol, and demonstrate realistic covert channel attacks on various virtualized ×86 systems. Our experiments show that covert channels do pose serious threats to information security in the cloud. Finally, we discuss our insights on covert channel mitigation in virtualized environments. Secure communications can be impeded by eavesdroppers in conventional relay systems. This paper proposes cooperative jamming strategies for two-hop relay networks where the eavesdropper can wiretap the relay channels in both hops. In these approaches, the normally inactive nodes in the relay network can be used as cooperative jamming sources to confuse the eavesdropper. Linear precoding schemes are investigated for two scenarios where single or multiple data streams are transmitted via a decode-and-forward (DF) relay, under the assumption that global channel state information (CSI) is available. For the case of single data stream transmission, we derive closed-form jamming beamformers and the corresponding optimal power allocation. Generalized singular value decomposition (GSVD)-based secure relaying schemes are proposed for the transmission of multiple data streams. The optimal power allocation is found for the GSVD relaying scheme via geometric programming. Based on this result, a GSVD-based cooperative jamming scheme is proposed that shows significant improvement in terms of secrecy rate compared to the approach without jamming. Furthermore, the case involving an eavesdropper with unknown CSI is also investigated in this paper. Simulation results show that the secrecy rate is dramatically increased when inactive nodes in the relay network participate in cooperative jamming. Information leakage of sensitive data has become one of the fast growing concerns among computer users. With adversaries turning to hardware for exploits, caches are frequently a target for timing channels since they present different timing profiles for cache miss and hit latencies. Such timing channels operate by having an adversary covertly communicate secrets to a spy simply through modulating resource timing without leaving any physical evidence. In this article, we demonstrate a new vulnerability exposed by cache coherence protocols where adversaries could manipulate the coherence states on certain cache blocks to alter cache access timing and communicate secrets illegitimately. Our threat model assumes the trojan and spy can either exploit explicitly shared read-only physical pages (e.g., shared library code), or use memory deduplication feature to implicitly force create shared physical pages. We demonstrate a template that adversaries may use to construct covert timing channels through manipulating combinations of coherence states and data placement in different caches. We investigate several classes of cache coherence protocols, and observe that both directory-based and snoopy protocols can be subject to covert timing channel attacks. We identify that the root cause of the vulnerability to be the existence of access latency difference for cache lines in read-only cache coherence states: Exlusive and Shared. For defense, we propose a slightly modified cache coherence scheme that will enable the last level cache to directly respond to read data requests in these read-only coherence states, and avoid any latency difference that could enable timing channels. We consider a state-dependent three-terminal full-duplex relay channel with the channel states noncausally available at only the source, that is, neither at the relay nor at the destination. This model has application to cooperation over certain wireless channels with asymmetric cognition capabilities and cognitive interference relay channels. We establish lower bounds on the channel capacity for both discrete memoryless (DM) and Gaussian cases. For the DM case, the coding scheme for the lower bound uses techniques of rate-splitting at the source, decode-and-forward (DF) relaying, and a Gel'fand-Pinsker-like binning scheme. In this coding scheme, the relay decodes only partially the information sent by the source. Due to the rate-splitting, this lower bound is better than the one obtained by assuming that the relay decodes all the information from the source, that is, full-DF. For the Gaussian case, we consider channel models in which each of the relay node and the destination node experiences on its link an additive Gaussian outside interference. We first focus on the case in which the links to the relay and to the destination are corrupted by the same interference; and then we focus on the case of independent interferences. We also discuss a model with correlated interferences. For each of the first two models, we establish a lower bound on the channel capacity. The coding schemes for the lower bounds use techniques of dirty paper coding or carbon copying onto dirty paper, interference reduction at the source and decode-and-forward relaying. The results reveal that, by opposition to carbon copying onto dirty paper and its root Costa's initial dirty paper coding (DPC), it may be beneficial in our setup that the informed source uses a part of its power to partially cancel the effect of the interference so that the uninformed relay benefits from this cancellation, and so the source benefits in turn. Private record linkage (PRL) is the problem of identifying pairs of records that are similar as per an input matching rule from databases held by two parties that do not trust one another. We identify three key desiderata that a PRL solution must ensure: (1) perfect precision and high recall of matching pairs, (2) a proof of end-to-end privacy, and (3) communication and computational costs that scale subquadratically in the number of input records. We show that all of the existing solutions for PRL? including secure 2-party computation (S2PC), and their variants that use non-private or differentially private (DP) blocking to ensure subquadratic cost -- violate at least one of the three desiderata. In particular, S2PC techniques guarantee end-to-end privacy but have either low recall or quadratic cost. In contrast, no end-to-end privacy guarantee has been formalized for solutions that achieve subquadratic cost. This is true even for solutions that compose DP and S2PC: DP does not permit the release of any exact information about the databases, while S2PC algorithms for PRL allow the release of matching records. In light of this deficiency, we propose a novel privacy model, called output constrained differential privacy, that shares the strong privacy protection of DP, but allows for the truthful release of the output of a certain function applied to the data. We apply this to PRL, and show that protocols satisfying this privacy model permit the disclosure of the true matching records, but their execution is insensitive to the presence or absence of a single non-matching record. We find that prior work that combine DP and S2PC techniques even fail to satisfy this end-to-end privacy model. Hence, we develop novel protocols that provably achieve this end-to-end privacy guarantee, together with the other two desiderata of PRL. Our empirical evaluation also shows that our protocols obtain high recall, scale near linearly in the size of the input databases and the output set of matching pairs, and have communication and computational costs that are at least 2 orders of magnitude smaller than S2PC baselines. Cloud computing relies on hypervisors to isolate virtual machines running on shared hardware. Since perfect isolation is difficult to achieve, sharing hardware induces threats. Covert channels were demonstrated to violate isolation and, typically, allow data exfiltration. Several covert channels have been proposed that rely on the processor's cache. However, these covert channels are either slow or impractical due to the addressing uncertainty. This uncertainty exists in particular in virtualized environments and with recent L3 caches which are using complex addressing. Using shared memory would elude addressing uncertainty, but shared memory is not available in most practical setups. We build C5, a covert channel that tackles addressing uncertainty without requiring any shared memory, making the covert channel fast and practical. We are able to transfer messages on modern hardware across any cores of the same processor. The covert channel targets the last level cache that is shared across all cores. It exploits the inclusive feature of caches, allowing a core to evict lines in the private first level cache of another core. We experimentally evaluate the covert channel in native and virtualized environments. In particular, we successfully establish a covert channel between virtual machines running on different cores. We measure a bitrate of 1291i¾?bps for a native setup, and 751i¾?bps for a virtualized setup. This is one order of magnitude above previous cache-based covert channels in the same setup. As machine learning (ML) applications become increasingly prevalent, protecting the confidentiality of ML models becomes paramount for two reasons: (a) models may constitute a business advantage to its owner, and (b) an adversary may use a stolen model to find transferable adversarial examples that can be used to evade classification by the original model. One way to protect model confidentiality is to limit access to the model only via well-defined prediction APIs. This is common not only in machine-learning-as-a-service (MLaaS) settings where the model is remote, but also in scenarios like autonomous driving where the model is local but direct access to it is protected, for example, by hardware security mechanisms. Nevertheless, prediction APIs still leak information so that it is possible to mount model extraction attacks by an adversary who repeatedly queries the model via the prediction API. In this paper, we describe a new model extraction attack by combining a novel approach for generating synthetic queries together with recent advances in training deep neural networks. This attack outperforms state-of-the-art model extraction techniques in terms of transferability of targeted adversarial examples generated using the extracted model (+15-30 percentage points, pp), and in prediction accuracy (+15-20 pp) on two datasets. We then propose the first generic approach to effectively detect model extraction attacks: PRADA. It analyzes how the distribution of consecutive queries to the model evolves over time and raises an alarm when there are abrupt deviations. We show that PRADA can detect all known model extraction attacks with a 100 success rate and no false positives. PRADA is particularly suited for detecting extraction attacks against local models. Today, the publication of microdata poses a privacy threat: anonymous personal records can be re-identified using third data sources. Past research has tried to develop a concept of privacy guarantee that an anonymized data set should satisfy before publication, culminating in the notion of t-closeness. To satisfy t-closeness, the records in a data set need to be grouped into Equivalence Classes (ECs), such that each EC contains records of indistinguishable quasi-identifier values, and its local distribution of sensitive attribute (SA) values conforms to the global table distribution of SA values. However, despite this progress, previous research has not offered an anonymization algorithm tailored for t-closeness. In this paper, we cover this gap with SABRE, a SA Bucketization and REdistribution framework for t-closeness. SABRE first greedily partitions a table into buckets of similar SA values and then redistributes the tuples of each bucket into dynamically determined ECs. This approach is facilitated by a property of the Earth Mover's Distance (EMD) that we employ as a measure of distribution closeness: If the tuples in an EC are picked proportionally to the sizes of the buckets they hail from, then the EMD of that EC is tightly upper-bounded using localized upper bounds derived for each bucket. We prove that if the t-closeness constraint is properly obeyed during partitioning, then it is obeyed by the derived ECs too. We develop two instantiations of SABRE and extend it to a streaming environment. Our extensive experimental evaluation demonstrates that SABRE achieves information quality superior to schemes that merely applied algorithms tailored for other models to t-closeness, and can be much faster as well. The large-scale monitoring of computer users' software activities has become commonplace, e.g., for application telemetry, error reporting, or demographic profiling. This paper describes a principled systems architecture---Encode, Shuffle, Analyze (ESA)---for performing such monitoring with high utility while also protecting user privacy. The ESA design, and its Prochlo implementation, are informed by our practical experiences with an existing, large deployment of privacy-preserving software monitoring. With ESA, the privacy of monitored users' data is guaranteed by its processing in a three-step pipeline. First, the data is encoded to control scope, granularity, and randomness. Second, the encoded data is collected in batches subject to a randomized threshold, and blindly shuffled, to break linkability and to ensure that individual data items get "lost in the crowd" of the batch. Third, the anonymous, shuffled data is analyzed by a specific analysis engine that further prevents statistical inference attacks on analysis results. ESA extends existing best-practice methods for sensitive-data analytics, by using cryptography and statistical techniques to make explicit how data is elided and reduced in precision, how only common-enough, anonymous data is analyzed, and how this is done for only specific, permitted purposes. As a result, ESA remains compatible with the established workflows of traditional database analysis. Strong privacy guarantees, including differential privacy, can be established at each processing step to defend against malice or compromise at one or more of those steps. Prochlo develops new techniques to harden those steps, including the Stash Shuffle, a novel scalable and efficient oblivious-shuffling algorithm based on Intel's SGX, and new applications of cryptographic secret sharing and blinding. We describe ESA and Prochlo, as well as experiments that validate their ability to balance utility and privacy. We initiate the study of privacy in pharmacogenetics, wherein machine learning models are used to guide medical treatments based on a patient's genotype and background. Performing an in-depth case study on privacy in personalized warfarin dosing, we show that suggested models carry privacy risks, in particular because attackers can perform what we call model inversion: an attacker, given the model and some demographic information about a patient, can predict the patient's genetic markers. As differential privacy (DP) is an oft-proposed solution for medical settings such as this, we evaluate its effectiveness for building private versions of pharmacogenetic models. We show that DP mechanisms prevent our model inversion attacks when the privacy budget is carefully selected. We go on to analyze the impact on utility by performing simulated clinical trials with DP dosing models. We find that for privacy budgets effective at preventing attacks, patients would be exposed to increased risk of stroke, bleeding events, and mortality. We conclude that current DP mechanisms do not simultaneously improve genomic privacy while retaining desirable clinical efficacy, highlighting the need for new mechanisms that should be evaluated in situ using the general methodology introduced by our work. We consider the problem of providing privacy, in the private information retrieval (PIR) sense, to users requesting data from a distributed storage system (DSS). The DSS uses an (n, k) Maximum Distance Separable (MDS) code to store the data reliably on unreliable storage nodes. Some of these nodes can be spies which report to a third party, such as an oppressive regime, which data is being requested by the user. An information theoretic PIR scheme ensures that a user can satisfy its request while revealing, to the spy nodes, no information on which data is being requested. A user can achieve PIR by downloading all the data in the DSS. However, this is not a feasible solution due to its high communication cost. We construct PIR schemes with low download communication cost. When there is b = 1 spy node in the DSS, we construct PIR schemes with download cost 1 1−R per unit of requested data (R = k n is the code rate), achieving the information theoretic limit for linear schemes. The proposed schemes are universal since they depend on the code rate, but not on the generator matrix of the code. When there are 2 ≤ b ≤ n − k spy nodes, we devise linear PIR schemes that have download cost equal to b + k per unit of requested data. In this paper, we study the problem of answering queries about private data that is spread across multiple different databases. For instance, a medical researcher may want to study a possible correlation between travel patterns and certain types of illnesses. The necessary information exists today - e.g., in airline reservation systems and hospital records - but it is maintained by two separate companies who are prevented by law from sharing this information with each other, or with a third party. This separation prevents the processing of such queries, even if the final answer, e.g., a correlation coefficient, would be safe to release. We present DJoin, a system that can process such distributed queries and can give strong differential privacy guarantees on the result. DJoin can support many SQL-style queries, including joins of databases maintained by different entities, as long as they can be expressed using DJoin's two novel primitives: BN-PSI-CA, a differentially private form of private set intersection cardinality, and DCR, a multi-party combination operator that can aggregate noised cardinalities without compounding the individual noise terms. Our experimental evaluation shows that DJoin can process realistic queries at practical timescales: simple queries on three databases with 15,000 rows each take between 1 and 7.5 hours. We formulate and study a privacy guarantee to data owners, who share information with clients by publishing views of a proprietary database. The owner identi.es the sensitive proprietary data using a secret query against the proprietary database. Given an extra view, the privacy guarantee ensures that potential attackers will not learn any information about the secret that could not already be obtained from the existing views. We de.ne “learning” as the modi.cation of the attacker's a-priori probability distribution on the set of possible secrets. We assume arbitrary a-priori distributions (including distributions that correlate the existence of particular tuples) and solve the problem when secret and views are expressed as unions of conjunctive queries with non-equalities, under integrity constraints. We consider guarantees (a) for given view extents (b) for given domain of the secret and (c) independent of the domain and extents.