As the size and source of network traffic increase, so does the challenge of monitoring and analyzing network traffic. Therefore, sampling algorithms are often used to alleviate these scalability issues. However, the use of high entropy data streams, through the use of either encryption or compression, further compounds the challenge as current state-of-the-art algorithms cannot accurately and efficiently differentiate between encrypted and compressed packets. In this paper, we propose a novel traffic classification method named High Entropy DistinGuishEr (HEDGE) to distinguish between compressed and encrypted traffic. HEDGE is based on the evaluation of the randomness of the data streams and can be applied to individual packets without the need to have access to the entire stream. The findings from the evaluation show that our approach outperforms current state of the art. We also make available our statistically sound dataset, based on known benchmarks, to the wider research community.
Encrypted data is known to be more uniformly distributed than unencrypted data @cite_20 @cite_53 . Therefore, a number of approaches rely on this characteristic to locate cryptographic keys stored in memory and file system dumps @cite_42 . The authors generally analysed big data streams, divided them into small blocks, and computed their entropy. Therefore, high entropy blocks may indicate the presence of encrypted data. Similarly, the authors in @cite_27 and @cite_1 first computed the entropy of packet payloads and then compared it with the entropy of uniformly randomly distributed sequences of the same length. However, the entropy estimation approach is not effective when the number of samples is small @cite_44 @cite_5 . Moreover, entropy measures are not reliable when other data which has high entropy is present such as compressed, MP3 or PDF files.
@cite_53 @cite_42 @cite_1 @cite_44 @cite_27 @cite_5 @cite_20
56231138 2170255835 2365029783 2243636408
Leakage of confidential information from an organization's networks has become a big threat to its information security. Egress monitoring and filtering have thus become popular for detecting such security breaches. Egress monitoring tools scan outgoing packets for keywords or their combinations present in the confidential documents. These content filtering techniques however fail when the data is encrypted. The solution proposed in this thesis is simple yet an effective approach to prevent information leakage when the data is encrypted. We assume that a policy is in place which disallows encrypted content from specific hosts, ports and applications and wish to detect any violations to this policy. This work aims at analyzing encrypted and unencrypted traffic flows across a gateway and detecting unauthorized encrypted traffic flows. The work discusses a low level approach to detect encryption, based on entropy calculation and packet analysis. The technique is based on the fact that encrypted data consists of a random distribution of symbols whose entropy is expected to be quite high as compared to an unencrypted file. Techniques to differentiate between encrypted and high entropy compressed traffic are also discussed. This thesis implements and compares statistical methods for a fast online detection of encrypted traffic from all the other unencrypted traffic flowing across a network. This paper explores what kinds of information two parties must communicate in order to correct errors which occur in a shared secret string W. Any bits they communicate must leak a significant amount of information about W --- that is, from the adversary's point of view, the entropy of W will drop significantly. Nevertheless, we construct schemes with which Alice and Bob can prevent an adversary from learning any useful information about W. Specifically, if the entropy of W is sufficiently high, then there is no function f(W) which the adversary can learn from the error-correction information with significant probability.This leads to several new results: (a) the design of noise-tolerant "perfectly one-way" hash functions in the sense of [7], which in turn leads to obfuscation of proximity queries for high entropy secrets W; (b) private fuzzy extractors [11], which allow one to extract uniformly random bits from noisy and nonuniform data W, while also insuring that no sensitive information about W is leaked; and (c) noise tolerance and stateless key re-use in the Bounded Storage Model, resolving the main open problem of Ding [10].The heart of our constructions is the design of strong randomness extractors with the property that the source W can be recovered from the extracted randomness and any string W' which is close to W. Abstract With the proliferation of smart grids, traditional utilities are struggling to handle the increasing amount of metering data. Outsourcing the metering data to heterogeneous distributed systems has the potential to provide efficient data access and processing. In an untrusted heterogeneous distributed system environment, employing data encryption prior to outsourcing can be an effective way to preserve user privacy. However, how to efficiently query encrypted multidimensional metering data stored in an untrusted heterogeneous distributed system environment remains a research challenge. In this paper, we propose a high performance and privacy-preserving query (P2Q) scheme over encrypted multidimensional big metering data to address this challenge. In the proposed scheme, encrypted metering data are stored in the server of an untrusted heterogeneous distributed system environment. A Locality Sensitive Hashing (LSH) based similarity search approach is then used to realize the similarity query. To demonstrate utility of the proposed LSH-based search approach, we implement a prototype using MapReduce for the Hadoop distributed environment. More specifically, for a given query, the proxy server will return K top similar data object identifiers. An enhanced Ciphertext-Policy Attribute-based Encryption (CP-ABE) policy is then used to control access to the search results. Therefore, only the requester with an authorized query attribute can obtain the correct secret keys to retrieve the metering data. We then prove that the P2Q scheme achieves data confidentiality and preserves the data owner’s privacy in a semi-trusted cloud. In addition, our evaluations demonstrate that the P2Q scheme can significantly reduce response time and provide high search efficiency without compromising on search quality (i.e. suitable for multidimensional big data search in heterogeneous distributed system, such as cloud storage system). This note describes an information theory problem that arose from some analysis of quantum key distribution protocols. The problem seems very natural and is very easy to state but has not to our knowledge been addressed before in the information theory literature: suppose that we have a random bit string y of length n and we reveal k bits at random positions, preserving the order but without revealing the positions, how much information about y is revealed? We show that while the cardinality of the set of compatible y strings depends only on n and k, the amount of leakage does depend on the exact revealed x string. We observe that the maximal leakage, measured as decrease in the Shannon entropy of the space of possible bit strings, corresponds to the x string being all zeros or all ones and that the minimum leakage corresponds to the alternating x strings. We derive a formula for the maximum leakage minimal entropy in terms of n and k. We discuss the relevance of other measures of information, in particular min-entropy, in a cryptographic context. Finally, we describe a simulation tool to explore these results.