As the size and source of network traffic increase, so does the challenge of monitoring and analyzing network traffic. Therefore, sampling algorithms are often used to alleviate these scalability issues. However, the use of high entropy data streams, through the use of either encryption or compression, further compounds the challenge as current state-of-the-art algorithms cannot accurately and efficiently differentiate between encrypted and compressed packets. In this paper, we propose a novel traffic classification method named High Entropy DistinGuishEr (HEDGE) to distinguish between compressed and encrypted traffic. HEDGE is based on the evaluation of the randomness of the data streams and can be applied to individual packets without the need to have access to the entire stream. The findings from the evaluation show that our approach outperforms current state of the art. We also make available our statistically sound dataset, based on known benchmarks, to the wider research community.
Encrypted data is known to be more uniformly distributed than unencrypted data @cite_20 @cite_53 . Therefore, a number of approaches rely on this characteristic to locate cryptographic keys stored in memory and file system dumps @cite_42 . The authors generally analysed big data streams, divided them into small blocks, and computed their entropy. Therefore, high entropy blocks may indicate the presence of encrypted data. Similarly, the authors in @cite_27 and @cite_1 first computed the entropy of packet payloads and then compared it with the entropy of uniformly randomly distributed sequences of the same length. However, the entropy estimation approach is not effective when the number of samples is small @cite_44 @cite_5 . Moreover, entropy measures are not reliable when other data which has high entropy is present such as compressed, MP3 or PDF files.
@cite_53 @cite_42 @cite_1 @cite_44 @cite_27 @cite_5 @cite_20
2109394932 2320415261 2114552878 2114771311 2000163531 56231138
THE problems of cryptography and secrecy systems furnish an interesting application of communication theory.1 In this paper a theory of secrecy systems is developed. The approach is on a theoretical level and is intended to complement the treatment found in standard works on cryptography.2 There, a detailed study is made of the many standard types of codes and ciphers, and of the ways of breaking them. We will be more concerned with the general mathematical structure and properties of secrecy systems. Targeted cancer therapies have shown promising results in patients, but few of these drugs provide long-term benefits because tumor cells rapidly develop drug resistance. . show that glioblastoma cells can become resistant to erlotinib, an epidermal growth factor receptor (EGFR)–targeted drug, by eliminating extrachromosomal copies of the mutant EGFR gene. After a period of drug withdrawal, the mutant EGFR gene reappears on extrachromosomal DNA, and the tumor cells become resensitized. The discovery that cancer cells can evade drug therapy by this “hide-and-seek” mechanism may help to optimize the dosing schedule of erlotinib in glioblastoma patients. D. A. Nathanson, B. Gini, J. Mottahedeh, K. Visnyei, T. Koga, G. Gomez, A. Eskin, K. Hwang, J. Wang, K. Masui, A. Paucar, H. Yang, M. Ohashi, S. Zhu, J. Wykosky, R. Reed, S. F. Nelson, T. F. Cloughesy, C. D. James, P. N. Rao, H. I. Kornblum, J. R. Heath, W. K. Cavenee, F. B. Furnari, P. S. Mischel, Targeted therapy resistance mediated by dynamic regulation of extrachromosomal mutant EGFR DNA. Science 343 , 72–76 (2014). [Abstract][Full Text] We propose a novel approach for real-time privacy preserving traffic filtering based on entropy estimation. The decision of the real-time classifier is based on the entropy of the payload from first packet of a flow. The aim of the classifier is to detect traffic with encrypted payload. As a proof of concept we show the applicability of our approach as a traffic filter for a Skype detection engine. Traces collected in laboratory and real-world environments show that the traffic is reduced by a reasonable amount while achieving similar or even improved detection quality. We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This "inconsistency" theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual 1 N formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if "bias-corrected" estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods.Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data. Consider a sequence p sub N of discrete probability measures, supported on m sub N points, and assume that we observe N independent and identically distributed (i.i.d.) samples from each p sub N . We demonstrate the existence of an estimator of the entropy, H(p sub N ), which is consistent even if the ratio N m sub N is bounded (and, as a corollary, even if this ratio tends to zero, albeit at a sufficiently slow rate). Leakage of confidential information from an organization's networks has become a big threat to its information security. Egress monitoring and filtering have thus become popular for detecting such security breaches. Egress monitoring tools scan outgoing packets for keywords or their combinations present in the confidential documents. These content filtering techniques however fail when the data is encrypted. The solution proposed in this thesis is simple yet an effective approach to prevent information leakage when the data is encrypted. We assume that a policy is in place which disallows encrypted content from specific hosts, ports and applications and wish to detect any violations to this policy. This work aims at analyzing encrypted and unencrypted traffic flows across a gateway and detecting unauthorized encrypted traffic flows. The work discusses a low level approach to detect encryption, based on entropy calculation and packet analysis. The technique is based on the fact that encrypted data consists of a random distribution of symbols whose entropy is expected to be quite high as compared to an unencrypted file. Techniques to differentiate between encrypted and high entropy compressed traffic are also discussed. This thesis implements and compares statistical methods for a fast online detection of encrypted traffic from all the other unencrypted traffic flowing across a network.