As the size and source of network traffic increase, so does the challenge of monitoring and analyzing network traffic. Therefore, sampling algorithms are often used to alleviate these scalability issues. However, the use of high entropy data streams, through the use of either encryption or compression, further compounds the challenge as current state-of-the-art algorithms cannot accurately and efficiently differentiate between encrypted and compressed packets. In this paper, we propose a novel traffic classification method named High Entropy DistinGuishEr (HEDGE) to distinguish between compressed and encrypted traffic. HEDGE is based on the evaluation of the randomness of the data streams and can be applied to individual packets without the need to have access to the entire stream. The findings from the evaluation show that our approach outperforms current state of the art. We also make available our statistically sound dataset, based on known benchmarks, to the wider research community.
Existing methods generally rely on continuous traffic information to enhance their accuracy by collecting information about packet transmission, the beginning and the end of a connection or of a file and etc. Therefore, real-time monitoring is inefficient using such schemes, since they require the analysis of huge volumes of data. However, these schemes can be useful in studying past-events or analysing only specific connections. Hence, to enable real-time monitoring, our aim is to analyse the payload of a random subset of packets and infer as much information as possible. Similar to our approach, the work of @cite_51 presents the first, to the best of our knowledge, technique to distinguish encrypted from compressed unencrypted network transmissions by analysing random packets. The authors applied three machine learning models, with the convolutional neural network (CNN) model achieving the best results. However, more efficient solutions are needed to perform practical real-time traffic classification even in the case of random data stream analysis, as the methods applied in @cite_51 (i.e. CNN and @math -NN) are computationally expensive and require proper training.
@cite_51
2801973044
Data encryption is the primary method of protecting the privacy of consumer device Internet communications from network observers. The ability to automatically detect unencrypted data in network traffic is therefore an essential tool for auditing Internet-connected devices. Existing methods identify network packets containing cleartext but cannot differentiate packets containing encrypted data from packets containing compressed unencrypted data, which can be easily recovered by reversing the compression algorithm. This makes it difficult for consumer protection advocates to identify devices that risk user privacy by sending sensitive data in a compressed unencrypted format. Here, we present the first technique to automatically distinguish encrypted from compressed unencrypted network transmissions on a per-packet basis. We apply three machine learning models and achieve a maximum 66.9 accuracy with a convolutional neural network trained on raw packet data. This result is a baseline for this previously unstudied machine learning problem, which we hope will motivate further attention and accuracy improvements. To facilitate continuing research on this topic, we have made our training and test datasets available to the public.