Online video-on-demand(VoD) services invariably maintain a view count for each video they serve, and it has become an important currency for various stakeholders, from viewers, to content owners, advertizers, and the online service providers themselves. There is often significant financial incentive to use a robot (or a botnet) to artificially create fake views. How can we detect the fake views? Can we detect them (and stop them) using online algorithms as they occur? What is the extent of fake views with current VoD service providers? These are the questions we study in the paper. We develop some algorithms and show that they are quite effective for this problem.
There appears to be similar issues whenever one wants to attract eyeballs on the Internet. @cite_17 detects fake accounts on online social networks (OSN). It ranks users according to their perceived likelihood of being fake based on social graph properties. By deploying their method on the largest OSN of Spain, @math 90 The use of entropy function has been proposed for anomaly detection problems before. In @cite_20 , the authors propose to use several information-theoretic measures for network anomaly detection. The entropy measures are applied to Unix system call data, BSM data, and network tcpdump data to illustrate the utilities. @cite_14 uses two-phase entropy measures to detect network anomalies by comparing the current network traffic against a baseline distribution. A Maximum Entropy principle is applied to estimate the distribution for normal network operation using pre-labeled training data. Then the relative entropy of the network traffic is applied with respect to the distribution of the normal behavior. @cite_15 proposes efficient streaming algorithms to implement the entropy measurement on high-speed links with low CPU and memory requirements.
@cite_15 @cite_14 @cite_20 @cite_17
2109885200 1959274303 2164210932 2187472121
Using entropy of traffic distributions has been shown to aid a wide variety of network monitoring applications such as anomaly detection, clustering to reveal interesting patterns, and traffic classification. However, realizing this potential benefit in practice requires accurate algorithms that can operate on high-speed links, with low CPU and memory requirements. In this paper, we investigate the problem of estimating the entropy in a streaming computation model. We give lower bounds for this problem, showing that neither approximation nor randomization alone will let us compute the entropy efficiently. We present two algorithms for randomly approximating the entropy in a time and space efficient manner, applicable for use on very high speed (greater than OC-48) links. The first algorithm for entropy estimation is inspired by the structural similarity with the seminal work of for estimating frequency moments, and we provide strong theoretical guarantees on the error and resource usage. Our second algorithm utilizes the observation that the performance of the streaming algorithm can be enhanced by separating the high-frequency items (or elephants) from the low-frequency items (or mice). We evaluate our algorithms on traffic traces from different deployment scenarios. In this paper, we present results of application of Tsallis entropy in detection of denial of service attacks. Two detectors, one based on Tsallis and the other one based on Shannon's entropy, have been applied in several attack simulations, and their properties have been compared. The simulated attack is Synchronize packet SYN flood. A simple packet distribution, that is, entropy of source addresses are considered. In both cases, cumulative sum control chart algorithm is used for change point detection. Properties of two detectors that are compared are detection delay and rate of true and false positives. The results show that Tsallis entropy-based detector can outperform with respect to false positive rate Shannon-based one but that requires careful tuning of Tsallis Q parameter that depends on characteristics of network traffic. The detection delay of two detectors is approximately the same. Copyright © 2015 John Wiley & Sons, Ltd. The increasing practicality of large-scale flow capture makes it possible to conceive of traffic analysis methods that detect and identify a large and diverse set of anomalies. However the challenge of effectively analyzing this massive data source for anomaly diagnosis is as yet unmet. We argue that the distributions of packet features (IP addresses and ports) observed in flow traces reveals both the presence and the structure of a wide range of anomalies. Using entropy as a summarization tool, we show that the analysis of feature distributions leads to significant advances on two fronts: (1) it enables highly sensitive detection of a wide range of anomalies, augmenting detections by volume-based methods, and (2) it enables automatic classification of anomalies via unsupervised learning. We show that using feature distributions, anomalies naturally fall into distinct and meaningful clusters. These clusters can be used to automatically classify anomalies and to uncover new anomaly types. We validate our claims on data from two backbone networks (Abilene and Geant) and conclude that feature distributions show promise as a key element of a fairly general network anomaly diagnosis framework. This thesis analyses the adequacy and the performance of the Apache Spark Streaming engine in the context of anomaly detection in water distribution networks (WDN). It builds on an already proposed scenario in which sensors are extensively deployed in a WDN allowing for distributed, near-real-time anomaly detection. For this purpose, it uses several variations of LISA statistics and according statistical tests. In order to show that Spark Streaming is applicable to such a setting, algorithms computing these statistics in Spark Streaming were developed. Subsequently, the resulting prototype was tested in a simulated WDN setting. Finally, the performance of the different algorithms as well as the impact of several network characteristics were measured. The results show that the calculation of LISA statistics can be achieved with reasonable performance by using Spark Streaming. Furthermore, they reveal certain characteristics and limitations of Spark Streaming.