This paper studies the structure of several real-world traces (including Facebook, High-Performance Computing, Machine Learning, and simulation generated traces) and presents a systematic approach to quantify and compare the structure of packet traces based on the entropy contained in the trace file. Insights into the structure of packet traces can lead to improved network algorithms that are optimized toward specific traffic patterns. We then present a methodology to quantify the temporal and non-temporal components of entropy contained in a packet trace, called the trace complexity, using randomization and compression. We show that trace complexity provides unique insights into the characteristics of various applications and argue that there is a need for traffic generation models that preserve the intrinsic structure of empirically measured application traces. We then propose a traffic generator model that is able to produce a synthetic trace that matches the complexity level of its corresponding real-world trace.
Information-theoretic approaches and compression methodologies have already been proven successful in capturing entropy in other domains such as email @cite_10 , or comment @cite_28 spam filtering, or estimating neural discharges @cite_11 . The study of traffic patterns and the design of models is an evergreen topic of high relevance in the networking literature, and examples where measurement studies spurred much research into traffic modeling dates back to the 1990s @cite_38 @cite_42 @cite_33 . Since then, a large number of methodologies have been developed @cite_20 @cite_12 @cite_32 , based on temporal statistics @cite_17 @cite_3 @cite_31 , spatial statistics @cite_43 @cite_36 , and physical @cite_13 and information-theoretic @cite_7 @cite_39 models. In contrast to prior work, we are primarily interested in the communication pattern itself, rather than in the volume or headers of the exchanged data. While our work builds upon many significant results developed over the last decades @cite_34 , we are not aware of any work which allows to systematically differentiate between temporal and non-temporal components of traffic traces.
@cite_13 @cite_38 @cite_33 @cite_7 @cite_28 @cite_36 @cite_42 @cite_32 @cite_17 @cite_3 @cite_39 @cite_43 @cite_31 @cite_34 @cite_10 @cite_12 @cite_20 @cite_11
2133985894 2152536325 2105818147 2074373872 2123449924 2138543759 2010488817 2158712333 2155669300 2622770015 2143022286 2146603609 2122962290 2104290684 2165773639 2146320425 2116393125
One role for workload generation is as a means for understanding how servers and networks respond to variation in load. This enables management and capacity planning based on current and projected usage. This paper applies a number of observations of Web server usage to create a realistic Web workload generation tool which mimics a set of real users accessing a server. The tool, called Surge (Scalable URL Reference Generator) generates references matching empirical measurements of 1) server file size distribution; 2) request size distribution; 3) relative file popularity; 4) embedded file references; 5) temporal locality of reference; and 6) idle periods of individual users. This paper reviews the essential elements required in the generation of a representative Web workload. It also addresses the technical challenges to satisfying this large set of simultaneous constraints on the properties of the reference stream, the solutions we adopted, and their associated accuracy. Finally, we present evidence that Surge exercises servers in a manner significantly different from other Web server benchmarks. As ATM high-speed, cell-relay networks will most likely first make their impact as backbones interconnecting enterprise networks consisting of Ethernet and other LANs, their proper design and control is crucial. Recent studies of high quality, high resolution traffic measurements in Bellcore Ethernets have revealed that this aggregate Ethernet traffic is self-similar ("fractal") in nature, quite different in "burstiness" features from traffic considered and studied up to now. This paper presents an analytical study of an ATM buffer driven with self-similar traffic. The probability of buffer occupancy is obtained. It is shown that this probability decreases with the buffer size not exponentially, as in traditionally Markovian traffic models, but algebraically. Demonstrates that Ethernet LAN traffic is statistically self-similar, that none of the commonly used traffic models is able to capture this fractal-like behavior, that such behavior has serious implications for the design, control, and analysis of high-speed, cell-based networks, and that aggregating streams of such traffic typically intensifies the self-similarity ("burstiness") instead of smoothing it. These conclusions are supported by a rigorous statistical analysis of hundreds of millions of high quality Ethernet traffic measurements collected between 1989 and 1992, coupled with a discussion of the underlying mathematical and statistical properties of self-similarity and their relationship with actual network behavior. The authors also present traffic models based on self-similar stochastic processes that provide simple, accurate, and realistic descriptions of traffic scenarios expected during B-ISDN deployment. > Network engineers and operators are faced with a number of challenges that arise in the context of network monitoring and measurement. These include: i) how much information is included in measurement traces and by how much can we compress those traces?, ii) how much information is captured by different monitoring paradigms and tools ranging from full packet header captures to flow-level captures (such as with NetFlow) to packet and byte counts (such as with SNMP)? and iii) how much joint information is included in traces collected at different points and can we take advantage of this joint information? In this paper we develop a network model and an information theoretic framework within which to address these questions. We use the model and the framework to first determine the benefits of compressing traces captured at a single monitoring point, and we outline approaches to achieve those benefits. We next consider the benefits of joint coding, or equivalently of joint compression of traces captured a different monitoring points. Finally, we examine the difference in information content when measurements are made at either the flow level or the packet byte count level. In all of these cases, the effect of temporal and spatial correlation on the answers to the above questions is examined. Both our model and its predictions are validated against measurements taken from a large operational network. In this work, we design a method for blog comment spam detection using the assumption that spam is any kind of uninformative content. To measure the "informativeness" of a set of blog comments, we construct a language and tokenization independent metric which we call content complexity, providing a normalized answer to the informal question "how much information does this text contain?" We leverage this metric to create a small set of features well-adjusted to comment spam detection by computing the content complexity over groupings of messages sharing the same author, the same sender IP, the same included links, etc. We evaluate our method against an exact set of tens of millions of comments collected over a four months period and containing a variety of websites, including blogs and news sites. The data was provided to us with an initial spam labeling from an industry competitive source. Nevertheless the initial spam labeling had unknown performance characteristics. To train a logistic regression on this dataset using our features, we derive a simple mislabeling tolerant logistic regression algorithm based on expectation-maximization, which we show generally outperforms the plain version in precision-recall space. By using a parsimonious hand-labeling strategy, we show that our method can operate at an arbitrary high precision level, and that it significantly dominates, both in terms of precision and recall, the original labeling, despite being trained on it alone. The content complexity metric, the use of a noise-tolerant logistic regression and the evaluation methodology are thus the three central contributions with this work. The notion of self-similarity has been shown to apply to wide-area and local-area network traffic. We show evidence that the subset of network traffic that is due to World Wide Web (WWW) transfers can show characteristics that are consistent with self-similarity, and we present a hypothesized explanation for that self-similarity. Using a set of traces of actual user executions of NCSA Mosaic, we examine the dependence structure of WWW traffic. First, we show evidence that WWW traffic exhibits behavior that is consistent with self-similar traffic models. Then we show that the self-similarity in such traffic can be explained based on the underlying distributions of WWW document sizes, the effects of caching and user preference in file transfer, the effect of user "think time", and the superimposition of many such transfers in a local-area network. To do this, we rely on empirically measured distributions both from client traces and from data independently collected at WWW servers. Modern disk drives read-ahead data and reorder incoming requests in a workload-dependent fashion. This improves their performance, but makes simple analytical models of them inadequate for performance prediction, capacity planning, workload balancing, and so on. To address this problem we have developed a new analytic model for disk drives that do readahead and request reordering. We did so by developing performance models of the disk drive components (queues, caches, and the disk mechanism) and a workload transformation technique for composing them. Our model includes the effects of workload-specific parameters such as request size and spatial locality. The result is capable of predicting the behavior of a variety of real-world devices to within 17 across a variety of workloads and disk drives. We present a detailed statistical analysis of a 2-hour long empirical sample of VBR video. The sample was obtained by applying a simple intraframe video compression code to an action movie. The main findings of our analysis are (1) the tail behavior of the marginal bandwidth distribution can be accurately described using “heavy-tailed” distributions (e.g., Pareto); (2) the autocorrelation of the VBR video sequence decays hyperbolically (equivalent to long-range dependence ) and can be modeled using self-similar processes. We combine our findings in a new (non-Markovian) source model for VBR video and present an algorithm for generating synthetic traffic. Trace-driven simulations show that statistical multiplexing results in significant bandwidth efficiency even when long-range dependence is present. Simulations of our source model show long-range dependence and heavy-tailed marginals to be important components which are not accounted for in currently used VBR video traffic models. We develop a new multiscale modeling framework for characterizing positive-valued data with long-range-dependent correlations (1 f noise). Using the Haar wavelet transform and a special multiplicative structure on the wavelet and scaling coefficients to ensure positive results, the model provides a rapid O(N) cascade algorithm for synthesizing N-point data sets. We study both the second-order and multifractal properties of the model, the latter after a tutorial overview of multifractal analysis. We derive a scheme for matching the model to real data observations and, to demonstrate its effectiveness, apply the model to network traffic synthesis. The flexibility and accuracy of the model and fitting procedure result in a close fit to the real data statistics (variance-time plots and moment scaling) and queuing behavior. Although for illustrative purposes we focus on applications in network traffic modeling, the multifractal wavelet model could be useful in a number of other areas involving positive data, including image processing, finance, and geophysics. In this paper, we propose an information theoretic framework within which to study the redundancy present in packet header traces. Packet level and flow level models are developed that capture both temporal and spatial correlation present in packet headers, which can be exploited for packet trace compression. Information theoretic bounds are established for lossless packet header compression. Dependencies between the potential compression ratio and network parameters, such as the average route length and average flow size are derived. Valuable insights are obtained to guide the design of efficient packet trace compression algorithms. Statistics for Spatial Data GEOSTATISTICAL DATA Geostatistics Spatial Prediction and Kriging Applications of Geostatistics Special Topics in Statistics for Spatial Data LATTICE DATA Spatial Models on Lattices Inference for Lattice Models SPATIAL PATTERNS Spatial Point Patterns Modeling Objects References Author Index Subject Index. Network, Web, and disk I O traffic are usually bursty and self-similar and therefore cannot be modeled adequately with Poisson arrivals. However, we wish to model these types of traffic and generate realistic traces, because of obvious applications for disk scheduling, network management, and Web server design. Previous models (like fractional Brownian motion and FARIMA, etc.) tried to capture the 'burstiness'. However, the proposed models either require too many parameters to fit and or require prohibitively large (quadratic) time to generate large traces. We propose a simple, parsimonious method, the b-model, which solves both problems: it requires just one parameter, and can easily generate large traces. In addition, it has many more attractive properties: (a) with our proposed estimation algorithm, it requires just a single pass over the actual trace to estimate b. For example, a one-day-long disk trace in milliseconds contains about 86 Mb data points and requires about 3 minutes for model fitting and 5 minutes for generation. (b) The resulting synthetic traces are very realistic: our experiments on real disk and Web traces show that our synthetic traces match the real ones very well in terms of queuing behavior. Compressibility of individual sequences by the class of generalized finite-state information-lossless encoders is investigated. These encoders can operate in a variable-rate mode as well as a fixed-rate one, and they allow for any finite-state scheme of variable-length-to-variable-length coding. For every individual infinite sequence x a quantity (x) is defined, called the compressibility of x , which is shown to be the asymptotically attainable lower bound on the compression ratio that can be achieved for x by any finite-state encoder. This is demonstrated by means of a constructive coding theorem and its converse that, apart from their asymptotic significance, also provide useful performance criteria for finite and practical data-compression tasks. The proposed concept of compressibility is also shown to play a role analogous to that of entropy in classical information theory where one deals with probabilistic ensembles of sequences rather than with individual sequences. While the definition of (x) allows a different machine for each different sequence to be compressed, the constructive coding theorem leads to a universal algorithm that is asymptotically optimal for all sequences. Spam filtering poses a special problem in text categorization, of which the defining characteristic is that filters face an active adversary, which constantly attempts to evade filtering. Since spam evolves continuously and most practical applications are based on online user feedback, the task calls for fast, incremental and robust learning algorithms. In this paper, we investigate a novel approach to spam filtering based on adaptive statistical data compression models. The nature of these models allows them to be employed as probabilistic text classifiers based on character-level or binary sequences. By modeling messages as sequences, tokenization and other error-prone preprocessing steps are omitted altogether, resulting in a method that is very robust. The models are also fast to construct and incrementally updateable. We evaluate the filtering performance of two different compression algorithms; dynamic Markov compression and prediction by partial matching. The results of our empirical evaluation indicate that compression models outperform currently established spam filters, as well as a number of methods proposed in previous studies. Various methods for estimating the self-similarity parameter and or the intensity of long-range dependence in a time series are available. Some are more reliable than others. To discover the ones that work best, we apply the different methods to simulated sequences of fractional Gaussian noise and fractional ARIMA (0, d, 0). We also provide here a theoretical justification for the method of residuals of regression. Measurements of LAN and WAN traffic show that network traffic exhibits variability on different scales. We examine a mechanism that gives rise to self-similar network traffic and discuss performance. The mechanism we study is the transfer of files or messages whose size is drawn from a heavy-tailed distribution. In a realistic client server network the degree to which file sizes are heavy-tailed can directly determine the degree of traffic self-similarity at the link level. This causal relationship is robust relative to changes in network resources, network topology, the influence of cross-traffic, and the distribution of interarrival times. Properties of the transport layer play an important role in preserving and modulating this relationship. The reliable transmission and flow control mechanisms of TCP serve to maintain the long-range dependency structure induced by heavy-tailed file size distributions. In contrast, if a non-flow-controlled and unreliable (UDP-based) transport protocol is used, the resulting traffic shows little self-similarity: although still bursty at short time scales, it has little long-range dependence. Performance implications of self-similarity are discussed as represented by various performance measures. Increased self-similarity as expected, results in degradation of performance. Queueing delay, in particular is discussed. Throughput-related measures such as packet loss and retransmission rate, however increase only gradually with increasing traffic self-similarity as long as reliable, flow-controlled transport protocol is used. Normalized Lempel-Ziv complexity, which measures the generation rate of new patterns along a digital sequence, is closely related to such important source properties as entropy and compression ratio, but, in contrast to these, it is a property of individual sequences. In this article, we propose to exploit this concept to estimate (or, at least, to bound from below) the entropy of neural discharges (spike trains). The main advantages of this method include fast convergence of the estimator (as supported by numerical simulation) and the fact that there is no need to know the probability law of the process generating the signal. Furthermore, we present numerical and experimental comparisons of the new method against the standard method based on word frequencies, providing evidence that this new approach is an alternative entropy estimator for binned spike trains.