An extensive body of research deals with estimating the correlation and the Hurst parameter of Internet traffic traces. The significance of these statistics is due to their fundamental impact on network performance. The coverage of Internet traffic traces is, however, limited since acquiring such traces is challenging with respect to, e.g., confidentiality, logging speed, and storage capacity. In this work, we investigate how the correlation of Internet traffic can be reliably estimated from random traffic samples. These samples are observed either by passive monitoring within the network, or otherwise by active packet probes at end systems. We analyze random sampling processes with different inter-sample distributions and show how to obtain asymptotically unbiased estimates from these samples. We quantify the inherent limitations that are due to limited observations and explore the influence of various parameters, such as sampling intensity, network utilization, or Hurst parameter on the estimation accuracy. We design an active probing method which enables simple and lightweight traffic sampling without support from the network. We verify our approach in a controlled network environment and present comprehensive Internet measurements. We find that the correlation exhibits properties such as long range dependence as well as periodicities and that it differs significantly across Internet paths and observation times.
The available bandwidth (avail-bw) in a network path is of major importance in congestion control, streaming applications, quality-of-service verification, server selection, and overlay networks. We describe an end-to-end methodology, called self-loading periodic streams (SLoPS), for measuring avail-bw. The basic idea in SLoPS is that the one-way delays of a periodic packet stream show an increasing trend when the stream's rate is higher than the avail-bw. We have implemented SLoPS in a tool called pathload. The accuracy of the tool has been evaluated with both simulations and experiments over real-world Internet paths. Pathload is nonintrusive, meaning that it does not cause significant increases in the network utilization, delays, or losses. We used pathload to evaluate the variability ("dynamics") of the avail-bw in Internet paths. The avail-bw becomes significantly more variable in heavily utilized paths, as well as in paths with limited capacity (probably due to a lower degree of statistical multiplexing). We finally examine the relation between avail-bw and TCP throughput. A persistent TCP connection can be used to measure roughly the avail-bw in a path, but TCP saturates the path and increases significantly the path delays and jitter. The packet pair technique estimates the capacity of a path (bottleneck bandwidth) from the dispersion (spacing) experienced by two back-to-back packets. We demonstrate that the dispersion of packet pairs in loaded paths follows a multimodal distribution, and discuss the queueing effects that cause the multiple modes. We show that the path capacity is often not the global mode, and so it cannot be estimated using standard statistical procedures. The effect of the size of the probing packets is also investigated, showing that the conventional wisdom of using maximum sized packet pairs is not optimal. We then study the dispersion of long packet trains. Increasing the length of the packet train reduces the measurement variance, but the estimates converge to a value, referred to as the asymptotic dispersion rate (ADR), that is lower than the capacity. We derive the effect of the cross traffic in the dispersion of long packet trains, showing that the ADR is not the available bandwidth in the path, as was assumed in previous work. Putting all the pieces together, we present a capacity estimation methodology that has been implemented in a tool called pathrate. In this paper, we determine which non-random sampling of fixe d size gives the best linear predictor of the sum of a finite spatial p opulation. We employ different multiscale superpopulation models and use the minimum mean-squared error as our optimality criterion. In a multiscale superpopulation tree models, the leaves represent the units of the pop ulation, interior nodes represent partial sums of the population, and the root node represents the total sum of the population. We prove that the optimal sampling pattern varies dramatically with the correlation structure of the t ree nodes. While uniform sampling is optimal for trees with “positive correlation progressio n”, it provides the worst possible sampling with “negative correlation progression.” As an analysis tool, we introduce and study a class of independent innovations trees that are of interest in their own right. We derive a fast water -filling algorithm to determine the optimal sampling of the leaves to estimate the root of an independent innovations tree. Most existing available-bandwidth measurement techniques are justified using a constant-rate fluid cross-traffic model. To achieve a better understanding of the performance of current bandwidth measurement techniques in general traffic conditions, this paper presents a queueing-theoretic foundation of single-hop packet-train bandwidth estimation under bursty arrivals of discrete cross-traffic packets. We analyze the statistical mean of the packet-train output dispersion and its mathematical relationship to the input dispersion, which we call the probing-response curve. This analysis allows us to prove that the single-hop response curve in bursty cross-traffic deviates from that obtained under fluid cross traffic of the same average intensity and to demonstrate that this may lead to significant measurement bias in certain estimation techniques based on fluid models. We conclude the paper by showing, both analytically and experimentally, that the response-curve deviation vanishes as the packet-train length or probing packet size increases, where the vanishing rate is decided by the burstiness of cross-traffic.   Although packet-pair probing has been used as one of the primary mechanisms to measure bottleneck capacity, cross-traffic intensity, and available bandwidth of end-to-end Internet paths, there is still no conclusive answer as to what information about the path is contained in the output packet-pair dispersions and how it is encoded. In this paper, we address this issue by deriving closed-form expression of packet-pair dispersion in the context of a single-hop path and general bursty cross-traffic arrival. Under the assumptions of cross-traffic stationarity and ASTA sampling, we examine the statistical properties of the information encoded in inter-packet spacings and derive the asymptotic average of the output packet-pair dispersions as a closed-form function of the input dispersion. We show that this result is different from what was obtained in prior work using fluid cross-traffic models and that this discrepancy has a significant impact on the accuracy of packet-pair bandwidth estimation.