To properly validate wireless networking solutions we depend on experimentation. Simulation very often produces less accurate results due to the use of models that are simplifications of the real phenomena they try to model. Networking experimentation may offer limited repeatability and reproducibility. Being influenced by external random phenomena such as noise, interference, and multipath, real experiments are hardly repeatable. In addition, they are difficult to reproduce due to testbed operational constraints and availability. Without repeatability and reproducibility, the validation of the networking solution under evaluation is questionable. In this paper, we show how the Trace-based Simulation (TS) approach can be used to accurately repeat and reproduce real experiments and, consequently, introduce a paradigm shift when it comes to the evaluation of wireless networking solutions. We present an extensive evaluation of the TS approach using the Fed4FIRE+ w-iLab.2 testbed. The results show that it is possible to repeat and reproduce real experiments using ns-3 trace-based simulations with more accuracy than in pure simulation, with average accuracy gains above 50 .
For simulation, two main approaches can be found: 1) , such as the one proposed in @cite_2 , where the authors capture traffic of real networks and try to reproduce the same experimental condition in simulation down to per packet resolution; 2) , as the one presented in @cite_3 , where the authors try to abstract all low level variables and reproduce the traffic delays and performance bottlenecks experienced in the real network at the Application layer. These approaches do not allow to keep improving the solution under evaluation.
@cite_3 @cite_2
2133107013 2804572498 2518482829 1986774378
This paper proposes a new approach for making simulations realistic. This approach is based on the principle of "trace driven simulation", i.e. using the results of the actual traffic traces analysis in order to reproduce the same experimental conditions in simulation. The main principle of the approach proposed in this paper deals with making simulation traffic sources replay under certain conditions - the actual traffic traces grabbed on actual networks. This paper describes the implementation of this approach in the NS simulator, and evaluates it by comparing the characteristics of the traces obtained with our replay approach with original data traces. The parameters that are considered for making the comparison are the usual traffic parameters as throughput, packet rate, etc., but also everything that is related to traffic dynamics, i.e. the second order statistical moments as autocorrelation of traffic or long range dependence. In wireless networking R&D we typically depend on experimentation to further evaluate a solution, as simulation is inherently a simplification of the real-world. However, experimentation is limited in aspects where simulation excels, such as repeatability and reproducibility. Real wireless experiments are hardly repeatable. Given the same input they can produce very different output results, since wireless communications are influenced by external random phenomena such as noise, interference, and multipath. Real experiments are also difficult to reproduce due to testbed operational constraints and availability. We have previously proposed the Trace-based Simulation (TS) approach, which uses the TraceBasedPropagationLossModel to successfully reproduce past experiments. Yet, in its current version, the TraceBasedPropagationLossModel only supports point-to-point scenarios. In this paper, we introduce a new version of the model that supports Multiple Access wireless scenarios. To validate the new version of the model, the network throughput was measured in a laboratory testbed. The experimental results were then compared to the network throughput achieved using the ns-3 trace-based simulation and a pure ns-3 simulation, confirming the TS approach is valid for multiple access scenarios too. Ns-3 is a widely used as a the network simulator of choice by researchers. It contains many well tested and high quality models of network protocols. However, the application layer models of ns-3 are very simplistic, and do not capture all aspects of real life applications. As a result, there is often a huge gap between the results of real experiments and the corresponding simulations. This problem is particularly exacerbated for wireless simulations, where many networking phenomena like wireless channel contention crucially depend on the application traffic characteristics. One way to bridge the gap between experiments and simulations is to incorporate knowledge from network traces into simulations. To this end, our work builds a trace-based application layer simulator in ns-3. Given a network trace collected from a user, our TraceReplay application layer model automatically generates traffic that is faithful to the real application in the ns-3 simulator. TraceReplay infers and replays only application layer delays like user think times, lets the simulator control the lower layer phenomena. TraceReplay extracts application layer characteristics from a single trace, and replays this information across many users in simulation, by using suitable randomization. Our model is also generic enough to replay any application layer protocol. Validation of our simulation model shows that simulation results obtained using TraceReplay are significantly different from those using other models, and are closer to experimental observations. We propose a novel methodology to generate realistic network flow traces to enable systematic evaluation of network monitoring systems in various traffic conditions. Our technique uses a graph-based approach to model the communication structure observed in real-world traces and to extract traffic templates. By combining extracted and user-defined traffic templates, realistic network flow traces that comprise normal traffic and customized conditions are generated in a scalable manner. A proof-of-concept implementation demonstrates the utility and simplicity of our method to produce a variety of evaluation scenarios. We show that the extraction of templates from real-world traffic leads to a manageable number of templates that still enable accurate re-creation of the original communication properties on the network flow level.