To properly validate wireless networking solutions we depend on experimentation. Simulation very often produces less accurate results due to the use of models that are simplifications of the real phenomena they try to model. Networking experimentation may offer limited repeatability and reproducibility. Being influenced by external random phenomena such as noise, interference, and multipath, real experiments are hardly repeatable. In addition, they are difficult to reproduce due to testbed operational constraints and availability. Without repeatability and reproducibility, the validation of the networking solution under evaluation is questionable. In this paper, we show how the Trace-based Simulation (TS) approach can be used to accurately repeat and reproduce real experiments and, consequently, introduce a paradigm shift when it comes to the evaluation of wireless networking solutions. We present an extensive evaluation of the TS approach using the Fed4FIRE+ w-iLab.2 testbed. The results show that it is possible to repeat and reproduce real experiments using ns-3 trace-based simulations with more accuracy than in pure simulation, with average accuracy gains above 50 .
Ns-3 is a widely used as a the network simulator of choice by researchers. It contains many well tested and high quality models of network protocols. However, the application layer models of ns-3 are very simplistic, and do not capture all aspects of real life applications. As a result, there is often a huge gap between the results of real experiments and the corresponding simulations. This problem is particularly exacerbated for wireless simulations, where many networking phenomena like wireless channel contention crucially depend on the application traffic characteristics. One way to bridge the gap between experiments and simulations is to incorporate knowledge from network traces into simulations. To this end, our work builds a trace-based application layer simulator in ns-3. Given a network trace collected from a user, our TraceReplay application layer model automatically generates traffic that is faithful to the real application in the ns-3 simulator. TraceReplay infers and replays only application layer delays like user think times, lets the simulator control the lower layer phenomena. TraceReplay extracts application layer characteristics from a single trace, and replays this information across many users in simulation, by using suitable randomization. Our model is also generic enough to replay any application layer protocol. Validation of our simulation model shows that simulation results obtained using TraceReplay are significantly different from those using other models, and are closer to experimental observations. This paper proposes a new approach for making simulations realistic. This approach is based on the principle of "trace driven simulation", i.e. using the results of the actual traffic traces analysis in order to reproduce the same experimental conditions in simulation. The main principle of the approach proposed in this paper deals with making simulation traffic sources replay under certain conditions - the actual traffic traces grabbed on actual networks. This paper describes the implementation of this approach in the NS simulator, and evaluates it by comparing the characteristics of the traces obtained with our replay approach with original data traces. The parameters that are considered for making the comparison are the usual traffic parameters as throughput, packet rate, etc., but also everything that is related to traffic dynamics, i.e. the second order statistical moments as autocorrelation of traffic or long range dependence.