To properly validate wireless networking solutions we depend on experimentation. Simulation very often produces less accurate results due to the use of models that are simplifications of the real phenomena they try to model. Networking experimentation may offer limited repeatability and reproducibility. Being influenced by external random phenomena such as noise, interference, and multipath, real experiments are hardly repeatable. In addition, they are difficult to reproduce due to testbed operational constraints and availability. Without repeatability and reproducibility, the validation of the networking solution under evaluation is questionable. In this paper, we show how the Trace-based Simulation (TS) approach can be used to accurately repeat and reproduce real experiments and, consequently, introduce a paradigm shift when it comes to the evaluation of wireless networking solutions. We present an extensive evaluation of the TS approach using the Fed4FIRE+ w-iLab.2 testbed. The results show that it is possible to repeat and reproduce real experiments using ns-3 trace-based simulations with more accuracy than in pure simulation, with average accuracy gains above 50 .
In wireless networking R&D we typically depend on experimentation to further evaluate a solution, as simulation is inherently a simplification of the real-world. However, experimentation is limited in aspects where simulation excels, such as repeatability and reproducibility. Real wireless experiments are hardly repeatable. Given the same input they can produce very different output results, since wireless communications are influenced by external random phenomena such as noise, interference, and multipath. Real experiments are also difficult to reproduce due to testbed operational constraints and availability. We have previously proposed the Trace-based Simulation (TS) approach, which uses the TraceBasedPropagationLossModel to successfully reproduce past experiments. Yet, in its current version, the TraceBasedPropagationLossModel only supports point-to-point scenarios. In this paper, we introduce a new version of the model that supports Multiple Access wireless scenarios. To validate the new version of the model, the network throughput was measured in a laboratory testbed. The experimental results were then compared to the network throughput achieved using the ns-3 trace-based simulation and a pure ns-3 simulation, confirming the TS approach is valid for multiple access scenarios too. A common problem in mobile networking research and development is the cost related to deploying and running real-world mobile testbeds. Due to cost and operational constraints, these testbeds usually run for short time periods but generate very unique and relevant results that are hard to reproduce. We propose the use of ns-3 as a solution to successfully reproduce real-world mobile testbed experiments. This is accomplished by feeding ns-3 with real testbed traces including node positions and radio link quality only. In order to validate our approach, the network throughput between a fixed Base Station and a Unmanned Aerial Vehicle (UAV) was measured in a real-world testbed. The experimental results were compared to the network throughput achieved using the ns-3 trace-based simulation and a plain ns-3 simulation. The obtained results show the high accuracy of the trace-based simulation, thus validating our approach.