New ideas in distributed systems (algorithms or protocols) are commonly tested by simulation, because experimenting with a prototype deployed on a realistic platform is cumbersome. However, a prototype not only measures performance but also verifies assumptions about the underlying system. We developed dfuntest—a testing framework for distributed applications that defines abstractions and test structure, and automates experiments on distributed platforms. Dfuntest aims to be jUnit’s analogue for distributed applications; a framework that enables the programmer to write robust and flexible scenarios of experiments. Dfuntest requires minimal bindings that specify how to deploy and interact with the application. Dfuntest’s abstractions allow execution of a scenario on a single machine, a cluster, a cloud, or any other distributed infrastructure, e.g. on PlanetLab. A scenario is a procedure; thus, our framework can be used both for functional tests and for performance measurements. We show how to use dfuntest to deploy our DHT prototype on 60 PlanetLab nodes and verify whether the prototype maintains a correct topology.
Dfuntest uses a centralized control for the testing process (also called a global tester @cite_0 ). It is easier to verify global properties of the application once the whole state is represented in a single process. A scenario with distributed control can be centralized by exposing the requested behavior in an external interface of the tested application. In contrast, decentralization of a centralized test is more difficult. However, a centralized control is less scalable, and thus it might prevent the framework from effective testing of larger deployments. In our experiments, we found out that it is not the case for mid-size deployments, as dfuntest managed to instrument 60 hosts on PlanetLab network (we were limited by hosts' availability, rather than dfuntest performance). We refer to @cite_0 @cite_6 @cite_7 for further discussion.
@cite_0 @cite_7 @cite_6
1513384635 1978429203 2026184507
Stabilizing network infrastructures has moved the focus of software system engineering to the development of distributed applications running on top of the network. The complexity of distributed systems and their inherent concurrency pose high requirements on their design and implementation. This is also true for the validation of the systems, in particular the test. Compared to protocol testing the test of distributed systems and applications requires different methods for deriving test cases and for running the test. In this paper, we discuss architectures for testing distributed, concurrent systems. We suggest three different models: a global tester that has total control over the distributed system under test (SUT) and, more interestingly, two types of distributed testers comprising several concurrent tester components. The test architectures rely on a grey-box testing approach that allows to observe internal interactions of the SUT by the tester. In order to assure a correct assessment of the test data collected by the distributed tester components, the tester has to maintain a correct global view on the SUT. This can be achieved either by the use of redundant points of control and observation or by test coordination procedures. We outline the features of each approach and discuss their benefits and shortages. Finally, we describe possible simplifications for the architectures. The problem of deciding whether an observed behavior is acceptable is the oracle problem. When testing from a finite state machine (FSM), it is easy to solve the oracle problem and so it has received relatively little attention for FSMs. However, if the system under test has physically distributed interfaces, called ports, then in distributed testing, we observe a local trace at each port and we compare the set of local traces with the set of allowed behaviors (global traces). This paper investigates the oracle problem for deterministic and nondeterministic FSMs and for two alternative definitions of conformance for distributed testing. We show that the oracle problem can be solved in polynomial time for the weaker notion of conformance (⊆w) but is NP-hard for the stronger notion of conformance (⊆), even if the FSM is deterministic. However, when testing from a deterministic FSM with controllable input sequences, the oracle problem can be solved in polynomial time and similar results hold for nondeterministic FSMs. Thus, in some cases, the oracle problem can be efficiently solved when using ⊆s and where this is not the case, we can use the decision procedure for ⊆w as a sound approximation. There has been much interest in testing from finite-state machines (FSMs). If the system under test can be modelled by the (minimal) FSM N then testing from an (minimal) FSM M is testing to check that N is isomorphic to M. In the distributed test architecture, there are multiple interfaces ports and there is a tester at each port. This can introduce controllability synchronization and observability problems. This paper shows that the restriction to test sequences that do not cause controllability problems and the inability to observe the global behaviour in the distributed test architecture, and thus relying only on the local behaviour at remote testers, introduces fundamental limitations into testing. There exist minimal FSMs that are not equivalent, and so are not isomorphic, and yet cannot be distinguished by testing in this architecture without introducing controllability problems. Similarly, an FSM may have non-equivalent states that cannot be distinguished in the distributed test architecture without causing controllability problems: these are said to be locally s-equivalent and otherwise they are locally s-distinguishable. This paper introduces the notion of two states or FSMs being locally s-equivalent and formalizes the power of testing in the distributed test architecture in terms of local s-equivalence. It introduces a polynomial time algorithm that, given an FSM M, determines which states of M are locally s-equivalent and produces minimal length input sequences that locally s-distinguish states that are not locally s-equivalent. An FSM is locally s-minimal if it has no pair of locally s-equivalent states. This paper gives an algorithm that takes an FSM M and returns a locally s-minimal FSM M′ that is locally s-equivalent to M.