Robotic code needs to be verified to ensure its safety and functional correctness, especially when the robot is interacting with people. Testing the real code in simulation is a viable option. It reduces the costs of experiments and provides detail that is lost when using formal methods. However, generating tests that cover interesting scenarios, while executing most of the code, is a challenge amplified by the complexity of the interactions between the environment and the software. Model-based test generation methods can automate otherwise manual processes and facilitate reaching rare scenarios during testing. In this paper, we compare the use of Belief-Desire-Intention (BDI) agents as models for test generation, with more conventional, model-based test generation, that exploits automata and model checking techniques, and random test generation methods, in terms of practicality, performance, scalability, and exploration ( coverage'). Simulators and automated testbenches were implemented in Robot Operating System (ROS) and Gazebo, for testing the code of two robots, BERT2 in a cooperative manufacture (table assembly) task, and Tiago as a home care assistant. The results highlight the clear advantages of using BDI agents for test generation, compared to random and conventional automata-based approaches. BDI agents naturally emulate the agency present in Human-Robot Interaction (HRI). They are thus more expressive and scale well in HRI applications.
In many robotics applications, test generation has been needed only for stimulating dedicated and simple components (equivalent to unit testing), such as choosing from a set of inputs for a controller @cite_22 , or generating a timing sequence for activating individual controllers @cite_1 . For these applications, random data generation or sampling @cite_10 might suffice to explore the state space or data ranges @cite_22 , along with alternatives such as constraint solving or optimization techniques @cite_1 . When testing a full robot system, however, the orchestration of different timed sequences in parallel (e.g. for emulated hardware components), coupled with several tasks of data instantiation (e.g. for sensor readings), is more complex. Sophisticated model-based approaches, such as those presented in this paper, offer a practical and viable solution for complex test generation problems @cite_23 @cite_13 @cite_9 . A model-based approach can be used in a hierarchical manner in order to coordinate lower-level random data generation and optimization with more complex, higher-level test generation tasks. A two layered approach is proposed in @cite_7 @cite_19 .
@cite_22 @cite_7 @cite_10 @cite_9 @cite_1 @cite_19 @cite_23 @cite_13
2107709519 2301698066 2233446961 2092949391
We present a technique that improves random test generation by incorporating feedback obtained from executing test inputs as they are created. Our technique builds inputs incrementally by randomly selecting a method call to apply and finding arguments from among previously-constructed inputs. As soon as an input is built, it is executed and checked against a set of contracts and filters. The result of the execution determines whether the input is redundant, illegal, contract-violating, or useful for generating more inputs. The technique outputs a test suite consisting of unit tests for the classes under test. Passing tests can be used to ensure that code contracts are preserved across program changes; failing tests (that violate one or more contract) point to potential errors that should be corrected. Our experimental results indicate that feedback-directed random test generation can outperform systematic and undirected random test generation, in terms of coverage and error detection. On four small but nontrivial data structures (used previously in the literature), our technique achieves higher or equal block and predicate coverage than model checking (with and without abstraction) and undirected random generation. On 14 large, widely-used libraries (comprising 780KLOC), feedback-directed random test generation finds many previously-unknown errors, not found by either model checking or undirected random generation. Industries such as flexible manufacturing and home care will be transformed by the presence of robotic assistants. Assurance of safety and functional soundness for these robotic systems will require rigorous verification and validation. We propose testing in simulation using Coverage-Driven Verification (CDV) to guide the testing process in an automatic and systematic way. We use a two-tiered test generation approach, where abstract test sequences are computed first and then concretized (e.g., data and variables are instantiated), to reduce the complexity of the test generation problem. To demonstrate the effectiveness of our approach, we developed a testbench for robotic code, running in ROS-Gazebo, that implements an object handover as part of a human-robot interaction (HRI) task. Tests are generated to stimulate the robot’s code in a realistic manner, through stimulating the human, environment, sensors, and actuators in simulation. We compare the merits of unconstrained, constrained and model-based test generation in achieving thorough exploration of the code under test, and interesting combinations of human-robot interactions. Our results show that CDV combined with systematic test generation achieves a very high degree of automation in simulation-based verification of control code for robots in HRI. Enabling robots to accomplish sophisticated tasks requires enhancing their capability to plan at multiple levels of discrete and continuous abstractions. Toward this goal, the proposed approach couples the ability of sampling-based motion planning to handle the complexity arising from high-dimensional robotic systems, nonlinear dynamics, and collision avoidance with the ability of discrete planning to handle discrete specifications. The approach makes it possible to specify tasks via Linear Temporal Logic (LTL) and automatically computes collision-free and dynamically-feasible motions that enable the robot to carry out assigned tasks. While discrete planning guides sampling-based motion planning, the latter feeds back information to further refine the guide and advance the search. Sampling is also used in the discrete space to shorten the length of the discrete plans and to expand the search toward new discrete states. Experiments with high-dimensional dynamical robot models performing various LTL tasks show significant computational speedups over related work. This paper presents a case study of search-based test generation for embedded system software units developed using the Function Block Diagrams (FBDs), a graphical language in the IEC 61131-3 standard aimed at programmable logic controllers (PLCs). We consider 279 different components from the train control software developed by Bombardier Transportation, a major rail vehicle manufacturer. The software is compiled into C code with a particular structure. We use a modified hill climbing algorithm for generating test data to maximize MC DC coverage for assignments with logical expressions in the C code, while retaining the semantics of the original FBD implementation. An experimental evaluation for comparing the effectiveness (coverage rate) and the efficiency (required number of executions) of hill climbing algorithm with random testing is presented. The results show that random testing performs well for most units under test, while around 30 of the artifacts significantly benefit from the hill climbing algorithm. Structural properties of the units that affect the performance of hill climbing and random testing are also discussed.