Context: Search-based software testing promises to provide users with the ability to generate high-quality test cases, and hence increase product quality, with a minimal increase in the time and effort required. One result that emerged out of a previous study to investigate the application of search-based software testing (SBST) in an industrial setting was the development of the Interactive Search-Based Software Testing (ISBST) system. ISBST allows users to interact with the underlying SBST system, guiding the search and assessing the results. An industrial evaluation indicated that the ISBST system could find test cases that are not created by testers employing manual techniques. The validity of the evaluation was threatened, however, by the low number of participants. Objective: This paper presents a follow-up study, to provide a more rigorous evaluation of the ISBST system. Method: To assess the ISBST system a two-way crossover controlled experiment was conducted with 58 students taking a Verification and Validation course. The NASA Task Load Index (NASA-TLX) is used to assess the workload experienced by the participants in the experiment. Results: The experimental results validated the hypothesis that the ISBST system generates test cases that are not found by the same participants employing manual testing techniques. A follow-up laboratory experiment also investigates the importance of interaction in obtaining the results. In addition to this main result, the subjective workload was assessed for each participant by means of the NASA-TLX tool. The evaluation showed that, while the ISBST system required more effort from the participants, they achieved the same performance. Conclusions: The paper provides evidence that the ISBST system develops test cases that are not found by manual techniques, and that interaction plays an important role in achieving that result.
A more serious problem is that the number of evaluations that a human can perform is limited, as boredom and fatigue will set in. This is even more of an issue at the lowest level of abstraction, where the human user is involved in evolving each candidate. Fatigue has already been identified as a major concern, and efforts to alleviate the problem have been proposed @cite_9 . Alternatives have, therefore, been proposed that make it easier for the human user to interact, by selecting candidate solutions they favor and dismissing those they do not @cite_13 ; or focusing on the search objectives or the fitness values more than on the candidate solutions themselves @cite_34 @cite_8 .
@cite_9 @cite_34 @cite_13 @cite_8
2099020927 2114395490 2040378535
We describe two approaches to reducing human fatigue in interactive evolutionary computation (IEC). A predictor function is used to estimate the human user's score, thus reducing the amount of effort required by the human user during the evolution process. The fuzzy system and four machine learning classifier algorithms are presented. Their performance in a real-world application, the IEC-based design of a micromachine resonating mass, is evaluated. The fuzzy system was composed of four simple rules, but was able to accurately predict the user's score 77 of the time on average. This is equivalent to a 51 reduction of human effort compared to using IEC without the predictor. The four machine learning approaches tested were k-nearest neighbors, decision tree, AdaBoosted decision tree, and support vector machines. These approaches achieved good accuracy on validation tests, but because of the great diversity in user scoring behavior, were unable to achieve equivalent results on the user test data. We propose Visualized IEC as an interactive evolutionary computation (IEC) with visualizing individuals in a multidimensional searching space in a 2D space. This visualization helps us envision the landscape of an n-D searching space; so that it is easier for us to join an EC search, by indicating the possible global optimum estimated in the 2D mapped space. We experimentally evaluate the effect of visualization using a benchmark function. We use self-organizing maps for this projection of individuals onto a 2D space. The experimental result shows that the convergence speed of the GA with human search on the visualized space is at least five times faster than a conventional GA. The paper introduces the concept of an Interactive Evolutionary Design System (IEDS) that supports the engineering designer during the conceptual preliminary stages of the design process. Requirement during these early stages relates primarily to design search and exploration across a poorly defined space as the designer's knowledge base concerning the problem area develops. Multiobjective satisfaction plays a major role, and objectives are likely to be ill-defined and their relative importance uncertain. Interactive evolutionary search and exploration provides information to the design team that contributes directly to their overall understanding of the problem domain in terms of relevant objectives, constraints, and variable ranges. This paper describes the development of certain elements within an interactive evolutionary conceptual design environment that allows off-line processing of such information leading to a redefinition of the design space. Such redefinition may refer to the inclusion or removal of objectives, changes concerning their relative importance, or the reduction of variable ranges as a better understanding of objective sensitivity is established. The emphasis, therefore, moves from a multiobjective optimization over a preset number of generations to a relatively continuous interactive evolutionary search that results in the optimal definition of both the variable and objective space relating to the design problem at hand. The paper describes those elements of the IEDS relating to such multiobjective information gathering and subsequent design space redefinition.