Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Literature review method <s> While empirical studies in software engineering are beginning to gain recognition in the research community, this subarea is also entering a new level of maturity by beginning to address the human aspects of software development. This added focus has added a new layer of complexity to an already challenging area of research. Along with new research questions, new research methods are needed to study nontechnical aspects of software engineering. In many other disciplines, qualitative research methods have been developed and are commonly used to handle the complexity of issues involving human behaviour. The paper presents several qualitative methods for data collection and analysis and describes them in terms of how they might be incorporated into empirical studies of software engineering, in particular how they might be combined with quantitative methods. To illustrate this use of qualitative methods, examples from real software engineering studies are used throughout. <s> BIB001 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Literature review method <s> Storytest-driven development is a complementary form of test-driven development (TDD) applied to overall system development. Using FitLibrary eases storytest-driven development, which brings together requirements and automated testing ideas and practices to support agile software development. Storytest-driven development helps bridge the gap between what organizations really need from a system and the system that developers actually deliver. This bridge builds on several factors. Storytest-driven development builds on and reuses ideas from previous requirements approaches, turning some of them "inside out." Storytests don't express class diagrams and use cases directly, for example, but they're likely to emerge; making such connections explicit is a fruitful area of future work. Another promising research area is to bring a stronger task-oriented (usability) perspective to storytests. <s> BIB002
The type of literature review reported in this paper can be classified as an integrative literature review  which identifies publications of relevance, evaluates and selects publications for review and systematize them such that common central concepts are identified and structured. Thus, an integrative literature review will develop an overview of existing research which initially is not related and form a basis for further research. The literature review was executed through three steps: 1. Identification of potentially relevant literature 2. Evaluation of relevance and quality to select literature for analysis and synthesis 3. Analysis and synthesis of literature The aim of performing this review was to identify all published studies on AAT. Review step 1 was done by searching the most relevant indexing services for research reports on software engineering. This included searches in ACM Digital library BIB002 , IEEE Explore  and Web of Science  . Together these indexes cover close to all relevant scientific journals, magazines and conferences related to software development. In addition, the proceedings from the first annual conference on Extreme Programming and Flexible Processes in Software Engineering  were manually searched (as these proceedings are not indexed electronically). All sources were searched using the following six search phrases: 1. "automated acceptance test" 2. "executable acceptance test" 3. "user acceptance test" 4. "customer acceptance test" 5. "fit AND software" 6. "fitnesse" Two researchers (the authors) reviewed the title and the source (journal, magazine or conference) of the identified papers to exclude findings obviously not relevant to software development. We also removed editorials, prefaces, article summaries, interviews, news, reviews, correspondence, discussions, comments, reader's letters and summaries of tutorials, workshops, panels and poster sessions. This resulted in 50 unique publications of which 13 presenting empirical results. Review step 2 was to evaluate each paper to ensure that it either explains or discusses the concept of -or use of -AAT or/and that it reports empirical findings. Based on a review of abstracts 31 papers were removed, keeping 19 which constitute the basis for the state-of-the-practice part of this paper. During the study of the full text, 5 more papers were found to be not relevant to our review and thus removed, leaving 14 papers in total for the synthesis. Review step 3 was the synthesis of the resulting papers. The resulting collection of papers differs with respect to thematic variation, study type and scientific quality. We found that only 8 of the 19 papers reported results from empirical studies in the sense that they describe the study method, analysis and validity of results -the rest of the papers can be classified as experience reports or elaborations of concepts or ideas. AAT is a relatively new and shaping practice; we decided to keep also strictly non-empirical papers such as lighter experience reports. This was done to make sure that potentially relevant findings and trends in this new field of research are not overlooked initially. As a result of this, one can not consider this synthesis a full systematic review  but an integrative literature review  . Thus this forms an overview of the current state of research of automated acceptance testing, showing topics and concepts that have been addressed in research so far and a summary of this experience. We believe that this is useful initially, but that this knowledge base should be verified and extended with later sound empirical studies. In addition we identify topics and concept that have not been investigated so far but that should be included in future research. This consideration is based partly on the book describing AAT with Fit  and conversations with people involved in acceptance testing. The analysis of the selected literature was inspired by the practice of constant comparison BIB001 where central concepts and topics emerge through repeated analysis of the reviewed literature.