Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Discussion <s> This report describes experiences of introducing executable acceptance testing in senior software engineering courses. Students in an agile environment completed a five iteration project with significant portion of requirements specified as executable acceptance tests. Furthermore, students were required to write their test suites for an additional component. Ability to learn and utilize the FIT acceptance testing framework is evaluated to find out if FIT tests can be used to replace requirement documents. The results of a survey of students' perceptions and experiences were encouraging. <s> BIB001 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Discussion <s> This report describes the practice of using executable acceptance testing for specifying programming assignments in software engineering courses. We summarize experiences from two courses introduced in two academic institutions over four semesters -- both from students' and instructors' perspectives. Examples of projects and the discussion of the assignment flows are given. The paper highlights testing as an all-encompassing activity in software development projects. It also contains recommendations for academics thinking of incorporating executable acceptance testing into their courses. <s> BIB002 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Discussion <s> This research investigates the process of Executable Acceptance Test-Driven Development (EATDD) in the context of specifying functional requirements using the FIT framework, when developing line-of-business applications. ::: It is guided by three key research questions: (1) how business and technology experts utilize EATDD in the software development lifecycle; (2) what kind of benefits and limitations EATDD manifests, and (3) to what extent improvements in software quality (if any) are associated with EATDD? ::: The research employs methods of quantitative and qualitative inquiries. Based on the findings of two academic observational studies, one academic quasiexperiment, and three industrial multi-case studies, the following main conclusions are drawn: (1) the use of Executable Acceptance Test-Driven Development is correlated with the enhanced communication in software teams; (2) executable acceptance tests are suitable for specifying functional requirements and are in fact unambiguous, consistent, verifiable, and usable (from both the business experts' and technology experts' perspectives); (3) EATDD provides sufficient evidence of requirements traceability in regulated environments; and (4) current state of tool support negatively impacts the maintainability and scalability of the artifacts produced in the course of EATDD. ::: In addition, our contribution includes emerged conceptualizations and the sociotechnical model of the EATDD process. These not only help to explain the ways how EATDD is used in practice, but also form a base for future investigations. <s> BIB003 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Discussion <s> This descriptive case study is about the dynamics of a software engineering team using executable acceptance test-driven development in a real world project. The experiences of a customer, a developer, and a tester were discussed. The observed consensus among multiple stakeholders speaks of the effectiveness of the practice in the given context. <s> BIB004 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Discussion <s> Storytest-driven development is a complementary form of test-driven development (TDD) applied to overall system development. Using FitLibrary eases storytest-driven development, which brings together requirements and automated testing ideas and practices to support agile software development. Storytest-driven development helps bridge the gap between what organizations really need from a system and the system that developers actually deliver. This bridge builds on several factors. Storytest-driven development builds on and reuses ideas from previous requirements approaches, turning some of them "inside out." Storytests don't express class diagrams and use cases directly, for example, but they're likely to emerge; making such connections explicit is a fruitful area of future work. Another promising research area is to bring a stronger task-oriented (usability) perspective to storytests. <s> BIB005
The discussion here compiles the results from the literature review and the case study. The intention of AAT and tools such as Fit or Selenium is among others to improve communication and collaboration, both internally in a software development team and with the customer. Expressing requirements as AAT is supposed to both make it easier for the customer-side to express their requirements and aid in knowing if the system under development meets those needs. One aspect of establishing an efficient practice of AAT is the ease of learning and introduction. We see from our findings that very short and simple courses seem sufficient -also for non-technical customers. Even so, we found that it took some time and experience to be able to see which parts of the system that need test coverage. None of the projects in our case study engaged the customers in writing tests, however Read et al. BIB001 indicate that persons without an IT background may find it harder to grasp. The developers we interviewed expressed that they felt that they had a better understanding of the solution being developed and the related business area. This indicates that just giving the customer side the opportunity to express acceptance tests using tools such as Fit or Selenium is not enough, and we believe that this process must be managed better. This can for example be done by identifying the right customer representative for different parts of the solution and assisting them in the definition of tests. Thus it may not be an issue about the tool itself but about how the process is managed. One promising technique that could address this issue is storytest-driven development BIB005 . In 2007 Melnik defended his PhD thesis, focusing on empirical analyses of executable acceptance test driven development BIB003 . While this thesis was not found in our search, articles used in our summary found the basis for his thesis. He concludes with there being a correlation between automated (executable) acceptance-test driven development and the enhanced communication in software teams. We find no industrial experience on what it takes for non IT professionals, such as most customers are, to specify requirements using acceptance tests. Yet it is likely that introducing AAT to customers would require more extensive training and support. The main impression from the literature review is that stating requirements in the form of acceptance tests generally works well. Melnik et al. found that 80% of the students in an experiment rated this way of expressing requirements higher than prose BIB002 . Correspondingly, our respondents experienced that writing acceptance tests improved their understanding of the domain and the system due to the separation of test and code. Yet we must underline that the customer did not express requirements this way -this task was handled by an internal in the project taking this responsibility on behalf of the customer for testing. Only one paper discusses quality of AAT in relation to quality of implementation. Melnik et al. found that there was no such correlation [11] , and state that further research is needed. We found that it seems that writing acceptance tests make developers reflect on the design and system behavior in advance of programming. Developers also reported the tests to be valuable documentation of the intended behavior of the system, thus resembling unit tests but on a higher level. One of the obvious reasons for any type of testing is to discover errors, non-conformance or flaws that need to be resolved. We see from Read's student experiment that developers discovered both inserted errors and even non-intentional errors BIB001 . Our interview respondents reported that the new testing practice helped them find and remove more errors than before. They also felt it made them able to see integration issues more clearly. One of the most evident findings in the interviews was that developers found it more convenient to share code with other developers for two reasons. First the tests worked as internal developerdocumentation of the code, making it easier to understand what it is intended to solve. Secondly the tests made it safer to alter the code as it immediately became visible if some error was introduced. Hence we see that the use of acceptance tests may improve communication and cooperation in the development team. This is in contrast to experience in one of the other studies. Gandhi et al. observed that using acceptance tests (in the early phases) actually hindered communication as developers put too much effort into expressing correct Fit documents, on the expense of expressing only appropriate tests [16] . This seems to be a balancing act; benefit should be weighted against cost. This balance deserves further research. Only a few of the analyzed articles addressed the customer communication aspect of AAT: Melnik and Maurer found that engaging customers with an information systems background resulted in a successful adoption of Fit BIB004 . As a contrast, Gandhi et al. observed, like us, that customers preferred oldfashioned requirements descriptions in plain text [16] ; so this matter is not fully investigated. Not discussed in the literature, but obvious in our interviews is the security and confidence amongst the developers as a result of having the system covered by tests explaining intended behavior. This effect is just the same as found in studies of unit testing  as well as other studies in agile development in general  . The tests help to reduce the fear or resistance of changing the code even if this would be beneficial in development and maintenance of the code. While AAT shows to be beneficial we also find issues related to costs. Naturally, defining tests costs time and attention. Having established a test that is closely connected to the code also implies a need of maintaining the test to keep test and code synchronized. Deng also underlines this  . One team reported that using Fit helped them have a conversation about the features, so they would be both economically feasible and still correct BIB003 . The respondents we interviewed reported maintenance to be a considerable cost, especially when the tests also contained hard-coded data for checking results. Another important issue found, that applies to testing in general, is that it is easy to write a test that completes successfully, yet it may still not cover the code properly. Thus, writing tests requires careful consideration or else they can give a false sense of security. None of the studies forming the base for this discussion report quantitative data documenting the pivot point: net benefit of AAT. We here rely on developers' preference of continuing to use AAT based on their experience as an indication of the feasibility of this testing practice. All of our respondents replied positively when asked about this. Other studies also report this opinion amongst developers [11, BIB001 ; similarly we would also like to see studies of customer opinions.