Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Using Fit to create requirements specifications <s> An XP customer needs to write and check acceptance tests. However, the format for defining the tests needs to be clear. Many acceptance test approaches use arcane formats which do not promote clarity for the customer, due to a conflict of interest between the complexities of automation and the needs of the customer. We discuss the evolution of acceptance tests to improve their clarity for the customer. Sat is an acceptance test system for testing socket-based servers with multiple clients. The first version used an XML file to define the tests in a test suite. Any errors detected were written to a text log. There were two problems with this first version. The XML format made it difficult to read and edit the tests. When an error was given, it was not easy to identify the place in the test where the problem occurred. Sat was altered to make use of Fit, a testing framework that uses HTML tables for defining tests and reporting any errors. We found the new version considerably easier to use. The tabular form makes it much simpler to read and alter the tests. Any errors are reported in a copy of the tables, in the place where they occur. We have also found it convenient to include information about the tests in the HTML, providing a form of "literate testing". <s> BIB001 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Using Fit to create requirements specifications <s> This report describes the practice of using executable acceptance testing for specifying programming assignments in software engineering courses. We summarize experiences from two courses introduced in two academic institutions over four semesters -- both from students' and instructors' perspectives. Examples of projects and the discussion of the assignment flows are given. The paper highlights testing as an all-encompassing activity in software development projects. It also contains recommendations for academics thinking of incorporating executable acceptance testing into their courses. <s> BIB002 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Using Fit to create requirements specifications <s> This report describes experiences of introducing executable acceptance testing in senior software engineering courses. Students in an agile environment completed a five iteration project with significant portion of requirements specified as executable acceptance tests. Furthermore, students were required to write their test suites for an additional component. Ability to learn and utilize the FIT acceptance testing framework is evaluated to find out if FIT tests can be used to replace requirement documents. The results of a survey of students' perceptions and experiences were encouraging. <s> BIB003 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Using Fit to create requirements specifications <s> This descriptive case study is about the dynamics of a software engineering team using executable acceptance test-driven development in a real world project. The experiences of a customer, a developer, and a tester were discussed. The observed consensus among multiple stakeholders speaks of the effectiveness of the practice in the given context. <s> BIB004 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Using Fit to create requirements specifications <s> This research investigates the process of Executable Acceptance Test-Driven Development (EATDD) in the context of specifying functional requirements using the FIT framework, when developing line-of-business applications. ::: It is guided by three key research questions: (1) how business and technology experts utilize EATDD in the software development lifecycle; (2) what kind of benefits and limitations EATDD manifests, and (3) to what extent improvements in software quality (if any) are associated with EATDD? ::: The research employs methods of quantitative and qualitative inquiries. Based on the findings of two academic observational studies, one academic quasiexperiment, and three industrial multi-case studies, the following main conclusions are drawn: (1) the use of Executable Acceptance Test-Driven Development is correlated with the enhanced communication in software teams; (2) executable acceptance tests are suitable for specifying functional requirements and are in fact unambiguous, consistent, verifiable, and usable (from both the business experts' and technology experts' perspectives); (3) EATDD provides sufficient evidence of requirements traceability in regulated environments; and (4) current state of tool support negatively impacts the maintainability and scalability of the artifacts produced in the course of EATDD. ::: In addition, our contribution includes emerged conceptualizations and the sociotechnical model of the EATDD process. These not only help to explain the ways how EATDD is used in practice, but also form a base for future investigations. <s> BIB005
In Fit the requirements specification is made in a tabular format. This table is connected to a set of fixtures to understand syntactically how the table is to be treated. Some insights have been made on how well Fit is able to specify such requirements, mostly via studies on student groups. The general impression is that Fit is a good tool for specifying requirements. Melnik and Maurer found that 80% of students in an experiment would (when asked afterwards) rather have requirements specified in Fit than in prose BIB002 . Similarly, Read et al. noted that the average student felt that Fit was adequately appropriate for defining functional requirements BIB003 . The same students rated writing Fit tests as just as difficult as writing JUnit 7 tests and requirements specifications in prose. When it came to creating the web service specified (the groups of students switched Fit tests among themselves) 43% of them felt they had been given good enough specifications to create the service, while 26% managed but had problems. The last 31% found the specifications to be insufficient. In a small survey of test notation tools by Geras et al., Fit scored best on 'ease of use' (together with scripts and XML)  . While Fit is intended to make the customer help write the tests, the customers don't always feel the same. In one project observed by Gandhi et al., the business analysts weren't interested in writing more than plain text, leaving it to the quality analysts to code the tests [16] . The authors suggest, in hindsight, that the Fit documents should rather be described as specifications. Likewise, Melnik and Maurer mention the use of previously-made info-sheets that the customer brings to each iteration meeting. These were internal documents containing diagrams, callouts and mock screen shots. They were not, however, to be treated as authoritative specifications BIB004 . According to the customer, these sheets made it easier to explain the context to the developers. In this particular project the customer ended up writing 40% of the tests, the developers and testers wrote 30% each. The customer felt that it was best to write these tests himself, as the domain was complex and that "…going from a general description to a test has some fluidity in interpretation." Likewise, Melnik describes another project where the tests were accompanied by embedded commentaries and occasional diagrams BIB005 . The tabular structure of the Fit tests was seen as superior to XML for writing tests for a chat server, especially because of its ease of changing the tests BIB001 . Whilst there is a growing amount of large IDE's for writing code, for instance IntelliJ  and Eclipse 9 , there are no equal tools for testing. This is also acknowledged by Melnik, he concludes the support for refactoring is the weakest point BIB005 . And as Andrea mentions, being able to write test code needs to be easier than writing normal code  . Writing acceptance tests isn't always the most prioritized thing to do. One customer felt that it was easy to put off writing them because they were complex beasts, and "…the thought of diving into that complexity, just when we thought we were done, was unpleasant" BIB004 . It was unpleasant rather than difficult, though, and the authors nicely phrase the problem as "The difficulty was not the practice [of writing Fit tests] itself, but the discipline of doing it." While finding the time (or inspiration) to write Fit tests might at times prove hard, there are some claims that doing just so leads to a better specification. The customer and the team in the said project say that using Fit led them to discovering a lot of missing pieces and inconsistencies in a story. [Ibid] This is potentially related to effects described by Melnik et al., where they claim Fit reduces noise, over-specification, and ambiguity  .