Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Challenges regarding Fit (cost and restrictions) <s> This report describes the practice of using executable acceptance testing for specifying programming assignments in software engineering courses. We summarize experiences from two courses introduced in two academic institutions over four semesters -- both from students' and instructors' perspectives. Examples of projects and the discussion of the assignment flows are given. The paper highlights testing as an all-encompassing activity in software development projects. It also contains recommendations for academics thinking of incorporating executable acceptance testing into their courses. <s> BIB001 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Challenges regarding Fit (cost and restrictions) <s> Ever in search of a silver bullet for automated functional testing for Web applications, many folks have turned to Selenium. Selenium is an open-source project for in-browser testing, originally developed by ThoughtWorks and now boasting an active community of developers and users. One of Selenium's stated goals is to become the de facto open-source replacement for proprietary tools such as WinRunner. Of particular interest to the agile community is that it offers the possibility of test-first design of Web applications, red-green signals for customer acceptance tests, and an automated regression test bed for the Web tier. This experience report describes the standard environment for testing with Selenium, as well as modifications we performed to incorporate our script pages into a wiki. It includes lessons we learned about continuous integration, script writing, and using the Selenium Recorder (renamed IDE). We also discuss how long it took to write and maintain the scripts in the iterative development environment, how close we came to covering all of the functional requirements with tests, how often the tests should be (and were) run, and whether additional automated functional testing below the GUI layer was still necessary and/or appropriate. While no silver bullet, Selenium has become a valuable addition to our agile testing toolkit, and is used on the majority of our Web application projects. It promises to become even more valuable as it gains widespread adoption and continues to be actively developed. <s> BIB002 </s> Automated Acceptance Testing: A Literature Review and an Industrial Case Study <s> Challenges regarding Fit (cost and restrictions) <s> This research investigates the process of Executable Acceptance Test-Driven Development (EATDD) in the context of specifying functional requirements using the FIT framework, when developing line-of-business applications. ::: It is guided by three key research questions: (1) how business and technology experts utilize EATDD in the software development lifecycle; (2) what kind of benefits and limitations EATDD manifests, and (3) to what extent improvements in software quality (if any) are associated with EATDD? ::: The research employs methods of quantitative and qualitative inquiries. Based on the findings of two academic observational studies, one academic quasiexperiment, and three industrial multi-case studies, the following main conclusions are drawn: (1) the use of Executable Acceptance Test-Driven Development is correlated with the enhanced communication in software teams; (2) executable acceptance tests are suitable for specifying functional requirements and are in fact unambiguous, consistent, verifiable, and usable (from both the business experts' and technology experts' perspectives); (3) EATDD provides sufficient evidence of requirements traceability in regulated environments; and (4) current state of tool support negatively impacts the maintainability and scalability of the artifacts produced in the course of EATDD. ::: In addition, our contribution includes emerged conceptualizations and the sociotechnical model of the EATDD process. These not only help to explain the ways how EATDD is used in practice, but also form a base for future investigations. <s> BIB003
Running test-driven development based on unit testing means defining small unit tests, code, and immediately seeing the result. A test defined at acceptance test level, however, can take a long time to code. According to a study by Deng et al., the average time to code a passing acceptance test was more than four hours, for some extending to several iterations  . The problem arises when trying to separate between 'unimplemented failures' and (the more important) regression failures. They have tried to solve this by creating an Eclipse addon, FitClipse, which knows if a test has previously passed or not. Another problem lies in the assumption that a passing test suite means the code is good. Melnik and Maurer touch briefly on the topic when they describe the "deceptive sense of security" students had when the tests passed, and warn about not thinking outside the box BIB001 . Melnik also mentions a team calling the collaborative writing of acceptance tests "bartering", and this team felt using Fit helped them implement features in an economically feasible yet still correct way BIB003 . Fit got the lowest score (2/12 points) of all test notation tools surveyed (albeit not a completed survey) when used for describing non-functional requirements  . This included performance, security, and usability. The opposite may also pose a problem. In a Selenium setting, small changes in the user interface (renaming a button) leads to failing the test even if the application (business logic) is intact BIB002 .