People-Centered Software Development: An Overview of Agile Methodologies <s> opment of Graphical User Interfaces <s> Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. In this paper we show that the test oracle used during testing also contributes significantly to test effectiveness and cost. A test oracle is a mechanism that determines whether a software executed correctly for a test case. We define a test oracle to contain two essential parts: oracle information that represents expected output, and an oracle procedure that compares the oracle information with the actual output. By varying the level of detail of oracle information and changing the oracle procedure, a test designer can create different types of test oracles. We design 11 types of test oracles and empirically compare them on four software systems. We seed faults in each software to create 100 faulty versions, execute 600 test cases on each version, for all 11 types of oracles. In all, we report results of 660,000 test runs on each software. We show (1) the time and space requirements of the oracles, (2) that faults are detected early in the testing process when using detailed oracle information and complex oracle procedures, although at a higher cost per test case, and (3) that employing expensive oracles results in detecting a large number of faults using relatively smaller number of test cases. <s> BIB001 </s> People-Centered Software Development: An Overview of Agile Methodologies <s> opment of Graphical User Interfaces <s> Ever in search of a silver bullet for automated functional testing for Web applications, many folks have turned to Selenium. Selenium is an open-source project for in-browser testing, originally developed by ThoughtWorks and now boasting an active community of developers and users. One of Selenium's stated goals is to become the de facto open-source replacement for proprietary tools such as WinRunner. Of particular interest to the agile community is that it offers the possibility of test-first design of Web applications, red-green signals for customer acceptance tests, and an automated regression test bed for the Web tier. This experience report describes the standard environment for testing with Selenium, as well as modifications we performed to incorporate our script pages into a wiki. It includes lessons we learned about continuous integration, script writing, and using the Selenium Recorder (renamed IDE). We also discuss how long it took to write and maintain the scripts in the iterative development environment, how close we came to covering all of the functional requirements with tests, how often the tests should be (and were) run, and whether additional automated functional testing below the GUI layer was still necessary and/or appropriate. While no silver bullet, Selenium has become a valuable addition to our agile testing toolkit, and is used on the majority of our Web application projects. It promises to become even more valuable as it gains widespread adoption and continues to be actively developed. <s> BIB002 </s> People-Centered Software Development: An Overview of Agile Methodologies <s> opment of Graphical User Interfaces <s> Graphical user interfaces (GUIs) are one of the most commonly used parts of today's software. Despite their ubiquity, testing GUIs for functional correctness remains an understudied area. A typical GUI gives many degrees of freedom to an end-user, leading to an enormous input event interaction space that needs to be tested. GUI test designers generate and execute test cases (modeled as sequences of user events) to traverse its parts; targeting a subspace in order to maximize fault detection is a nontrivial task. In this vein, in previous work, we used informal GUI code examination and personal intuition to develop an event-interaction graph (EIG). In this article we empirically derive the EIG model via a pilot study, and the resulting EIG validates our intuition used in previous work; the empirical derivation process also allows for model evolution as our understanding of GUI faults improves. Results of the pilot study show that events interact in complex ways; a GUI's response to an event may vary depending on the context established by preceding events and their execution order. The EIG model helps testers to understand the nature of interactions between GUI events when executed in test cases and why certain events detect faults, so that they can better traverse the event space. New test adequacy criteria are defined for the EIG; new algorithms use these criteria and EIG to systematically generate test cases that are shown to be effective on four fielded open-source applications. <s> BIB003 </s> People-Centered Software Development: An Overview of Agile Methodologies <s> opment of Graphical User Interfaces <s> Despite its importance, testing graphical user interfaces (GUIs) is frequently seen as a difficult activity. The challenges present in GUI testing may discourage developers and testers, to the point that they may eventually stop writing tests. This paper describes the common problems that make creation and maintenance of GUI tests complex. Our intention is to present a GUI testing strategy that includes writing robust GUI tests, creating testable GUIs and testing legacy/existing GUIs. Simultaneously, we discuss the features that a tool should offer to simplify GUI testing. We cover the intricacies that developers and testers face when writing, maintaining and troubleshooting GUI tests, and offer practical solutions to each of them.The ideas explained in this paper have been successfully implemented in the open source project FEST, a GUI testing library for Java Swing GUIs. <s> BIB004 </s> People-Centered Software Development: An Overview of Agile Methodologies <s> opment of Graphical User Interfaces <s> Although graphical user interfaces (GUIs) constitute a large part of the software being developed today and are typically created using rapid prototyping, there are no effective regression testing techniques for GUIs. The needs of GUI regression testing differ from those of traditional software. When the structure of a GUI is modified, test cases from the original GUI's suite are either reusable or unusable on the modified GUI. Because GUI test case generation is expensive, our goal is to make the unusable test cases usable, thereby helping to retain the suite's event coverage. The idea of reusing these unusable (obsolete) test cases has not been explored before. This article shows that a large number of test cases become unusable for GUIs. It presents a new GUI regression testing technique that first automatically determines the usable and unusable test cases from a test suite after a GUI modification, then determines the unusable test cases that can be repaired so that they can execute on the modified GUI, and finally uses repairing transformations to repair the test cases. This regression testing technique along with four repairing transformations has been implemented. An empirical study for four open-source applications demonstrates that (1) this approach is effective in that many of the test cases can be repaired, and is practical in terms of its time performance, (2) certain types of test cases are more prone to becoming unusable, and (3) certain types of “dominator” events, when modified, make a large number of test cases unusable. <s> BIB005 </s> People-Centered Software Development: An Overview of Agile Methodologies <s> opment of Graphical User Interfaces <s> End customers increasingly access the deliveredfunctionality in software systems through a GUI.Unfortunately, limited data is currently available on howdefects in these systems affect customers. This paperpresents a study of customer-reported GUI defects fromtwo different industrial software systems developed atABB. This study includes data on defect impact, location,and resolution times. The results show that (1) 65% ofthe defects resulted in a loss of some functionality to theend customer, (2) the majority of the defects found (60%)were in the GUI, as opposed to the application itself, and(3) defects in the GUI took longer to fix, on average,than defects in the underlying application. The resultsare now being used to improve testing activities at ABB. <s> BIB006 </s> People-Centered Software Development: An Overview of Agile Methodologies <s> opment of Graphical User Interfaces <s> Lack of tool support is hindering the growing interest in incorporating usability evaluation into agile software development practices. To address this concern we have developed ActiveStory Enhanced, a tool for creation and remote evaluation of low-fidelity prototypes. Building on its predecessor, ActiveStory Enhanced remotely collects more forms of usability data and provides new visualizations of this data to aid in the detection of usability flaws. <s> BIB007 </s> People-Centered Software Development: An Overview of Agile Methodologies <s> opment of Graphical User Interfaces <s> This paper introduces rule-based exploratory testing, an approach to GUI testing that combines aspects of manual exploratory testing with rule-based test automation. This approach uses short, automated rules to increase the bug-detection capability of recorded exploratory test sessions. A preliminary evaluation found that this approach can be used to detect both general and application-specific bugs, but that rules for general bugs are easier to transfer between applications. Also, despite the advantages of keyword-based testing, it interferes with the transfer of rules between applications. <s> BIB008
Further, user interfaces can be either event-driven or loop-driven. Event-driven interfaces primarily respond to input from the user. Examples of event-driven interfaces include traditional desktop applications and web pages. Loop-driven interfaces are primarily driven by the passage of time, but will also take user input into account. Many computer games are excellent examples of loop-driven interfaces. The difference is that in an event-driven interface a sequence of interactions will produce the same result regardless of timing, but in a loop-driven interface this is unpredictable. For the purposes of this chapter, let us consider only event-driven graphical user interfaces (GUIs) based on mouse, keyboard, or touch interaction. While powerful patterns for dealing with the complexity of GUIs exist (e.g. the Model-ViewController pattern), there is still a significant amount of code present in a GUI -in fact, 45-60% of an application's code can be dedicated to its GUI  . In line with this, one case study found that 60% of software defects found after release relate to GUI code, and of these defects, 65% result in a loss of functionality BIB006 . Taken together, these studies suggest that GUI testing is an area of significant concern. However, automated GUI testing is far from straightforward. In order to better understand what makes GUI testing a daunting task, let us consider four fundamental concerns of automated software testing made especially clear in this context: • Complexity, • Verification, • Change, and • Cross-Process Testing The complexity of an application refers to the number of alternative actions that are possible. GUIs allow a great amount of freedom to user interaction, making them very complex. When testing the functionality of a GUI-based application using automated tests, two factors are of prime importance: the number of steps in the test; and the number of times each action in the GUI is taken within a test BIB003 . In Section 3.4, the implications of the complexity of modern GUIs will be explored in more detail. In order to notice that a bug has been triggered, a test must also contain verifications that will be able to notice that bug BIB001 . This is especially tricky when considering that many aspects of GUIs are subjective. For example, it can be difficult to create a test for determining whether a web page was rendered correctly. Third, GUIs tend to change drastically over the course of development. A GUI test can show up as failing although the underlying code is actually working [19] [20] . This is especially important since a large number of false alarms from the GUI testing suite will cause developers to lose confidence in their regression suite BIB002 . Finally, these difficulties are compounded by the fact that GUI tests generally interact with a GUI from a different process. This means that the test will not have access to the internals of the GUI it is testing. Instead of simply c that object within the GUI. elements from the root win widget -as it appeared whe this cross-process testing, i For example, the Button cla can be tested through the In The Button itself has 136 p use by test code. It can be d of the properties that isn't e With these concerns in testing actually is. There ar GUI; and performing syste ample the Wikipedia entry page that the Wikipedia l below it is a "Main page" h instead we want to verify th paragraph takes us to a pag lopedia," then we are test BIB006 http://msdn.microso 17 http://msdn.microso BIB001 From http://en.wikip Graphical_user_inte are Development: An Overview of Agile Methodologies calling a method on an object, it's necessary to first loc . This is generally done by traversing the tree of graph ndow object until a widget matching details of the desi en the test was created -is found. Additionally, because it's rare for all information about a widget to be expos ass as implemented in Windows Presentation Foundatio nvokePattern interface in the Windows Automation AP properties, but InvokePattern exposes only 20 of these difficult to create strong tests in the (common) case that exposed is important to the functionality of a feature. mind, we need to consider what the purpose of our G re two distinct forms of GUI testing: testing the look of m testing of the application through its GUI. Take for for GUI testing, Fig. 13 . If we want to verify that on logo appears as the upper-leftmost widget, that direc hyperlink, and so on, we are testing the look of the GUI hat clicking on the link to software engineering in the f e titled "Software engineering -Wikipedia, the free enc ting the functionality of the system as a whole. Ag spx for the present, let us consider the second of these approaches. Essentially, we are performing ATDD through the application's GUI instead of below it. It is possible to write GUI tests for use in test-driven development of a GUI manually using available GUI testing tools. For example, it is entirely possible to write a Selenium  test by hand before a GUI exists even though Selenium is primarily a capture/replay tool (CRT) -a testing application that records a series of interactions with a system and records them in a format that can be replayed later as a test. This approach has been supported in the past by tool like TestNG-Abbot  and FEST BIB004 , but has not received widespread uptake. This could be due to the fact that test authors need to know a large amount of detailed information about the GUI to be created in order to write a test. A simpler approach to UITDD involves the creation of an automated low-fidelity prototype using a program like ActiveStory Enhanced BIB007 or SketchFlow BIB005 . These prototypes are event-based GUIs that respond to user input in the same way in which actual GUIs do. This means that they generate events when a user interacts with them. These events can be captured using a CRT, like white BIB002 or LEET BIB008 , in the same way in which they can be used to record events from an actual GUI. These events can then be replayed on the actual GUI, with one caveat: the elements in the prototype that are generating events need to have the same identifying information as the equivalent elements in the actual GUI. Consider for example the prototype shown in Fig. 14 . It was created in SketchFlow, which means that each widget will raise recordable events when interacted with. We can use this prototype both for testing the actual GUI and testing the actual application through its GUI. From the prototype, we can use information about, for example, the arrangement of widgets to create tests of the GUI, or we could use the functionality demonstrated through the prototype to create acceptance tests of the actual application. For example, we can fill in the fields as shown in Fig. 14 , then click the "Clear Report" button and verify that the fields have been cleared. We can then use this test for verification of both the form and functionality of the actual application, Fig. 15 . There are several advantages to this approach. First, this approach to UITDD has the advantage of being able to make use of CRTs, which makes it much easier to create tests than it would be to create them by hand. Second, by integrating mediumfidelity prototyping into the TDD process, we are creating another opportunity for testing -usability testing, as described in Section 2.3.3. This means that it is possible to detect usability errors early in the development process, which not only makes them cheaper to fix, but also reduces the number of changes to the GUI that will be necessary later in the software development process. This reduces the risk that changes to the GUI will break GUI tests since there will be fewer of them. Third, this approach reduces the appar us specify which parts of i flows of each feature of the we will automatically know events) we need to focus on