A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> This paper presents a technique to select a representative set of test cases from a test suite that provides the same coverage as the entire test suite. This selection is performed by identifying, and then eliminating, the redundant and obsolete test cases in the test suite. The representative set replaces the original test suite and thus, potentially produces a smaller test suite. The representative set can also be used to identify those test cases that should be rerun to test the program after it has been changed. Our technique is independent of the testing methodology and only requires an association between a testing requirement and the test cases that satisfy the requirement. We illustrate the technique using the data flow testing methodology. The reduction that is possible with our technique is illustrated by experimental results. <s> BIB001 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal. Various goals are possible; one involves rate of fault detection, a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test and let software engineers begin correcting faults earlier than might otherwise be possible. One application of prioritization techniques involves regression testing, the retesting of software following modifications; in this context, prioritization techniques can take advantage of information gathered about the previous execution of test cases to obtain test case orderings. We describe several techniques for using test execution information to prioritize test cases for regression testing, including: 1) techniques that order test cases based on their total coverage of code components; 2) techniques that order test cases based on their coverage of code components not previously covered; and 3) techniques that order test cases based on their estimated ability to reveal faults in the code components that they cover. We report the results of several experiments in which we applied these techniques to various test suites for various programs and measured the rates of fault detection achieved by the prioritized test suites, comparing those rates to the rates achieved by untreated, randomly ordered, and optimally ordered suites. <s> BIB002 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> This paper deals with testing of nondeterministic software systems. We assume that a model of the nondeterministic system is given by a directed graph with two kind of vertices: states and choice points. Choice points represent the nondeterministic behaviour of the implementation under test (IUT). Edges represent transitions. They have costs and probabilities. Test case generation in this setting amounts to generation of a game strategy. The two players are the testing tool (TT) and the IUT. The game explores the graph. The TT leads the IUT by selecting an edge at the state vertices. At the choice points the control goes to the IUT. A game strategy decides which edge should be taken by the TT in each state. This paper presents three novel algorithms 1) to determine an optimal strategy for the bounded reachability game, where optimality means maximizing the probability to reach any of the given final states from a given start state while at the same time minimizing the costs of traversal; 2) to determine a winning strategy for the bounded reachability game, which guarantees that given final vertices are reached, regardless how the IUT reacts; 3) to determine a fast converging edge covering strategy, which guarantees that the probability to cover all edges quickly converges to 1 if TT follows the strategy. <s> BIB003 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> Testing is a major cost factor in software development. Test automation has been proposed as one solution to reduce these costs. Test automation tools promise to increase the number of tests they run and the frequency at which they run them. So why not automate every test? In this paper we discuss the question "When should a test be automated?" and the trade-off between automated and manual testing. We reveal problems in the overly simplistic cost models commonly used to make decisions about automating testing. We introduce an alternative model based on opportunity cost and present influencing factors on the decision of whether or not to invest in test automation. Our aim is to stimulate discussion about these factors as well as their influence on the benefits and costs of automated testing in order to support researchers and practitioners reflecting on proposed automation approaches. <s> BIB004 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> Pseudo-exhaustive testing uses the empirical observation that, for broad classes of software, a fault is likely triggered by only a few variables interacting. The method takes advantage of two relatively recent advances in software engineering: algorithms for efficiently generating covering arrays to represent software interaction test suites, and automated generation of test oracles using model checking. An experiment with a module of the Traffic Collision Avoidance System (TCAS) illustrates the approach testing pairwise through 6-way interactions. We also outline current and future work applying the test methodology to a large real-world application, the Personal Identity Verification (PIV) smart card. <s> BIB005 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> Software testing is an important but expensive phase of software development life cycle. During software testing and retesting, development organizations always desire to validate the software from different views. But exhaustive testing requires program execution with all possible combinations of values for program variables, which is impractical due to resource constraints. For many applications, it is possible to generate test cases automatically. But the core problem is the selection of effective test cases necessary to validate the program during the maintenance phase. This target can only be achieved by eliminating all the redundant test cases from the generated pool of test suites. In this paper, we propose a novel test case reduction technique called TestFilter that uses the statement-coverage criterion for reduction of test cases. To demonstrate the applicability of this approach, we conduct an experimental study. The results show that our technique is beneficial in identifying non-redundant test cases at a little cost. Ultimately it is beneficial to optimize time & cost spent on testing and it is also helpful during regression testing. <s> BIB006 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> Abstract: Software developers frequently conduct regression testing on a series of major, minor, or bug-fix software or firmware releases. However, retesting all test cases for each release is time-consuming. For example, it takes about 36 test-bed-days to thoroughly exercise a test suite made up of 2320 test cases for the MPLS testing area that contains 57,758 functions in Cisco IOS. The cost is infeasible for a series of regression testing on the MPLS area. Thus, the test suite needs to be reduced intelligently, not just randomly, and its fault detection capability must be kept as much as possible. The mode of safe regression test selection approach is adopted for seeking a subset of modification-traversing test cases to substitute for fault-revealing test cases. The algorithms, CW-NumMin, CW-CostMin, and CW-CostCov-B, apply the safe-mode approach in selecting test cases for achieving full-modified function coverage. It is assumed that modified functions are fault-prone, and the fault distribution of the testing area is Pareto-like. Moreover, we also assume that once a subject program is getting more mature, its fault concentration will become stronger. Only function coverage criterion is adopted because of the scalability of a software system with large code. The metrics of test's function reachability and function's test intensity are defined in this study for algorithms. Both CW-CovMax and CW-CostMin algorithms are not safe-mode, but the approaches they use still attempt to obtain a test suite with a maximal amount of function coverage under certain constraints, i.e. the effective-confidence level and time restriction. We conclude that the most effective algorithm in this study can significantly reduce the cost (time) of regression testing on the MPLS testing area to 1.10%, on the average. Approaches proposed here can be effectively and efficiently applied to the regression testing on bug-fix releases of a software system with large code, especially to the releases having very few modified functions with low test intensities. <s> BIB007 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> Most test suite reduction techniques aim to select, from a given test suite, a minimal representative subset of test cases that retains the same code coverage as the suite. Empirical studies have shown, however, that test suites reduced in this manner may lose fault detection capability. Techniques have been proposed to retain certain redundant test cases in the reduced test suite so as to reduce the loss in fault-detection capability, but these still do concede some degree of loss. Thus, these techniques may be applicable only in cases where loose demands are placed on the upper limit of loss in fault-detection capability. In this work we present an on-demand test suite reduction approach, which attempts to select a representative subset satisfying the same test requirements as an initial test suite conceding at most l% loss in fault-detection capability for at least c% of the instances in which it is applied. Our technique collects statistics about loss in fault-detection capability at the level of individual statements and models the problem of test suite reduction as an integer linear programming problem. We have evaluated our approach in the contexts of three scenarios in which it might be used. Our results show that most test suites reduced by our approach satisfy given fault detection capability demands, and that the approach compares favorably with an existing test suite reduction approach. <s> BIB008 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> In order to improve the effectiveness of fault localization, researchers are interested in test-suite reduction to provide suitable test-suite inputs. Different test-suite reduction approaches have been proposed. However, the results are usually not ideal. Reducing the test-suite improperly or excessively can even negatively affect fault-localization effectiveness. In this paper, we propose a two-step test-suite reduction approach to remove the test cases which have little or no effect on fault localization, and improve the distribution evenness of concrete execution paths of test cases. This approach consists of coverage matrix based reduction and path vector based reduction, so it analyzes not only the test cases coverage but also the concrete path information. We design and implement experiments to verify the effect of our approach. The experimental results show that our reduced test-suite can improve fault-localization effectiveness. On average, our approach can reduce the size of a test-suite in 47.87% (for Siemens programs) and 23.03% (for space program). At the same time, on average our approach can improve the fault-localization effectiveness, 2.12 on Siemens programs and 0.13 on space program by Tarantula approach. <s> BIB009 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> Man-At-The-End (MATE) attacks and fortifications are difficult to analyze, model, and evaluate predominantly for three reasons: firstly, the attacker is human and, therefore, utilizes motivation, creativity, and ingenuity. Secondly, the attacker has limitless and authorized access to the target. Thirdly, all major protections stand up to a determined attacker till a certain period of time. Digital assets range from business to personal use, from consumer devices to home networks, the public Internet, the cloud, and the Internet of Things - where traditional computer and network security are inadequate to address MATE attacks. MATE is fundamentally a hard problem. Much of the extant focus to deal with MATE attacks is purely technical; though security is more than just a technical issue. The main objective of the paper is to mitigate the consequences of MATE attacks through the human element of security and highlight the need for this element to form a part of a holistic security strategy alongside the necessary techniques and technologies. This paper contributes by taking software protection (SP) research to a new realm of challenges. Moreover, the paper elaborates the concept of MATE attacks, the different forms, and the analysis of MATE versus insider threats to present a thematic taxonomy of a MATE attack. The ensuing paper also highlights the fundamental concept of digital assets, and the core protection mechanisms and their qualitative comparison against MATE attacks. Finally, we present state-of-the-art trends and cutting-edge future research directions by taking into account only the human aspects for young researchers and professionals. <s> BIB010 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> The revolutionary concept of Software Defined Networks (SDNs) potentially provides flexible and well-managed next-generation networks. All the hype surrounding the SDNs is predominantly because of its centralized management functionality, the separation of the control plane from the data forwarding plane, and enabling innovation through network programmability. Despite the promising architecture of SDNs, security was not considered as part of the initial design. Moreover, security concerns are potentially augmented considering the logical centralization of network intelligence. Furthermore, the security and dependability of the SDN has largely been a neglected topic and remains an open issue. The paper presents a broad overview of the security implications of each SDN layer/interface. This paper contributes further by devising a contemporary layered/interface taxonomy of the reported security vulnerabilities, attacks, and challenges of SDN. We also highlight and analyze the possible threats on each layer/interface of SDN to help design secure SDNs. Moreover, the ensuing paper contributes by presenting the state-of-the-art SDNs security solutions. The categorization of solutions is followed by a critical analysis and discussion to devise a comprehensive thematic taxonomy. We advocate the production of secure and dependable SDNs by presenting potential requirements and key enablers. Finally, in an effort to anticipate secure and dependable SDNs, we present the ongoing open security issues, challenges and future research directions. <s> BIB011 </s> A survey on Test Suite Reduction frameworks and tools <s> Introduction <s> Offering real-time data security for petabytes of data is important for cloud computing. A recent survey on cloud security states that the security of users' data has the highest priority as well as concern. We believe this can only be able to achieve with an approach that is systematic, adoptable and well-structured. Therefore, this paper has developed a framework known as Cloud Computing Adoption Framework (CCAF) which has been customized for securing cloud data. This paper explains the overview, rationale and components in the CCAF to protect data security. CCAF is illustrated by the system design based on the requirements and the implementation demonstrated by the CCAF multi-layered security. Since our Data Center has 10 petabytes of data, there is a huge task to provide real-time protection and quarantine. We use Business Process Modeling Notation (BPMN) to simulate how data is in use. The use of BPMN simulation allows us to evaluate the chosen security performances before actual implementation. Results show that the time to take control of security breach can take between 50 and 125 hours. This means that additional security is required to ensure all data is well-protected in the crucial 125 hours. This paper has also demonstrated that CCAF multi-layered security can protect data in real-time and it has three layers of security: 1) firewall and access control; 2) identity management and intrusion prevention and 3) convergent encryption. To validate CCAF, this paper has undertaken two sets of ethical-hacking experiments involved with penetration testing with 10,000 trojans and viruses. The CCAF multi-layered security can block 9,919 viruses and trojans which can be destroyed in seconds and the remaining ones can be quarantined or isolated. The experiments show although the percentage of blocking can decrease for continuous injection of viruses and trojans, 97.43 percent of them can be quarantined. Our CCAF multi-layered security has an average of 20 percent better performance than the single-layered approach which could only block 7,438 viruses and trojans. CCAF can be more effective when combined with BPMN simulation to evaluate security process and penetrating testing results. <s> BIB012
Software testing ensures the quality and reliability of a System Under Test (SUT) by revealing maximum possible defects BIB003 . However, software testing is the most expensive quality assurance practice since it consumes up to 50% of the total software development cost BIB004 . Although, exhaustive testing (e.g., running all possible paths in the SUT  ) is putative to provide high confidence to development organizations regarding the SUT quality BIB005 . However, exhaustive testing is impractical due to time and budget constraints  . Moreover, execution of the entire test suite is also impractical, if high human interventions are required for testing a software system  ). In the literature, researchers have reported different testing scenarios BIB002 BIB007 , which concludes that exhaustive testing requires a considerable amount of testing budget. Rothermel et al. BIB002 reported that testing a product containing 20,000 lines of code requires seven weeks and several hundred thousand dollars to execute the entire test suite. Moreover, Lin et al. BIB007 performed an empirical analysis of regression testing using 57,758 functions and 2,320 test cases. They reported that the running time of a single test case ranges from 10 minutes to 100 minutes, which is based on the configuration sequence and required test activities. Finally, they estimated that to execute all 2,320 test cases would require 36 test-bed days. To deal with the aforementioned exhaustive testing problems, development organizations are attracted to adopt optimal testing strategies BIB003 . Analytics is useful in software design and testing  . Similarly, when we design a software, do not forget diverse security concerns BIB010 BIB011 to ensure that the software design supports penetration testing (ethical hacking) against hacking BIB012 . In the literature, three main techniques have been discussed to support regression testing  : (i) test suite reduction: to find a reduced suite by permanently eliminating redundant test cases according to certain criteria, (ii) test case selection: to select such previously generated test cases that cover the modified portion of the software, and (iii) test case prioritization: to determine the ordering of test cases based on a particular objective such as to increase the rate of Fault Detection Effectiveness (FDE). In our context, the real challenge is to determine a subset of non-redundant test cases, which finds a maximum number of possible defects similar to the original test suite BIB006 . A tester can meet this challenge by randomly selecting the generated test cases , but such random selection may end up as exclusion of essential test cases BIB008 . Consequently, it has a negative impact on the fault detection capability of the reduced suite. A practical approach to solve the test-suite size problem is to find a minimal subset of test cases automatically, while keeping their fault detection capability similar to the original test suite. Test Suite Reduction (TSR) approach focuses on finding a minimal test suite by permanently discarding the redundant test cases from the original test suite BIB009 . The optimal TSR problem is formally defined by Harrold et al. BIB001 as stated below: