Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Confirmation bias, as the term is typically used in the psychological literature, connotes the seeking or interpreting of evidence in ways that are partial to existing beliefs, expectations, or a h... <s> BIB001 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Objective: Our objective is to describe how softwareengineering might benefit from an evidence-basedapproach and to identify the potential difficultiesassociated with the approach.Method: We compared the organisation and technicalinfrastructure supporting evidence-based medicine (EBM)with the situation in software engineering. We consideredthe impact that factors peculiar to software engineering(i.e. the skill factor and the lifecycle factor) would haveon our ability to practice evidence-based softwareengineering (EBSE).Results: EBSE promises a number of benefits byencouraging integration of research results with a view tosupporting the needs of many different stakeholdergroups. However, we do not currently have theinfrastructure needed for widespread adoption of EBSE.The skill factor means software engineering experimentsare vulnerable to subject and experimenter bias. Thelifecycle factor means it is difficult to determine howtechnologies will behave once deployed.Conclusions: Software engineering would benefit fromadopting what it can of the evidence approach providedthat it deals with the specific problems that arise from thenature of software engineering. <s> BIB002 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Software managers and practitioners often must make decisions about what technologies to employ on their projects. They might be aware of problems with their current development practices (for example, production bottlenecks or numerous defect reports from customers) and want to resolve them. Or, they might have read about a new technology and want to take advantage of its promised benefits. However, practitioners can have difficulty making informed decisions about whether to adopt a new technology because there's little objective evidence to confirm its suitability, limits, qualities, costs, and inherent risks. This can lead to poor decisions about technology adoption. Software engineers might make incorrect decisions about adopting new techniques it they don't consider scientific evidence about the techniques' efficacy. They should consider using procedures similar to ones developed for evidence-based medicine. Software companies are often under pressure to adopt immature technologies because of market and management pressures. We suggest that practitioners consider evidence-based software engineering as a mechanism to support and improve their technology adoption decisions. <s> BIB003 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Context: Making best use of the growing number of empirical studies in Software Engineering, for making decisions and formulating research questions, requires the ability to construct an objective summary of available research evidence. Adopting a systematic approach to assessing and aggregating the outcomes from a set of empirical studies is also particularly important in Software Engineering, given that such studies may employ very different experimental forms and be undertaken in very different experimental contexts.Objectives: To provide an introduction to the role, form and processes involved in performing Systematic Literature Reviews. After the tutorial, participants should be able to read and use such reviews, and have gained the knowledge needed to conduct systematic reviews of their own.Method: We will use a blend of information presentation (including some experiences of the problems that can arise in the Software Engineering domain), and also of interactive working, using review material prepared in advance. <s> BIB004 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Case study is a suitable research methodology for software engineering research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims at providing an introduction to case study methodology and guidelines for researchers conducting case studies and readers studying reports of such studies. The content is based on the authors' own experience from conducting and reading case studies. The terminology and guidelines are compiled from different methodology handbooks in other research domains, in particular social science and information systems, and adapted to the needs in software engineering. We present recommended practices for software engineering case studies as well as empirically derived and evaluated checklists for researchers and readers of case study research. <s> BIB005 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> In order to draw valid conclusions when aggregating evidence it is important to describe the context in which industrial studies were conducted. This paper structures the context for empirical industrial studies and provides a checklist. The aim is to aid researchers in making informed decisions concerning which parts of the context to include in the descriptions. Furthermore, descriptions of industrial studies were surveyed. <s> BIB006 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Like other sciences and engineering disciplines, software engineering requires a cycle of model building, experimentation, and learning. Experiments are valuable tools for all software engineers who are involved in evaluating and choosing between different methods, techniques, languages and tools. The purpose of Experimentation in Software Engineering is to introduce students, teachers, researchers, and practitioners to empirical studies in software engineering, using controlled experiments. The introduction to experimentation is provided through a process perspective, and the focus is on the steps that we have to go through to perform an experiment. The book is divided into three parts. The first part provides a background of theories and methods used in experimentation. Part II then devotes one chapter to each of the five experiment steps: scoping, planning, execution, analysis, and result presentation. Part III completes the presentation with two examples. Assignments and statistical material are provided in appendixes. Overall the book provides indispensable information regarding empirical studies in particular for experiments, but also for case studies, systematic literature reviews, and surveys. It is a revision of the authors book, which was published in 2000. In addition, substantial new material, e.g. concerning systematic literature reviews and case study research, is introduced. The book is self-contained and it is suitable as a course book in undergraduate or graduate studies where the need for empirical studies in software engineering is stressed. Exercises and assignments are included to combine the more theoretical material with practical aspects. Researchers will also benefit from the book, learning more about how to conduct empirical studies, and likewise practitioners may use it as a cookbook when evaluating new methods or techniques before implementing them in their organization. <s> BIB007 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Context is a central concept in empirical software engineering. It is one of the distinctive features of the discipline and it is an in-dispensable part of software practice. It is likely responsible for one of the most challenging methodological and theoretical problems: study-to-study variation in research findings. Still, empirical software engineering research is mostly concerned with attempts to identify universal relationships that are independent of how work settings and other contexts interact with the processes important to software practice. The aim of this paper is to provide an overview of how context affects empirical research and how empirical software engineering research can be better 'contextualized' in order to provide a better understanding of what works for whom, where, when, and why. We exemplify the importance of context with examples from recent systematic reviews and offer recommendations on the way forward. <s> BIB008 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Background - Validity threats should be considered and consistently reported to judge the value of an empirical software engineering research study. The relevance of specific threats for a particular research study depends on the worldview or philosophical worldview of the researchers of the study. Problem/Gap - In software engineering, different categorizations exist, which leads to inconsistent reporting and consideration of threats. Contribution - In this paper, we relate different worldviews to software engineering research methods, identify generic categories for validity threats, and provide a categorization of validity threats with respect to their relevance for different world views. Thereafter, we provide a checklist aiding researchers in identifying relevant threats. Method - Different threat categorizations and threats have been identified in literature, and are reflected on in relation to software engineering research. Results - Software engineering is dominated by the pragmatist worldviews, and therefore use multiple methods in research. Maxwell's categorization of validity threats has been chosen as very suitable for reporting validity threats in software engineering research. Conclusion - We recommend to follow a checklist approach, and reporting first the philosophical worldview of the researcher when doing the research, the research methods and all threats relevant, including open, reduced, and mitigated threats. <s> BIB009 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably. Objective: This paper presents guidelines for conducting literature reviews using a snowballing approach, and they are illustrated and evaluated by replicating a published systematic literature review. Method: The guidelines are based on the experience from conducting several systematic literature reviews and experimenting with different approaches. Results: The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review. Conclusions: It is concluded that using snowballing, as a first search strategy, may very well be a good alternative to the use of database searches. <s> BIB010 </s> Software Testing Process Models Benefits&Drawbacks: a Systematic Literature Review <s> Threats to validity <s> Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey literature in addition to the published (formal) literature. Only a few MLRs have been published in SE so far. We aim at raising the awareness for MLRs in SE by addressing two research questions (RQs): (1) What types of knowledge are missed when a SLR does not include the multivocal literature in a SE field? and (2) What do we, as a community, gain when we include the multivocal literature and conduct MLRs? To answer these RQs, we sample a few example SLRs and MLRs and identify the missing and the gained knowledge due to excluding or including the grey literature. We find that (1) grey literature can give substantial benefits in certain areas of SE, and that (2) the inclusion of grey literature brings forward certain challenges as evidence in them is often experience and opinion based. Given these conflicting viewpoints, the authors are planning to prepare systematic guidelines for performing MLRs in SE. <s> BIB011
Evidence Based Software Engineering (EBSE) emerged over the years as a way to support software engineering practices with a more scientific approach BIB003 BIB002 . Since the emergence of EBSE, a large number of SLRs were performed in the area  . However, running SLRs poses several challenges in terms of limitations and validity threats, being many of the phases subjective with possibilities of introducing researchers bias and errors BIB004   . There are many classifications of validity threats that can be applied to software engineering empirical studies (e.g. for case studies - BIB005 , or experiments in software engineering - BIB007 ). To report threats to validity, we selected the guidelines proposed by Petersen and Gencel BIB009 and used in  , as they are particularly fit for qualitative stud-ies such as SLRs-slightly different than traditional validity threats (internal, construct, conclusion, external threats) typically used in software engineering studies BIB007 . Theoretical Validity. Theoretical validity refers to threats in building theory out of the observed phenomenon BIB009 . Researcher bias is a theoretical threat in the selection of studies that were part of the SLR. The SLR was conducted by one main researcher, however, the selection of articles was validated by a second researcher to reduce this selection bias. Final results of included papers might be influenced by the views of the two researchers, we reduced this threat by adopting SLR protocols   . Publication bias is another threat, as the published research might only discuss positive aspects / projects, while negative or controversial aspects / results might not be published. Especially when considering advantages and drawbacks of testing processes, the positive reported research might heavily surpass negative research due to this biasthis however, might not reflect the real state in industry. Reading this SLR has to take into account the presence of this bias, as it is difficult to completely remove. The quality of the sample of studies obtained with respect to the targeted population is another theoretical threat. It is known that an SLR cannot cover the whole population of research published in the area, but rather provide a subset  . We mitigated this threat by using different search queries for the search engines, as each one has its own peculiarities. This provided a better initial result-set to start with. We further decided to increase the quality of the selected studies by including only studies that met a certain threshold of reference citations. Descriptive Validity. Descriptive validity deals with the accuracy of the facts reported by researchers based on the observed phenomenon BIB009 . Poorly designed data extraction forms and recording of data can be a descriptive validity threat. In our case, we used the JabRef tool (http://www.jabref.org) to store, manage, and annotate reference lists. Such tool was used for collaboration between researchers, tracking all the selection process. We believe this threat was in full control, as we could always go back to a previous step of the SLR process, if needed, and the whole reporting process was based on a single repositories for collaboration across researchers. About the extensiveness of the review, we did not consider the inclusion of grey literature BIB011 or the snowballing technique BIB010 to search related literature. The reason was mainly for time constraints, but also other challenges that are implied in grey literature, like the definition of a strategy for the identification of the credibility of sources  . Interpretative (Construct) Validity. Interpretative validity, deals with how much reliable are the conclusions that have been drawn from the evidence collected BIB009 . Researchers bias can play a role also in this context, as interpretation of data might be steered according to previous beliefs and thoughts, with confirmation bias being one of such biases BIB001 . We believe this threat to validity is reduced in the current study, as the research protocol involved evaluation of the data and drawing of conclusions from the researchers involved. External Validity. External validity deals with how much the findings could be generalized to other cases BIB009 . In our context, the relevance would be on how much we can generalize empirical results from the reviewed articles to other industrial cases. In software engineering, there is high variability of results that depends on the context BIB008 BIB006 . In this SLR, we reported the domain in which different testing processes were applied: we cannot ensure that results can be generalized even to other cases within the same domain: as such, readers need to take into account this threat when consulting the SLR findings. Repeatability / Reproducibility Validity. Repeatability / reproducibility Validity deals with how much repeatable (reproducible) the performed process is. Reproducible research can be defined as any research that can be reproduced from the published materials by other researchers independently  . Throughout the study, we followed the concept of reproducible research   . Following the SLR protocol   and reporting accurately all the steps was one way to increase the repeatability and reproducibility of the current study. Some parts, such as running the queries ex-post and all the parts that involve human judgement might involve challenges, but all the intermediary steps of research as stored in JabRef files will be available to other researchers. More advanced structured approaches to build search strings could be used, such as PICO (Patient/Intervention/Control/Outcome)  . We believe this is a limited threat, as, in our case, we defined keywords from the research questions and used them for constructing search strings. To improve the results, we had several iterations trying different queries and optimizing them according to the search results.