Choreography in the embedded systems domain: A systematic literature review <s> Threats to validity <s> Background - Validity threats should be considered and consistently reported to judge the value of an empirical software engineering research study. The relevance of specific threats for a particular research study depends on the worldview or philosophical worldview of the researchers of the study. Problem/Gap - In software engineering, different categorizations exist, which leads to inconsistent reporting and consideration of threats. Contribution - In this paper, we relate different worldviews to software engineering research methods, identify generic categories for validity threats, and provide a categorization of validity threats with respect to their relevance for different world views. Thereafter, we provide a checklist aiding researchers in identifying relevant threats. Method - Different threat categorizations and threats have been identified in literature, and are reflected on in relation to software engineering research. Results - Software engineering is dominated by the pragmatist worldviews, and therefore use multiple methods in research. Maxwell's categorization of validity threats has been chosen as very suitable for reporting validity threats in software engineering research. Conclusion - We recommend to follow a checklist approach, and reporting first the philosophical worldview of the researcher when doing the research, the research methods and all threats relevant, including open, reduced, and mitigated threats. <s> BIB001 </s> Choreography in the embedded systems domain: A systematic literature review <s> Threats to validity <s> Abstract Context Systematic mapping studies are used to structure a research area, while systematic reviews are focused on gathering and synthesizing evidence. The most recent guidelines for systematic mapping are from 2008. Since that time, many suggestions have been made of how to improve systematic literature reviews (SLRs). There is a need to evaluate how researchers conduct the process of systematic mapping and identify how the guidelines should be updated based on the lessons learned from the existing systematic maps and SLR guidelines. Objective To identify how the systematic mapping process is conducted (including search, study selection, analysis and presentation of data, etc.); to identify improvement potentials in conducting the systematic mapping process and updating the guidelines accordingly. Method We conducted a systematic mapping study of systematic maps, considering some practices of systematic review guidelines as well (in particular in relation to defining the search and to conduct a quality assessment). Results In a large number of studies multiple guidelines are used and combined, which leads to different ways in conducting mapping studies. The reason for combining guidelines was that they differed in the recommendations given. Conclusion The most frequently followed guidelines are not sufficient alone. Hence, there was a need to provide an update of how to conduct systematic mapping studies. New guidelines have been proposed consolidating existing findings. <s> BIB002
The threats to validity that were addressed in this study include descriptive, theoretical and conclusion validity, generalisability and repeatability. These validity threats, according to Petersen and Gencel BIB001 , are applicable in the software engineering field and adopted in this SLR following the recommendations by Petersen et al. BIB002 , who applied them to ensure the validity of systematic mapping studies. Since systematic mapping studies and systematic literature reviews are based on a similar idea, which is to thoroughly explore the existing literature and to produce new knowledge based on the selected literature, addressing these validity threats was seen as reasonable for this study's purpose. Descriptive validity is related to the extent to which observations are objectively described BIB001 . In systematic literature reviews, the threat to descriptive validity is related to the accuracy of the description of the data extracted from the primary studies BIB002 . This threat was addressed by planning, extracting and thoroughly describing the data relevant for the stated RQs and by using a specialised tool during these tasks. The extracted data were categorised into the bibliography, research, strategy and adoption data groups. NVivo, a specialised software package for qualitative analysis  , was used to support the extraction, storage and management of the extracted data. Theoretical validity is related to the appropriateness of the research approach to derive the intended outcomes BIB001 . In systematic literature reviews, a threat to theoretical validity is related to the extent to which the implementation of the search protocol is likely to result in studies that are relevant to the study's goal BIB002 . Theoretical validity was addressed by applying the following countermeasures: a) a thorough description of the rationale behind the keywords used for the search string is provided, b) the search string that was derived based on the keywords was piloted and the search results were checked against the studies know to researchers, c) two researchers were involved in deciding whether the publication is suitable for this study and d) reference lists of the selected primary studies were manually checked to identify additional relevant studies. Conclusion or interpretation validity is related to the association between the results and the data based on which those results were derived BIB001 . In systematic literature reviews, threats to conclusion validity include subjective decisions that may occur during the selection of primary studies and data extraction BIB002 . Conclusion validity was addressed by applying the following countermeasures: a) all involved researchers were familiar with the search protocol, b) at least two researchers were involved in the selection of the studies based on reading the abstract and based on scanning the introduction, conclusion and illustrations, c) criteria used during the selection were piloted beforehand and adjusted based on the pilot results and d) all the disagreements over the selections were handled through the joint discussions. Generalisability is related to the extent to which the study results can be generalised, or applied in different contexts BIB001 . In systematic literature reviews, generalisability is related to the extent to which the selected primary studies can represent the topic in focus BIB002 . To ensure generalisability, we applied the following countermeasures: a) a systematic literature review method that is well-accepted in the software engineering field was adopted and rigorously implemented, b) broad coverage of the literature sources was achieved by searching eight relevant databases, and c) the reference lists of the selected studies were manually checked to additionally support the literature search. Repeatability is related to the ability to replicate a study process and to achieve identical results BIB001 . In systematic literature reviews, a threat to reliability is related to the possibility of obtaining different sets of primary studies if the study process is replicated BIB002 . To address this threat, a thorough description of the implementation of the search protocol is provided. This makes it possible to replicate this study with the same outcomes. This study also runs the risk of positive publication bias. Positive publication bias suggests that studies with negative results are never or rarely published, making the search process and the selected studies biased towards positive results. Although Petersen et al. BIB002 considers publication bias to be part of theoretical validity, this threat was presented separately since it is inherent in studies of this kind and consequently not addressed in this review.