Sustainability in Software Engineering: A Systematic Literature Review for Building up a Knowledge Base <s> C. Threats to Validity <s> Participatory sensing is the process whereby individuals and communities use ever-more-capable mobile phones and cloud services to collect and analyze systematic data for use in discovery. The convergence of technology and analytical innovation with a citizenry that is increasingly comfortable using mobile phones and online social networking sets the stage for this technology to dramatically impact many aspects of our daily lives. Participatory sensing can serve as a powerful "make a case" technology to support advocacy and civic engagement. It can provide a framework in which citizens can bring to light a civic bottleneck, hazard, personal-safety concern, cultural asset, or other data relevant to urban and natural-resources planning and services, all using data that are systematic and can be validated (http://whatsinvasive.com.) The same systems can be used as tools for sustainability. For example, individuals and communities can explore their transportation and consumption habits and corporations can promote more sustainable practices among employees (http://peir.cens.ucla.edu and http://biketastic.com.) <s> BIB001 </s> Sustainability in Software Engineering: A Systematic Literature Review for Building up a Knowledge Base <s> C. Threats to Validity <s> Sustainability has become one of the "grand challenges" of our civilization. Because of their pervasiveness, the way we design, and consequently use, software-intensive systems has a significant impact on sustainability. This gives software requirements engineering an important role to play in society. However, there is currently no specific support for handling sustainability requirements, while such support exists and has proved useful for other quality requirements like security or usability. This paper reports on a software project in which sustainability requirements were treated as first class quality requirements, and as such systematically elicited, analysed and documented. The authors intended to assess how current techniques support these activities. Beyond raising awareness on the importance of sustainability concerns in requirements engineering, this experience report suggests that, while a lot of work remains to be done, small and easy steps may already lead us to more sustainable systems. It also contributes to the agenda of requirements engineering researchers concerned with sustainability. <s> BIB002 </s> Sustainability in Software Engineering: A Systematic Literature Review for Building up a Knowledge Base <s> C. Threats to Validity <s> Background: Supporting sustainability in software engineering is becoming an active area of research and we want to contribute the first SLR in this field. Researchers who are motivated to contribute to that topic need to accumulate a body of knowledge as starting point through time-consuming literature research. Aim: We aim to provide an overview of different aspects of sustainability in software engineer- ing research with regard to research activity, investigated topics, identified limitations, proposed approaches, used methods, available studies, and considered domains. Method: The applied method is a Systematic Literature Review (SLR) in five reliable and commonly-used databases according to the (quasi-standard) protocol by Kitchenham et al. [288]. We assessed the 100 first results of each database ordered by relevance with respect to the search query. Results: Of 500 classified publications, we regard 96 as relevant for our research questions. Most of the excluded publications were ruled out because of an unfitting usage of terms within the search query. We sketch a taxonomy of their topics and domains, and provide lists of used methods and proposed approaches. Conclusions: Currently, there is little research coverage on the different aspects of sustainability in software engineering. Future work includes extending the study by reviewing a higher number of publications, including dedicated journal and workshop searches, and snowballing. <s> BIB003
There is a number of threats to validity that we are aware of and tried to minimize by different mitigation actions. 1) Researcher's bias: The semi-automatic part of the search was performed by five researchers. There could be a researcher's bias as the first selection was performed by only one researcher. We minimized the effects of such a bias by two measures: • We explicitly stated the research questions, inclusion and exclusion criteria, and the rationale for performing the search. • The first selection was reviewed and assessed by two expert reviewers from different institutions (TUM, UCLM). Differences were subsequently discussed, resolved and commonly agreed upon. • The detailed analysis of the principle researcher was reviewed by all three expert reviewers (TUM, UCLM, UPC). 2) Search string validity: The search string validity can be questioned under two aspects: On one hand whether it filtered out too many publications that would have been relevant, and on the other hand whether it included too many irrelevant results and was, in either case, not the adequate search string. Indicators for too many false positives are purely hardware papers, but as the automatically found Green IT publications all contained part of the second parenthesis of the search string, they were included in the results. Then, purely application in environmental domains, for example, agricultural support systems with no explicit relation to sustainability but relevant in case they exhibited an explicit link to sustainability in their content. Furthermore, "environment" used in the sense of system environment, not nature -these samples had to be excluded by hand as well as "ecosystem" used as population of interacting systems, for example, agents. Indicators for too many relevant exclusions were that we found less relevant results than we would have expected. This can either be due to a search string that was too restrictive, to a search that was not extensive enough, or to the fact that there is rather little published yet on that specific topic. Not all publications we would have expected showed up early in the search results. For example, we missed Cabot et al.  , as they treat goal modeling for supporting sustainability in the context of conference organisation. Mahaux et al. BIB002 were also missing in the results, with their work on exploring sustainability requirements. BIB003 3) Database query evaluation: We did not have any information on which database performed which kind of search query evaluation, and a lazy versus an eager database query evaluation of the search string would probably have a significant impact on the search results, considering that we reviewed the first 100 most relevant results. In case of a "lazy" search string evaluation, the results might have included more references matching early parts of the search string than compared to matching later parts. In that case, the results might be slightly biased in terms of favoring the terms "sustainab*" and "software engineering" and subordinating "green" and "software systems". As many of the results contained the term "software system" and not "software engineering", we are confident that there was no bias introduced by database query evaluation. 4) Cross-validation of the search engines: We received hardly any double entries in the automatic search results. We would have expected some double entries in the more general databases ScienceDirect and WebOfScience. We decided not to use meta search engines in our first iteration of the SLR because relying on only one meta search engine would have made us completely dependent of the reliability of that engine, and using various meta search engines would have led to highly redundant results, as a pre-check showed. Interestingly, Web of Science found Estrin BIB001 highly ranked, which originates from IEEEXplore but was not included in the IEEEXplore results (at least not within the first 100 results). This might be a hint towards different search query evaluation. It would be one interesting step in future work to replicate the searches on more databases and meta search engines and explicitly compare the coverage.