Researchers, students and practitioners often encounter a situation when the build process of a third-party software system fails. In this paper, we aim to confirm this observation present mainly as anecdotal evidence so far. Using a virtual environment simulating a programmer's one, we try to fully automatically build target archives from the source code of over 7,200 open source Java projects. We found that more than 38 of builds ended in failure. Build log analysis reveals the largest portion of errors are dependency-related. We also conduct an association study of factors affecting build success.
found that 18 A study @cite_10 conducted at Google showed that 30 According to , 59 described association of various project characteristics with the success of their build. They analyzed existing builds on a continuous integration server, while we created a virtual environment simulating a developer's local system.
@cite_10
2591823109 2065181971 2099805222 1991292921
Growth in Google's code size and feature churn rate has seen increased reliance on continuous integration (CI) and testing to maintain quality. Even with enormous resources dedicated to testing, we are unable to regression test each code change individually, resulting in increased lag time between code check-ins and test result feedback to developers. We report results of a project that aims to reduce this time by: (1) controlling test workload without compromising quality, and (2) distilling test results data to inform developers, while they write code, of the impact of their latest changes on quality. We model, empirically understand, and leverage the correlations that exist between our code, test cases, developers, programming languages, and code-change and test-execution frequencies, to improve our CI and development processes. Our findings show: very few of our tests ever fail, but those that do are generally "closer" to the code they test, certain frequently modified code and certain users tools cause more breakages, and code recently modified by multiple developers (more than 3) breaks more often. In this position paper, we present and demonstrate the idea of using an interdisciplinary literature review to accelerate the research on continuous integration practice. A common suggestion has been that build waiting time in continuous integration cycle should be less than 10 minutes. This guideline is based on practitioners' opinion and has not been further investigated. The objective of this study is to understand the effects of build waiting time in software engineering and to get input from waiting time research in other disciplines. The objective is met by performing two literature reviews, first on build waiting time and second on waiting times in the contexts of service operation, web use and computer use. The found effects of build waiting time were categorized into continuous integration specific, cognitive and emotional. Two minute build waiting time was considered optimal, but under 10 minutes was considered acceptable. Insight from other waiting time research suggests that the perceptions of waiting time are important and the perceptions can be lowered by providing feedback and giving developers other activities during the integration. Slow builds can be extremely costly; for example, one of our customers loses forty percent of their developers' productivity to waiting on builds. Build avoidance - building only what must be built to integrate a change - solves the problem but makes the build unreliable unless the build script captures all dependences of build outputs on sources. Therefore, in practice, integrating even a minor change often requires rebuilding a large software system from scratch.Our solution can be summarized as "trust but verify". Exploiting the fact that large software systems tend to be composed of loosely coupled parts, we semi-automatically divide a large, all-or-nothing build into many small "mini-builds", without changing the original build. Each mini-build declares its dependences, which are trusted while scheduling mini-builds to integrate a change but verified by executing each mini-build in a sandbox in which only its declared dependences are available.We implemented these ideas in a tool suite called Grexmk and applied Grexmk to two large all-or-nothing builds. On the first build, we demonstrated linear speedups to six build machines. On the second build, we sped up the average time for an incremental build by a factor of 1.2. In continuous integration development environments, software engineers frequently integrate new or changed code with the mainline codebase. This can reduce the amount of code rework that is needed as systems evolve and speed up development time. While continuous integration processes traditionally require that extensive testing be performed following the actual submission of code to the codebase, it is also important to ensure that enough testing is performed prior to code submission to avoid breaking builds and delaying the fast feedback that makes continuous integration desirable. In this work, we present algorithms that make continuous integration processes more cost-effective. In an initial pre-submit phase of testing, developers specify modules to be tested, and we use regression test selection techniques to select a subset of the test suites for those modules that render that phase more cost-effective. In a subsequent post-submit phase of testing, where dependent modules as well as changed modules are tested, we use test case prioritization techniques to ensure that failures are reported more quickly. In both cases, the techniques we utilize are novel, involving algorithms that are relatively inexpensive and do not rely on code coverage information -- two requirements for conducting testing cost-effectively in this context. To evaluate our approach, we conducted an empirical study on a large data set from Google that we make publicly available. The results of our study show that our selection and prioritization techniques can each lead to cost-effectiveness improvements in the continuous integration process.