We study the evolution of the largest known corpus of publicly available source code, i.e., the Software Heritage archive (4B unique source code files, 1B commits capturing their development histories across 50M software projects). On such corpus we quantify the growth rate of original, never-seen-before source code files and commits. We find the growth rates to be exponential over a period of more than 40 years. We then estimate the multiplication factor, i.e., how much the same artifacts (e.g., files or commits) appear in different contexts (e.g., commits or source code distribution places). We observe a combinatorial explosion in the multiplication of identical source code files across different commits. We discuss the implication of these findings for the problem of tracking the provenance of source code artifacts (e.g., where and when a given source code file or commit has been observed in the wild) for the entire body of publicly available source code. To that end we benchmark different data models for capturing software provenance information at this scale and growth rate. We identify a viable solution that is deployable on commodity hardware and appears to be maintainable for the foreseeable future.
The study of software evolution has been at the heart of software engineering since the seminal Mythical Man Month'' @cite_0 and Lehman's laws @cite_27 . The tidal wave of , making available a growing corpus of publicly available software, has spawned an impressive literature of evolution studies. Some 10 years ago a comprehensive survey @cite_15 showed predominance of studies on the evolution of individual projects. Since then large scale studies have become frequent and the question of how Lehman's laws need to be adapted to account for modern software development has attracted renewed attention, as shown in a recent survey @cite_30 that advocates for more empirical studies to corroborate findings in the literature.
@cite_0 @cite_27 @cite_30 @cite_15
2060345601 1985236007 2170825784 2006624349
1. The Tar Pit. 2. The Mythical Man-Month. 3. The Surgical Team. 4. Aristocracy, Democracy, and System Design. 5. The Second-System Effect. 6. Passing the Word. 7. Why Did the Tower of Babel Fail? 8. Calling the Shot. 9. Ten Pounds in a Five-Pound Sack. 10. The Documentary Hypothesis. 11. Plan to Throw One Away. 12. Sharp Tools. 13. The Whole and the Parts. 14. Hatching a Castrophe. 15. The Other Face. 16. No Silver Bullet -- Essence and Accident. 17. "No Silver Bullet" ReFired. 18. Propositions of The Mythical Man-Month: True or False? 19. The Mythical Man-Month After 20 Years. Epilogue. Notes and references. Index. 0201835959T04062001 The paper presents interpretations of some recently discovered laws of evolution and conservation in the largeprogram life cycle. Program development and maintenance processes are managed and implemented by people; thus in the long term they could be expected to be unpredictable, dependant on the judgments, whims, and actions of programming process participants (e.g., managers, programmers, and product users). Yet, observed, measured, and modeled regularities suggest laws that are closer to biological laws or those of modern physics than to those currently formulated in other areas subject to human influence (e.g., economics and sociology). After a brief discussion of the first four laws, to highlight underlying phenomena and natural attributes of the program evolution process, the paper concentrates on a fifth law and shows how, and why, this law represents a conservation phenomenon: the Conservation of Familiarity. After more than 40 years of life, software evolution should be considered as a mature field. However, despite such a long history, many research questions still remain open, and controversial studies about the validity of the laws of software evolution are common. During the first part of these 40 years, the laws themselves evolved to adapt to changes in both the research and the software industry environments. This process of adaption to new paradigms, standards, and practices stopped about 15 years ago, when the laws were revised for the last time. However, most controversial studies have been raised during this latter period. Based on a systematic and comprehensive literature review, in this article, we describe how and when the laws, and the software evolution field, evolved. We also address the current state of affairs about the validity of the laws, how they are perceived by the research community, and the developments and challenges that are likely to occur in the coming years. We review the empirical research on Free Libre and Open-Source Software (FLOSS) development and assess the state of the literature. We develop a framework for organizing the literature based on the input-mediator-output-input (IMOI) model from the small groups literature. We present a quantitative summary of articles selected for the review and then discuss findings of this literature categorized into issues pertaining to inputs (e.g., member characteristics, technology use, and project characteristics), processes (software development practices, social processes, and firm involvement practices), emergent states (e.g., social states and task-related states), and outputs (e.g. team performance, FLOSS implementation, and project evolution). Based on this review, we suggest topics for future research, as well as identify methodological and theoretical issues for future inquiry in this area, including issues relating to sampling and the need for more longitudinal studies.