We study the evolution of the largest known corpus of publicly available source code, i.e., the Software Heritage archive (4B unique source code files, 1B commits capturing their development histories across 50M software projects). On such corpus we quantify the growth rate of original, never-seen-before source code files and commits. We find the growth rates to be exponential over a period of more than 40 years. We then estimate the multiplication factor, i.e., how much the same artifacts (e.g., files or commits) appear in different contexts (e.g., commits or source code distribution places). We observe a combinatorial explosion in the multiplication of identical source code files across different commits. We discuss the implication of these findings for the problem of tracking the provenance of source code artifacts (e.g., where and when a given source code file or commit has been observed in the wild) for the entire body of publicly available source code. To that end we benchmark different data models for capturing software provenance information at this scale and growth rate. We identify a viable solution that is deployable on commodity hardware and appears to be maintainable for the foreseeable future.
With few exceptions @cite_26 , though, file-level provenance has received little attention in the research community. We believe this is due to the lack of a reference archive of public source code on which file-level provenance tracking can be implemented once and then reused by other researchers. In the final part of this paper we discuss the implications of our findings about public source code and explore the feasibility of such provenance service'' approach, relying on as a proxy of public source code.
@cite_26
1975365403
Software clone detection has made substantial progress in the last 15 years, and software clone analysis is starting to provide real insight into how and why code clones are born, evolve, and sometimes die. In this position paper, we make the case that there is a more general problem lurking in the background: software artifact provenance analysis. We argue that determining the origin of software artifacts is an increasingly important problem with many dimensions. We call for simple and lightweight techniques that can be used to help narrow the search space, so that more expensive techniques - including manual examination can be used effectively on a smaller candidate set. We predict the problem of software provenance will lead towards new avenues of research for the software clones community.