We study the evolution of the largest known corpus of publicly available source code, i.e., the Software Heritage archive (4B unique source code files, 1B commits capturing their development histories across 50M software projects). On such corpus we quantify the growth rate of original, never-seen-before source code files and commits. We find the growth rates to be exponential over a period of more than 40 years. We then estimate the multiplication factor, i.e., how much the same artifacts (e.g., files or commits) appear in different contexts (e.g., commits or source code distribution places). We observe a combinatorial explosion in the multiplication of identical source code files across different commits. We discuss the implication of these findings for the problem of tracking the provenance of source code artifacts (e.g., where and when a given source code file or commit has been observed in the wild) for the entire body of publicly available source code. To that end we benchmark different data models for capturing software provenance information at this scale and growth rate. We identify a viable solution that is deployable on commodity hardware and appears to be maintainable for the foreseeable future.
Software provenance tracking is an essential building block of several studies, in particular on vulnerability tracking, license analysis @cite_32 , and reuse @cite_42 . Provenance can be looked at different granularities @cite_41 . On one end of the spectrum, tracking the origin of code is useful when studying coding patterns across repositories @cite_3 @cite_28 . On the opposite end, tracking the origin of whole is useful when looking at the evolution of forks or project popularity @cite_4 . In between, tracking -level provenance has been for more than a decade a key element of industrial tools for license compliance offered by companies like BlackDuck, Palamida, Antelink, nexB, TripleCheck, or FossID, leading to patent portfolios @cite_25 .
@cite_4 @cite_41 @cite_28 @cite_42 @cite_32 @cite_3 @cite_25
1575826986 1975365403 2964064835 1555930296
As increasing amounts of valuable information are produced and persist digitally, the ability to determine the origin of data becomes important. In science, medicine, commerce, and government, data provenance tracking is essential for rights protection, regulatory compliance, management of intelligence and medical data, and authentication of information as it flows through workplace tasks. In this paper, we show how to provide strong integrity and confidentiality assurances for data provenance information. We describe our provenance-aware system prototype that implements provenance tracking of data writes at the application layer, which makes it extremely easy to deploy. We present empirical results that show that, for typical real-life workloads, the run-time overhead of our approach to recording provenance with confidentiality and integrity guarantees ranges from 1 -13 . Software clone detection has made substantial progress in the last 15 years, and software clone analysis is starting to provide real insight into how and why code clones are born, evolve, and sometimes die. In this position paper, we make the case that there is a more general problem lurking in the background: software artifact provenance analysis. We argue that determining the origin of software artifacts is an increasingly important problem with many dimensions. We call for simple and lightweight techniques that can be used to help narrow the search space, so that more expensive techniques - including manual examination can be used effectively on a smaller candidate set. We predict the problem of software provenance will lead towards new avenues of research for the software clones community. Software and systems traceability is widely accepted as an essential element for supporting many software development tasks. Today's version control systems provide inbuilt features that allow developers to tag each commit with one or more issue ID, thereby providing the building blocks from which project-wide traceability can be established between feature requests, bug fixes, commits, source code, and specific developers. However, our analysis of six open source projects showed that on average only 60 of the commits were linked to specific issues. Without these fundamental links the entire set of project-wide links will be incomplete, and therefore not trustworthy. In this paper we address the fundamental problem of missing links between commits and issues. Our approach leverages a combination of process and text-related features characterizing issues and code changes to train a classifier to identify missing issue tags in commit messages, thereby generating the missing links. We conducted a series of experiments to evaluate our approach against six open source projects and showed that it was able to effectively recommend links for tagging issues at an average of 96 recall and 33 precision. In a related task for augmenting a set of existing trace links, the classifier returned precision at levels greater than 89 in all projects and recall of 50 . Traceability is a key issue to ensure consistency among software artifacts of subsequent phases of the development cycle. However, few works have so far addressed the theme of tracing object oriented (OO) design into its implementation and evolving it. This paper presents an approach to checking the compliance of OO design with respect to source code and support its evolution. The process works on design artifacts expressed in the OMT (Object Modeling Technique) notation and accepts Cpp source code. It recovers an “as is” design from the code, compares the recovered design with the actual design and helps the user to deal with inconsistencies. The recovery process exploits the edit distance computation and the maximum match algorithm to determine traceability links between design and code. The output is a similarity measure associated to designdcode class pairs, which can be classified as matched and unmatched by means of a maximum likelihood threshold. A graphic display of the design with different green levels associated to different levels of match and red for the unmatched classes is provided as a support to update the design and improve its traceability to the code.