We study the evolution of the largest known corpus of publicly available source code, i.e., the Software Heritage archive (4B unique source code files, 1B commits capturing their development histories across 50M software projects). On such corpus we quantify the growth rate of original, never-seen-before source code files and commits. We find the growth rates to be exponential over a period of more than 40 years. We then estimate the multiplication factor, i.e., how much the same artifacts (e.g., files or commits) appear in different contexts (e.g., commits or source code distribution places). We observe a combinatorial explosion in the multiplication of identical source code files across different commits. We discuss the implication of these findings for the problem of tracking the provenance of source code artifacts (e.g., where and when a given source code file or commit has been observed in the wild) for the entire body of publicly available source code. To that end we benchmark different data models for capturing software provenance information at this scale and growth rate. We identify a viable solution that is deployable on commodity hardware and appears to be maintainable for the foreseeable future.
Software popularity is a valuable information to modern open source developers, who constantly want to know if their systems are attracting new users, if new releases are gaining acceptance, or if they are meeting user's expectations. In this paper, we describe a study on the popularity of software systems hosted at GitHub, which is the world's largest collection of open source software. GitHub provides an explicit way for users to manifest their satisfaction with a hosted repository: the stargazers button. In our study, we reveal the main factors that impact the number of stars of GitHub projects, including programming language and application domain. We also study the impact of new features on project popularity. Finally, we identify four main patterns of popularity growth, which are derived after clustering the time series representing the number of stars of 2,279 popular GitHub repositories. We hope our results provide valuable insights to developers and maintainers, which could help them on building and evolving systems in a competitive software market. In a well designed software system, units of related functionality are organized into modules and classes, which are in turn arranged into inheritance trees, package hierarchies, components, libraries, frameworks, and services. The trade-offs between simplicity versus flexibility and power are carefully considered, and interfaces are designed that expose the key functional properties of a component while hiding much of the complexity of the implementation details. However, over time the design integrity of a well-engineered system tends to decay as new features are added, as new quality attributes are emphasized, and as old architectural knowledge is lost when experienced development personnel shift to new jobs. Consequently, as developers and as users we often find ourselves looking at a piece of functionality or other design artifact and wondering, "Why is this here?" That is, we would like to examine the provenance of an artifact to understand its history and why it is where it is within the current design of the system. In this brief paper, we sketch some of the dimensions of the broad problem of extracting and reasoning about the provenance of software development artifacts. As a motivating example, we also describe some recent related work that uses hashing to quickly and accurately identify version information of embedded Java libraries. We motivate the need to model for software artifact provenance.We sketch the dimensions of the problem space, and discuss its analysis.We describe an example, using hashing to identify library version information.  A software product is often dependent on a large number of third-party components.To assess potential risks, such as security vulnerabilities and license violations, a list of components and their versions in a product is important for release engineers and security analysts.Since such a list is not always available, a code comparison technique named Software Bertillonage has been proposed to test whether a product likely includes a copy of a particular component or not.Although the technique can extract candidates of reused components, a user still has to manually identify the original components among the candidates.In this paper, we propose a method to automatically select the most likely origin of components reused in a product, based on an assumption that a product tends to include an entire copy of a component rather than a partial copy.More concretely, given a Java product and a repository of jar files of existing components, our method selects jar files that can provide Java classes to the product in a greedy manner.To compare the method with the existing technique, we have conducted an evaluation using randomly created jar files including up to 1,000 components.The Software Bertillonage technique reports many candidates; the precision and recall are 0.357 and 0.993, respectively.Our method reports a list of original components whose precision and recall are 0.998 and 0.997.  The tens of thousands of high-quality open source software projects on the Internet raise the exciting possibility of studying software development by finding patterns across truly large source code repositories. This could enable new tools for developing code, encouraging reuse, and navigating large projects. In this paper, we build the first giga-token probabilistic language model of source code, based on 352 million lines of Java. This is 100 times the scale of the pioneering work by The giga-token model is significantly better at the code suggestion task than previous models. More broadly, our approach provides a new “lens” for analyzing software projects, enabling new complexity metrics based on statistical analysis of large corpora. We call these metrics data-driven complexity metrics. We propose new metrics that measure the complexity of a code module and the topical centrality of a module to a software project. In particular, it is possible to distinguish reusable utility classes from classes that are part of a program's core logic based solely on general information theoretic criteria. The invention relates to a computer device for managing documents, in particular software projects developed in co-operation. A memory stores contents of documents having time references. An extractor separates the document contents into document elements. A signature generator returns signatures of element contents. An imprint generator associates for each document the time reference thereof and the signatures of its elements. A time clock module calculates the results of a time election function. A supervisor operates the generator on the document contents. For each signature value, it calls the time clock module with a list of imprint time references containing the signature value. A unified time reference is obtained for each signature value. Each signature value and its unified time reference are stored.