We study the evolution of the largest known corpus of publicly available source code, i.e., the Software Heritage archive (4B unique source code files, 1B commits capturing their development histories across 50M software projects). On such corpus we quantify the growth rate of original, never-seen-before source code files and commits. We find the growth rates to be exponential over a period of more than 40 years. We then estimate the multiplication factor, i.e., how much the same artifacts (e.g., files or commits) appear in different contexts (e.g., commits or source code distribution places). We observe a combinatorial explosion in the multiplication of identical source code files across different commits. We discuss the implication of these findings for the problem of tracking the provenance of source code artifacts (e.g., where and when a given source code file or commit has been observed in the wild) for the entire body of publicly available source code. To that end we benchmark different data models for capturing software provenance information at this scale and growth rate. We identify a viable solution that is deployable on commodity hardware and appears to be maintainable for the foreseeable future.
While Mining Software Research (MSR) research @cite_1 is thriving, realizing large-scale empirical studies on software growth remains a challenging undertaking depending on complex tasks such as collecting massive amounts of source code @cite_16 and building suitable platforms for analyzing them @cite_33 @cite_39 . Hence, up to now, most studies have resorted to selecting relatively small subsets Some studies have analyzed up to a few million projects, but this is still a tiny fraction of all public source code. of the full corpus, using different criteria, and introducing biases that are difficult to estimate. For instance, an analysis of the growth of the Debian distribution spanning two decades has been performed in @cite_38 , observing initial superlinear growth of both the number of packages and their size. But Debian is a collection maintained by humans, so the number of packages in it depends on the effort that the Debian community can consent.
@cite_38 @cite_33 @cite_1 @cite_39 @cite_16
2055253889 2899407111 2039432697 1989479444
Mining Software Repositories (MSR) is an applied and practise-oriented field aimed at solving real problems encountered by practitioners and bringing value to Industry. We believe that empirical studies on both Open Source Software (OSS) and Closed or Proprietary Source (CSS PSS) is required in MSR research to increase generalizability or transferability of findings and reduce external (or threats) validity concerns. Furthermore, we believe that a collaboration between University and Industry is must or important in achieving the stated goals and agenda of MSR research (such as deployment and technology transfer). We analyse past five years of research papers published in MSR series of conferences (2010-2014) and count the number of studies using solely OSS data or solely CSS data or both OSS and CSS data. We also count the number of papers published by authors solely from Universities, solely from Industry and from both University and Industry. We present our findings which indicate lack of University-Industry collaboration (measured using co-authorship in scientific publications) and paucity of empirical studies on CSS PSS data. Our analysis reveals that out of 187 studies over a period of 5 years, 90:9 studies are conducted solely on OSS dataset. We present our findings which indicate that only 14:43 of the studies involve a University-Industry collaboration. Software repositories contain historical and valuable information about the overall development of software systems. Mining software repositories (MSR) is nowadays considered one of the most interesting growing fields within software engineering. MSR focuses on extracting and analyzing data available in software repositories to uncover interesting, useful, and actionable information about the system. Even though MSR plays an important role in software engineering research, few tools have been created and made public to support developers in extracting information from Git repository. In this paper, we present PyDriller, a Python Framework that eases the process of mining Git. We compare our tool against the state-of-the-art Python Framework GitPython, demonstrating that PyDriller can achieve the same results with, on average, 50 less LOC and significantly lower complexity. URL: https: github.com ishepard pydriller Materials: https: doi.org 10.5281 zenodo.1327363 Pre-print: https: doi.org 10.5281 zenodo.1327411 In the last decade, a variety of studies on mining software repositories has been conducted. Mining repositories has a potential to obtain useful knowledge for the future development and maintenance. When software repositories are mined, large commits in them are often excluded from mining targets because large commits include merging and we believe that large commits include peripheral modifications, which may affect negative impacts on mining code repositories. However, if large commits include code modifications, excluding large commits loses such modifications unintentionally. Moreover, such data cleansing assumes that there are no peripheral modifications in small commits. In this paper, we investigate how much peripheral modifications are included in commits in code repositories. As a result, we found that excluding large commits is insufficient to remove hindrances in commits for mining code repositories. In today's software-centric world, ultra-large-scale software repositories, e.g. SourceForge (350,000+ projects), GitHub (250,000+ projects), and Google Code (250,000+ projects) are the new library of Alexandria. They contain an enormous corpus of software and information about software. Scientists and engineers alike are interested in analyzing this wealth of information both for curiosity as well as for testing important hypotheses. However, systematic extraction of relevant data from these repositories and analysis of such data for testing hypotheses is hard, and best left for mining software repository (MSR) experts! The goal of Boa, a domain-specific language and infrastructure described here, is to ease testing MSR-related hypotheses. We have implemented Boa and provide a web-based interface to Boa's infrastructure. Our evaluation demonstrates that Boa substantially reduces programming efforts, thus lowering the barrier to entry. We also see drastic improvements in scalability. Last but not least, reproducing an experiment conducted using Boa is just a matter of re-running small Boa programs provided by previous researchers.