This all is with respect to Microsoft project CodeBook:CodeBookThere is huge amount of code in the repository, many classes , a call hierarchy of functions, testcases etc. I am interested in knowing how this crawling process takes place, and how this data is sorted?
What is the basic process and tools needed for crawling a source code repository for the purpose of data mining?
version control;repository;data mining;crawlers
I know that there is some research in the area of social network analysis to large software projects with regards to both knowledge management as well as the identification of defects, but to the best of my knowledge, it's mostly academic at this point. I did find Supporting Collaborative Software Development through the Visualization of Socio-Technical Dependencies, a paper by researchers at the University of California, Irvine and Universidade Federal do Par which mentions an Eclipse plugin for such work. The plugin website also links to some other publications by this group.A professor at my university also has some academic interests in defects and software security through similar approaches. Before I graduated, he gave a presentation related to some of his work. I believe the presentation was based on this paper, titled Predicting Failures with Developer Networks and Social Network Analysis. It doesn't appear to be what you're looking for exactly, but there might be some interesting related work in network analysis based on source code repositories.I took a quick glance at the CodeBook project that you linked to. It appears that in order to get the level of detail that they are describing, you need some kind of knowledge about the organization of the project. That might be difficult with an open-source project, but I would expect that you can at least learn the basis of the analysis that they are doing.At the very least, perhaps these papers might give you some other documents, publications, or keywords that you can use to help get started. Some keywords to perhaps get you started: social network analysis, socio-technological dependencies, distributed software development, developer network.