<commit_before># GitAnalyzer

**GitAnalyzer** is a tool finding the code, design and project management issues by using the behavioral analysis metrics on Git repository. From practice, project managers love the GitAnalyzer at first sight.

![ui-view](/screenshots/ui-view.png)

This tool is inspired by the book of "[Your Code as a Crime Scene](https://pragprog.com/book/atcrime/your-code-as-a-crime-scene)" by <NAME>. This book has great proposals of behavioral analysis on version control repositories, and GitAnalyzer takes the best of them (little bit rethought to don't rely on actual code). The main idea permeating this tool is that you don't need to spend hours doing static code analysis. You have all the information in your Git repository. Just look how your team behaves with each its part.

Reason to build GitAnalyzer is making a repository analysis tool with convenient user interface and easy to change.

## Non-functional features

- Works with several Git repositories and merging the statistics from all of them.
- Supports multi-team and multi-location projects
- Supports incremental updates to support up-to-date information without any manual actions.
- Caches the statistics in memory within uptime. Caches statistics to disk to decrease network communications in startup.

### Basic concepts

- **Change**. In GitAnalyzer the change is an atomic unit of measurement. Change here is a commit made to the single file by the single author within the single task. Change is a fractional value: e.g. if 4 commits made in same task for same file by the same author each commit will represent 0.25 part of this change. So amount of changes can be fractional.
- **Task**. Some commits are just continuous commits related to same task. Weight of such commits should be distributed across themselves. Currently we can only keep track of tasks using the comments of commits. You specify the prefix of the JIRA by special property, and system will group all the commits related to the same JIRA number.
- **Files and modules**. File is a source file in file system. Module is a root directory for source subtree acting as a separate unit. GitAnalyzer finds modules by existance of the build descriptor in a root directory of module, or by existance of sub-directory "src."
- **Author**. Author is an actual author in Git, not a committer. Author is identified by couple Name-E-mail. Author should represent a single person. When you have a situation when some person has several names/e-mails, You can unite them all via the property file by declaring them as aliases. All contribution made from these aliases will be grouped to the single author then.
- **Filtering**. Some commits are undesirable in metrics. These are: service commits, commits made by scripts, merge commits and commits generated by git-rebase. Build descriptors are also undesirable in some metrics (e.g. dependency graph). These files and commits are filtered out and don't appear in results.

### Application modes

GitAnalyzer can be run as either a server or a standalone client. There are 3 startup modes for it:
1. **Server mode**. It starts a Java process on the server machine and provides an HTTP interface for thin client. GitAnalyzer performs the incremental updates in background to make the data always up-to-date. To run application in server mode use command-line parameter ```-server```.
2. **Client mode**. It starts a Java process with reduced server as well as a client browser. When user closes the browser the server shuts down. Statistics gathering performs once. To run application in client mode use command-line parameter ```-client``` or without parameters. This is a default mode.
3. **Installation mode**. It starts in command-line mode, installs the application to directory you specified, and gives up. To install the application use command-line parameter ```-install```.

## Collected metrics

*All examples are made on base of Hibernate repository as of Dec 04, 2016.*

### Maintenance cost

Every file in repository is being changed intermittently. Some files are changed much more often than others. These changes are often caused by bad design, complexity, tight coupling and other undesired factors. In ideal world we should have equal amount of changes distributed across the whole codebase.

Every change comes not for free for companies. They pay developers making these changes, QA testing regression, support, etc.. So, these undesired changes often take more than 80% of the project's cost.

In this metric, we calculate the amount of changes per file, then order them by descending, and draw a plot.

![lc-files-chart](/screenshots/lc-files-chart.png)

Integral value: area below the plot is the overall changes' cost in money. This area for top 5% files is the same as for all remaining files. And reducing this value can significantly reduce the maintenance cost of project.

A table below the plot shows the files to be addressed.

![lc-files-table](/screenshots/lc-files-table.png)

Table is ranged by changes percent. First column shows the absolute amount of changes, second is relative amount (%). Last two columns open the pop-up showing the authors and teams mostly contributing to this file. Red rows mark the files to address. They have ten times bigger amount than ideal. If Amount is above the ideal, the row is amber. Green rows are below the ideal.

This page also shows this metric on module basis.

![lc-modules-chart](/screenshots/lc-modules-chart.png)

![lc-modules-table](/screenshots/lc-modules-table.png)

### Merge bottlenecks

This metric shows files and modules changed by unique authors. The more autors some file (or module) has the more conflicts it produces at merge. Such files and modules are merge bottlenecks in project. Common advice here: introducing code ownership to codebase; looking where SRP principle is violated.

The first bar chart shows the top 30 modules by count of unique authors. Highest bars are merge bottlenecks to be addressed.

![mb-modules-chart](/screenshots/mb-modules-chart.png)

Tables below show the modules and files ordered by by count of unique authors.

![mb-modules-table](/screenshots/mb-modules-table.png)

![mb-files-table](/screenshots/mb-files-table.png)

Red rows are files (modules) with count > 100. Amber: > 20. Last two columns open the pop-up showing the authors and teams mostly contributing to the given file (module).

### Stability analysis

This is a temporal metric based on the first and last changes in files. Usually the file or module is being changed first 3 months. And then when application architecture stabilizes, it's kept untouched for years. But some files/modules are changing within all their lifetime. This is an indicator that architecture is unstable in these places. There could be another cause of that: some files/modules are changed often because of multiple change requests to corresponding functionality. This could be a management issue, or just a false positive result.

Scatter plot ranges all the modules by first and last commit. First change here is a first commit. Last change is a recent commit. X axis is a time between first and last commit. Y axis is a time sinse the last commit. We usually don't want to see X more than Y. So, the red zone shows all "unstable" modules. The farther point from the line y=x, the less stable the module's architecture. The hottest are in bottom-right corner.

![stability-modules-chart](/screenshots/stability-modules-chart.png)

Table below shows the detailed age information.

![stability-modules-table](/screenshots/stability-modules-table.png)

Age is a time from first change until now. Time from last change is self-descriptive.<br>
(*Stability %*) is (*Time from last change*) / (*Age*) * 100%.<br>
Red rows indicate *Stability* less than 20%. Amder: between 20% and 50%.

The same metrics is collected on the file basis.

![stability-files-table](/screenshots/stability-files-table.png)

### Activity analysis

This temporal metric shows the authors', teams' and whole project's activity on monthly basis. This metric can narrate many interesting stories above the project. For example, if some team has instantly decreasing line this could mean that team has either low motivation or some other factors preventing normal contribution. If the plot has waves that can mean that work is doing by raids: silent periods alternate with periods of rush. Sometimes contribution grows or falls significantly at a time, and it's interesting to find a cause of that.

![a-all](/screenshots/a-all.png)

![a-team](/screenshots/a-team.png)

![a-author](/screenshots/a-author.png)

X axis is a timeline quantized by months. Y axis is amount of changes.

### Implicit dependencies

This metric shows the actual dependencies between files or modules. Some dependencies are explicitly set and expected by design. For example, your web application depends on some library providing authorization. But some dependences are just wondering. For example, if change in database causes changes in UI module, this dependency is implicit and undesirable. Finding such dependencies is a complex task. But finding and decoupling them gets benefits in decreasing complexity of code. Such dependencies can be found by analyzing which files (modules) are often being changed together (i.e appear in the same commit.) Then we can build a graph of such files (modules) and find clusters in this graph.

![id-modules-graph](/screenshots/id-modules-graph.png)

Nodes are modules; edges are connections. Every node and edge of this graph is weighted. Weight of edge is a count of commits where two modules appeared together. Weight of node is a sum of all connecting edges. The higher weight the thicker lines. Edges having weight less than 3 are filtered out. Nodes having no edges are filtered out.

![id-files-table](/screenshots/id-files-table.png)

In this table: *count of connections* is a count of commits where two files appeared together. Rows with number less than 3 are filtered out. Rows are ordered by count of connections from greatest to least. First 10% of rows are red. Next 40% are amber.

### Contributors performance

This metric shows absolute and relative amount of changes per on team bases, on location basis and for every author individually. It can be used for team performance analysis only if all the people are doing very similar and routine work. Why? Because precision of this metric is arguable. It shows just a number of changes, but doesn't take complesxity of these changes into account. Currently so many non-measurable factors affect performance that we can say we have no precise metrics for that. All of them are just empirical.

Using the filtering by mask you can look who is contributing more to different file types. For example, who is writing unit tests and who is not. You can also filter by module path and look who is mostly contributing to particular file or module.
Performance by locations shows two maps. First map shows the absolute and relative amount of changes per location. Second map shows the efficiency: average amount of changes per author.

![c-locations-map](/screenshots/c-locations-map.png)
![c-locations-map-eff](/screenshots/c-locations-map-eff.png)

Performance by teams shows two bar charts. First chart shows the absolute and relative amount of changes per team. Second map shows the efficiency: average amount of changes per author.

![c-teams-chart](/screenshots/c-teams-chart.png)
![c-teams-chart-eff](/screenshots/c-teams-chart-eff.png)

Teams performance is also detailed in table.

![c-teams-table](/screenshots/c-teams-table.png)

In this table: *Count of changes* is absolute amount, *Summary contribution* is relative amount, and *Changes per member* is efficiency.

The last table shows the individual performance in absolute and relative numbers.

![c-authors-table](/screenshots/c-authors-table.png)

### Top contributors of the file/module

These tables show which teams and authors are mostly contributing to the module. It allows you find who is responsible for that file/module. Or who has bigger expertise of this file/module. This is useful when you want to get an advice or assistance, or assign some work to appropriate person.

![popup-topauthors-file](popup-topauthors-file.png)
![popup-topauthors-module](popup-topauthors-module.png)
![popup-topteams-file](popup-topteams-file.png)
![popup-topteams-module](popup-topteams-module.png)

These tables can be opened from links "*Authors*" and "*Teams*" wherever you find them in other tables.

### Top contribution of the author/team

These tables show which files and modules are mostly changed by the given team/author. In other words, who is the actual owner of particular code.

![popup-topfiles-author](popup-topfiles-author.png)
![popup-topfiles-team](popup-topfiles-team.png)
![popup-topmodules-author](popup-topmodules-author.png)
![popup-topmodules-team](popup-topmodules-team.png)

These tables can be opened from links "*Files*" and "*Modules*" wherever you find them in other tables.

## Additional features

Date and file path filtering

![form-filter](form-filter.png)

Search and export to CSV in tables

![form-search-export](form-search-export.png)


# Installation and configuration

## Prerequisites
- **Java 1.8** or higher should be installed.
- **Git** should be installed.<commit_msg>MD fix<commit_after># GitAnalyzer

**GitAnalyzer** is a tool finding the code, design and project management issues by using the behavioral analysis metrics on Git repository. From practice, project managers love the GitAnalyzer at first sight.

![ui-view](/screenshots/ui-view.png)

This tool is inspired by the book of "[Your Code as a Crime Scene](https://pragprog.com/book/atcrime/your-code-as-a-crime-scene)" by <NAME>. This book has great proposals of behavioral analysis on version control repositories, and GitAnalyzer takes the best of them (little bit rethought to don't rely on actual code). The main idea permeating this tool is that you don't need to spend hours doing static code analysis. You have all the information in your Git repository. Just look how your team behaves with each its part.

Reason to build GitAnalyzer is making a repository analysis tool with convenient user interface and easy to change.

## Non-functional features

- Works with several Git repositories and merging the statistics from all of them.
- Supports multi-team and multi-location projects
- Supports incremental updates to support up-to-date information without any manual actions.
- Caches the statistics in memory within uptime. Caches statistics to disk to decrease network communications in startup.

### Basic concepts

- **Change**. In GitAnalyzer the change is an atomic unit of measurement. Change here is a commit made to the single file by the single author within the single task. Change is a fractional value: e.g. if 4 commits made in same task for same file by the same author each commit will represent 0.25 part of this change. So amount of changes can be fractional.
- **Task**. Some commits are just continuous commits related to same task. Weight of such commits should be distributed across themselves. Currently we can only keep track of tasks using the comments of commits. You specify the prefix of the JIRA by special property, and system will group all the commits related to the same JIRA number.
- **Files and modules**. File is a source file in file system. Module is a root directory for source subtree acting as a separate unit. GitAnalyzer finds modules by existance of the build descriptor in a root directory of module, or by existance of sub-directory "src."
- **Author**. Author is an actual author in Git, not a committer. Author is identified by couple Name-E-mail. Author should represent a single person. When you have a situation when some person has several names/e-mails, You can unite them all via the property file by declaring them as aliases. All contribution made from these aliases will be grouped to the single author then.
- **Filtering**. Some commits are undesirable in metrics. These are: service commits, commits made by scripts, merge commits and commits generated by git-rebase. Build descriptors are also undesirable in some metrics (e.g. dependency graph). These files and commits are filtered out and don't appear in results.

### Application modes

GitAnalyzer can be run as either a server or a standalone client. There are 3 startup modes for it:
- **Server mode**. It starts a Java process on the server machine and provides an HTTP interface for thin client. GitAnalyzer performs the incremental updates in background to make the data always up-to-date. To run application in server mode use command-line parameter ```-server```.
- **Client mode**. It starts a Java process with reduced server as well as a client browser. When user closes the browser the server shuts down. Statistics gathering performs once. To run application in client mode use command-line parameter ```-client``` or without parameters. This is a default mode.
- **Installation mode**. It starts in command-line mode, installs the application to directory you specified, and gives up. To install the application use command-line parameter ```-install```.

## Collected metrics

*All examples are made on base of Hibernate repository as of Dec 04, 2016.*

### Maintenance cost

Every file in repository is being changed intermittently. Some files are changed much more often than others. These changes are often caused by bad design, complexity, tight coupling and other undesired factors. In ideal world we should have equal amount of changes distributed across the whole codebase.

Every change comes not for free for companies. They pay developers making these changes, QA testing regression, support, etc.. So, these undesired changes often take more than 80% of the project's cost.

In this metric, we calculate the amount of changes per file, then order them by descending, and draw a plot.

![lc-files-chart](/screenshots/lc-files-chart.png)

Integral value: area below the plot is the overall changes' cost in money. This area for top 5% files is the same as for all remaining files. And reducing this value can significantly reduce the maintenance cost of project.

A table below the plot shows the files to be addressed.

![lc-files-table](/screenshots/lc-files-table.png)

Table is ranged by changes percent. First column shows the absolute amount of changes, second is relative amount (%). Last two columns open the pop-up showing the authors and teams mostly contributing to this file. Red rows mark the files to address. They have ten times bigger amount than ideal. If Amount is above the ideal, the row is amber. Green rows are below the ideal.

This page also shows this metric on module basis.

![lc-modules-chart](/screenshots/lc-modules-chart.png)

![lc-modules-table](/screenshots/lc-modules-table.png)

### Merge bottlenecks

This metric shows files and modules changed by unique authors. The more autors some file (or module) has the more conflicts it produces at merge. Such files and modules are merge bottlenecks in project. Common advice here: introducing code ownership to codebase; looking where SRP principle is violated.

The first bar chart shows the top 30 modules by count of unique authors. Highest bars are merge bottlenecks to be addressed.

![mb-modules-chart](/screenshots/mb-modules-chart.png)

Tables below show the modules and files ordered by by count of unique authors.

![mb-modules-table](/screenshots/mb-modules-table.png)

![mb-files-table](/screenshots/mb-files-table.png)

Red rows are files (modules) with count > 100. Amber: > 20. Last two columns open the pop-up showing the authors and teams mostly contributing to the given file (module).

### Stability analysis

This is a temporal metric based on the first and last changes in files. Usually the file or module is being changed first 3 months. And then when application architecture stabilizes, it's kept untouched for years. But some files/modules are changing within all their lifetime. This is an indicator that architecture is unstable in these places. There could be another cause of that: some files/modules are changed often because of multiple change requests to corresponding functionality. This could be a management issue, or just a false positive result.

Scatter plot ranges all the modules by first and last commit. First change here is a first commit. Last change is a recent commit. X axis is a time between first and last commit. Y axis is a time sinse the last commit. We usually don't want to see X more than Y. So, the red zone shows all "unstable" modules. The farther point from the line y=x, the less stable the module's architecture. The hottest are in bottom-right corner.

![stability-modules-chart](/screenshots/stability-modules-chart.png)

Table below shows the detailed age information.

![stability-modules-table](/screenshots/stability-modules-table.png)

Age is a time from first change until now. Time from last change is self-descriptive.<br>
(*Stability %*) is (*Time from last change*) / (*Age*) * 100%.<br>
Red rows indicate *Stability* less than 20%. Amder: between 20% and 50%.

The same metrics is collected on the file basis.

![stability-files-table](/screenshots/stability-files-table.png)

### Activity analysis

This temporal metric shows the authors', teams' and whole project's activity on monthly basis. This metric can narrate many interesting stories above the project. For example, if some team has instantly decreasing line this could mean that team has either low motivation or some other factors preventing normal contribution. If the plot has waves that can mean that work is doing by raids: silent periods alternate with periods of rush. Sometimes contribution grows or falls significantly at a time, and it's interesting to find a cause of that.

![a-all](/screenshots/a-all.png)

![a-team](/screenshots/a-team.png)

![a-author](/screenshots/a-author.png)

X axis is a timeline quantized by months. Y axis is amount of changes.

### Implicit dependencies

This metric shows the actual dependencies between files or modules. Some dependencies are explicitly set and expected by design. For example, your web application depends on some library providing authorization. But some dependences are just wondering. For example, if change in database causes changes in UI module, this dependency is implicit and undesirable. Finding such dependencies is a complex task. But finding and decoupling them gets benefits in decreasing complexity of code. Such dependencies can be found by analyzing which files (modules) are often being changed together (i.e appear in the same commit.) Then we can build a graph of such files (modules) and find clusters in this graph.

![id-modules-graph](/screenshots/id-modules-graph.png)

Nodes are modules; edges are connections. Every node and edge of this graph is weighted. Weight of edge is a count of commits where two modules appeared together. Weight of node is a sum of all connecting edges. The higher weight the thicker lines. Edges having weight less than 3 are filtered out. Nodes having no edges are filtered out.

![id-files-table](/screenshots/id-files-table.png)

In this table: *count of connections* is a count of commits where two files appeared together. Rows with number less than 3 are filtered out. Rows are ordered by count of connections from greatest to least. First 10% of rows are red. Next 40% are amber.

### Contributors performance

This metric shows absolute and relative amount of changes per on team bases, on location basis and for every author individually. It can be used for team performance analysis only if all the people are doing very similar and routine work. Why? Because precision of this metric is arguable. It shows just a number of changes, but doesn't take complesxity of these changes into account. Currently so many non-measurable factors affect performance that we can say we have no precise metrics for that. All of them are just empirical.

Using the filtering by mask you can look who is contributing more to different file types. For example, who is writing unit tests and who is not. You can also filter by module path and look who is mostly contributing to particular file or module.
Performance by locations shows two maps. First map shows the absolute and relative amount of changes per location. Second map shows the efficiency: average amount of changes per author.

![c-locations-map](/screenshots/c-locations-map.png)

![c-locations-map-eff](/screenshots/c-locations-map-eff.png)

Performance by teams shows two bar charts. First chart shows the absolute and relative amount of changes per team. Second map shows the efficiency: average amount of changes per author.

![c-teams-chart](/screenshots/c-teams-chart.png)

![c-teams-chart-eff](/screenshots/c-teams-chart-eff.png)

Teams performance is also detailed in table.

![c-teams-table](/screenshots/c-teams-table.png)

In this table: *Count of changes* is absolute amount, *Summary contribution* is relative amount, and *Changes per member* is efficiency.

The last table shows the individual performance in absolute and relative numbers.

![c-authors-table](/screenshots/c-authors-table.png)

### Top contributors of the file/module

These tables show which teams and authors are mostly contributing to the module. It allows you find who is responsible for that file/module. Or who has bigger expertise of this file/module. This is useful when you want to get an advice or assistance, or assign some work to appropriate person.

![popup-topauthors-file](popup-topauthors-file.png)

![popup-topauthors-module](popup-topauthors-module.png)

![popup-topteams-file](popup-topteams-file.png)

![popup-topteams-module](popup-topteams-module.png)

These tables can be opened from links "*Authors*" and "*Teams*" wherever you find them in other tables.

### Top contribution of the author/team

These tables show which files and modules are mostly changed by the given team/author. In other words, who is the actual owner of particular code.

![popup-topfiles-author](popup-topfiles-author.png)

![popup-topfiles-team](popup-topfiles-team.png)

![popup-topmodules-author](popup-topmodules-author.png)

![popup-topmodules-team](popup-topmodules-team.png)

These tables can be opened from links "*Files*" and "*Modules*" wherever you find them in other tables.

## Additional features

Date and file path filtering

![form-filter](form-filter.png)

Search and export to CSV in tables

![form-search-export](form-search-export.png)


# Installation and configuration

## Prerequisites
- **Java 1.8** or higher should be installed.
- **Git** should be installed.
{'fraction_non_alphanumeric': 0.0633024783517468, 'fraction_numerical': 0.0030232905344879067, 'mean_word_length': 4.479141104294478, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 13, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7261005', 'n_tokens_mistral': 6742, 'n_tokens_neox': 6415, 'n_words': 3805}