<commit_before>![Gitlogg](https://github.com/dreamyguy/gitlogg/blob/master/docs/gitlogg-icon-github.png "Parse the 'git log' of one or several 'git' repositories into a sanitised and distributable 'JSON' file")

> _Parse the 'git log' of one or several 'git' repositories into a sanitised and distributable 'JSON' file._

## Why?

`git log` is a wonderful tool. However its output can be not only surprisingly inconsistent, but also long, difficult to scan and to distribute.

**Gitlogg** sanitises the `git log` and outputs it to `JSON`, a format that can easily be consumed by other applications. As long as the repositories being scanned are kept up to date, **Gitlogg** will return fresh data every time it runs.

#### **Gitlogg** addresses the following challenges:

* `git log` can only be used on a repository at a time.
* `git log` can't be easily consumed by other applications in its original format.
* `git log` doesn't return **impact**, which is the cumulative change brought by a single commit. Very interesting graphs can be built with that data, as shown on [sidhree.com][1].
* Fields that allow user input, like `subject`, need to be sanitised to be consumed.
* File changes shown under `--stat` or `--shortstat` are currently not available as placeholders under `--pretty=format:<string>`, and it is cumbersome to get commit logs to output neatly in single lines - with stats.
* It is hard to retrieve commits made on a specific but generic moment, like "11pm"; at the "27th minute" of an hour; on a "Sunday"; on "March"; on "GMT -5"; on the "53rd second of a minute".
* Some commits don't have stats, and that can cause the structure of the output to break, making it harder to distribute it.

#### Script execution feedback

**Gitlogg** is not a very complex application, but I still made an effort to provide some feedback on what is happening under the hood. Below are some screenshots of dialogs one can expect to see while executing it:

![Error 001](https://raw.githubusercontent.com/dreamyguy/gitlogg/master/docs/error-001.png "'Error 001' message as on release v0.1.3")
> **Øh nøes!** The path to the folder containing all repositories *does not exist!*

![Error 002](https://raw.githubusercontent.com/dreamyguy/gitlogg/master/docs/error-002.png "'Error 002' message as on release v0.1.3")
> **Øh nøes!** The path to the folder containing all repositories *exists, but is empty!*

![Success!](https://raw.githubusercontent.com/dreamyguy/gitlogg/master/docs/success.png "Success messages as on release v0.1.3")
> **Success!** `JSON` parsed, based on **9** different repositories with a total of **25,537** commits.

Note that I've included two huge repos (react & react-native, that have 7,813 & 10,065 commits respectively at the time of this writting) for the sake of demonstration. The resulting parsed `JSON` file has 715,040 lines. All that done in less than 25 seconds.

*I have successfully compiled **`470`** repositories at once* (all repos under the organization I work for). Then I got these specs:

* `gitlogg.tmp` generated in `154s` (`~2.57mins`)
* `JSON` output parsed in `2792ms`
* `JSON` file size: `121,5MB`
* Commits processed: `118,117`
* Parsed `JSON` file, lines: `3,307,280`

## Requirementss

[NodeJS][2] and [BabelJS][3].

1. Install `NodeJS` (visit [their page][2] to find the right install for your system).
2. Install `BabelJS` globally by running `npm install babel-cli -g`. One can also choose to install it locally by simply running `npm install babel-cli`, but in most cases it is smarter to install `-cli` packages globally.
3. Run `npm install` to install the remaining dependencies.

## The `JSON` output

The output will look like this (first commit for **Font Awesome**):

    {
      "commits": [
        {
          "repository": "Font-Awesome",
          "commit_nr": 1
          "commit_hash": "7ed221e28df1745a20009329033ac690ef000575",
          "author_name": "<NAME>",
          "author_email": "<EMAIL>",
          "author_date": "Fri Feb 17 09:27:26 2012 -0500",<commit_msg>Correct typo<commit_after>![Gitlogg](https://github.com/dreamyguy/gitlogg/blob/master/docs/gitlogg-icon-github.png "Parse the 'git log' of one or several 'git' repositories into a sanitised and distributable 'JSON' file")

> _Parse the 'git log' of one or several 'git' repositories into a sanitised and distributable 'JSON' file._

## Why?

`git log` is a wonderful tool. However its output can be not only surprisingly inconsistent, but also long, difficult to scan and to distribute.

**Gitlogg** sanitises the `git log` and outputs it to `JSON`, a format that can easily be consumed by other applications. As long as the repositories being scanned are kept up to date, **Gitlogg** will return fresh data every time it runs.

#### **Gitlogg** addresses the following challenges:

* `git log` can only be used on a repository at a time.
* `git log` can't be easily consumed by other applications in its original format.
* `git log` doesn't return **impact**, which is the cumulative change brought by a single commit. Very interesting graphs can be built with that data, as shown on [sidhree.com][1].
* Fields that allow user input, like `subject`, need to be sanitised to be consumed.
* File changes shown under `--stat` or `--shortstat` are currently not available as placeholders under `--pretty=format:<string>`, and it is cumbersome to get commit logs to output neatly in single lines - with stats.
* It is hard to retrieve commits made on a specific but generic moment, like "11pm"; at the "27th minute" of an hour; on a "Sunday"; on "March"; on "GMT -5"; on the "53rd second of a minute".
* Some commits don't have stats, and that can cause the structure of the output to break, making it harder to distribute it.

#### Script execution feedback

**Gitlogg** is not a very complex application, but I still made an effort to provide some feedback on what is happening under the hood. Below are some screenshots of dialogs one can expect to see while executing it:

![Error 001](https://raw.githubusercontent.com/dreamyguy/gitlogg/master/docs/error-001.png "'Error 001' message as on release v0.1.3")
> **Øh nøes!** The path to the folder containing all repositories *does not exist!*

![Error 002](https://raw.githubusercontent.com/dreamyguy/gitlogg/master/docs/error-002.png "'Error 002' message as on release v0.1.3")
> **Øh nøes!** The path to the folder containing all repositories *exists, but is empty!*

![Success!](https://raw.githubusercontent.com/dreamyguy/gitlogg/master/docs/success.png "Success messages as on release v0.1.3")
> **Success!** `JSON` parsed, based on **9** different repositories with a total of **25,537** commits.

Note that I've included two huge repos _(*react* & *react-native*, that have 7,813 & 10,065 commits respectively at the time of this writting)_ for the sake of demonstration. The resulting parsed `JSON` file has 715,040 lines. All that done in less than 25 seconds.

_I have successfully compiled **`470`** repositories at once_ (all repos under the organization I work for). Then I got these specs:

* `gitlogg.tmp` generated in `154s` (`~2.57mins`)
* `JSON` output parsed in `2792ms`
* `JSON` file size: `121,5MB`
* Commits processed: `118,117`
* Parsed `JSON` file, lines: `3,307,280`

## Requirementss

[NodeJS][2] and [BabelJS][3].

1. Install `NodeJS` (visit [their page][2] to find the right install for your system).
2. Install `BabelJS` globally by running `npm install babel-cli -g`. One can also choose to install it locally by simply running `npm install babel-cli`, but in most cases it is smarter to install `-cli` packages globally.
3. Run `npm install` to install the remaining dependencies.

## The `JSON` output

The output will look like this (first commit for **Font Awesome**):

    {
      "commits": [
        {
          "repository": "Font-Awesome",
          "commit_nr": 1
          "commit_hash": "7ed221e28df1745a20009329033ac690ef000575",
          "author_name": "<NAME>",
          "author_email": "<EMAIL>",
          "author_date": "Fri Feb 17 09:27:26 2012 -0500",
{'fraction_non_alphanumeric': 0.1061859488825781, 'fraction_numerical': 0.034819113470798864, 'mean_word_length': 4.259740259740259, 'pattern_counts': {'":': 14, '<': 9, '<?xml version=': 0, '>': 17, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1606566', 'n_tokens_mistral': 2597, 'n_tokens_neox': 2305, 'n_words': 1100}