<commit_before>---
author: vadim
date: 2018-04-17
title: "Detecting licenses in code with Go and ML"
image: /post/gld/wthpl.png
description: "Detecting the license of an open source projects is harder than it seems. We have created go-license-detector, a Go library and command line application to solve that task."
categories: ["science", "technical"]
---

While working on [Public Git Archive](https://github.com/src-d/datasets/PublicGitArchive), we
thought that it would be handy to include the license of each project in the index file, so that
people could easily filter "grey" repositories without a clear license. Besides, we were
curious about the licenses distribution. GitHub already detects licenses by leveraging
the Ruby library [benbalter/licensee](https://github.com/benbalter/licensee) and the easy solution was
to query the GitHub API. However, we were not satisfied with
its detection quality: many projects which actually contain the license file in a non-standard
format are missed, and some are misclassified. This is how
[go-license-detector](https://github.com/src-d/go-license-detector) was born.

The goals were defined from the very beginning:

1. Favor false positives over false negatives (target data mining instead of compliance).
2. Perform fast.
3. Detect as many licenses as possible on the [hand-collected and hand-checked dataset of 1,000 top-starred repositories
on GitHub](https://github.com/src-d/go-license-detector/blob/master/licensedb/dataset.zip).
4. Comply with SPDX [licenses list](https://github.com/spdx/license-list-data) and
[detection guidelines](https://spdx.org/spdx-license-list/matching-guidelines).

(1) means that we should rather label a project with a slightly inaccurate license than miss its
license completely. The open source compliance departments will not be satisfied with this choice,
as they need the opposite: the missed projects are manually studied. (2) restricts from using a
scripting language such as Python or Ruby, and we chose Go for our implementation. (3) leads
to technical tricks, hacks and heuristics which result in complex code.
(4) is the only way to obtain the database of 400 different licenses validated by professional
lawyers.

The following table compares the current go-license-detector with GitHub's built-in license detector,
Google's licenseclassifier and <NAME>'s `lc` on the
[reference 1k dataset](https://github.com/src-d/go-license-detector/blob/master/licensedb/dataset.zip):

|Detector|Detection rate|Time to scan, sec|
|:-------|:----------------------------------------:|:-----------------------------------------|
|[go-license-detector](https://github.com/src-d/go-license-detector)| 99% \\(\\quad(\\frac{897}{902})\\) | 16 |
|[benbalter/licensee](https://github.com/benbalter/licensee)| 75% \\(\\quad(\\frac{673}{902})\\) | 111 |
|[google/licenseclassifier](https://github.com/google/licenseclassifier)| 76% \\(\\quad(\\frac{682}{902})\\) | 907 |
|[boyter/lc](https://github.com/boyter/lc)| 88% \\(\\quad(\\frac{797}{902})\\) | 548 |

The total number of repositories in the dataset is 958, however, only 902 contain any pointer to
the license - we looked through each of them. The rest are mainly "awesome lists" and Chinese projects
and translations of the western books. The latter are specific to Chinese developer community
with ~absent~ different licensing policies. We filed issues for the maintainers to clarify the license
[\[1\]](https://github.com/DrkSephy/es6-cheatsheet/issues/90)
[\[2\]](https://github.com/kdn251/interviews/issues/63)
[\[3\]](https://github.com/markerikson/react-redux-links/issues/87)
[\[4\]](https://github.com/CodeHubApp/CodeHub/issues/441)
[\[5\]](https://github.com/h4cc/awesome-elixir/issues/4400)
[\[6\]](https://github.com/fffaraz/awesome-cpp/issues/449).
We also encountered two licenses which were not included into SPDX and reported them:
[\[1\]](https://github.com/spdx/license-list-XML/issues/611)
[\[2\]](https://github.com/spdx/license-list-XML/issues/612).

#### How we measured time

Hardware: Intel Core i7-7500U (2x2 threads), 2x8GB LPDDR3@1867MHz.

```bash
$ cd $(go env GOPATH)/src/gopkg.in/src-d/go-license-detector.v2/licensedb
$ mkdir dataset && cd dataset
$ unzip ../dataset.zip
$ # src-d/go-license-detector
$ time license-detector * \<commit_msg>Update the gld time<commit_after>---
author: vadim
date: 2018-04-17
title: "Detecting licenses in code with Go and ML"
image: /post/gld/wthpl.png
description: "Detecting the license of an open source projects is harder than it seems. We have created go-license-detector, a Go library and command line application to solve that task."
categories: ["science", "technical"]
---

While working on [Public Git Archive](https://github.com/src-d/datasets/PublicGitArchive), we
thought that it would be handy to include the license of each project in the index file, so that
people could easily filter "grey" repositories without a clear license. Besides, we were
curious about the licenses distribution. GitHub already detects licenses by leveraging
the Ruby library [benbalter/licensee](https://github.com/benbalter/licensee) and the easy solution was
to query the GitHub API. However, we were not satisfied with
its detection quality: many projects which actually contain the license file in a non-standard
format are missed, and some are misclassified. This is how
[go-license-detector](https://github.com/src-d/go-license-detector) was born.

The goals were defined from the very beginning:

1. Favor false positives over false negatives (target data mining instead of compliance).
2. Perform fast.
3. Detect as many licenses as possible on the [hand-collected and hand-checked dataset of 1,000 top-starred repositories
on GitHub](https://github.com/src-d/go-license-detector/blob/master/licensedb/dataset.zip).
4. Comply with SPDX [licenses list](https://github.com/spdx/license-list-data) and
[detection guidelines](https://spdx.org/spdx-license-list/matching-guidelines).

(1) means that we should rather label a project with a slightly inaccurate license than miss its
license completely. The open source compliance departments will not be satisfied with this choice,
as they need the opposite: the missed projects are manually studied. (2) restricts from using a
scripting language such as Python or Ruby, and we chose Go for our implementation. (3) leads
to technical tricks, hacks and heuristics which result in complex code.
(4) is the only way to obtain the database of 400 different licenses validated by professional
lawyers.

The following table compares the current go-license-detector with GitHub's built-in license detector,
Google's licenseclassifier and <NAME>'s `lc` on the
[reference 1k dataset](https://github.com/src-d/go-license-detector/blob/master/licensedb/dataset.zip):

|Detector|Detection rate|Time to scan, sec|
|:-------|:----------------------------------------:|:-----------------------------------------|
|[go-license-detector](https://github.com/src-d/go-license-detector)| 99% \\(\\quad(\\frac{897}{902})\\) | 13.5 |
|[benbalter/licensee](https://github.com/benbalter/licensee)| 75% \\(\\quad(\\frac{673}{902})\\) | 111 |
|[google/licenseclassifier](https://github.com/google/licenseclassifier)| 76% \\(\\quad(\\frac{682}{902})\\) | 907 |
|[boyter/lc](https://github.com/boyter/lc)| 88% \\(\\quad(\\frac{797}{902})\\) | 548 |

The total number of repositories in the dataset is 958, however, only 902 contain any pointer to
the license - we looked through each of them. The rest are mainly "awesome lists" and Chinese projects
and translations of the western books. The latter are specific to Chinese developer community
with ~absent~ different licensing policies. We filed issues for the maintainers to clarify the license
[\[1\]](https://github.com/DrkSephy/es6-cheatsheet/issues/90)
[\[2\]](https://github.com/kdn251/interviews/issues/63)
[\[3\]](https://github.com/markerikson/react-redux-links/issues/87)
[\[4\]](https://github.com/CodeHubApp/CodeHub/issues/441)
[\[5\]](https://github.com/h4cc/awesome-elixir/issues/4400)
[\[6\]](https://github.com/fffaraz/awesome-cpp/issues/449).
We also encountered two licenses which were not included into SPDX and reported them:
[\[1\]](https://github.com/spdx/license-list-XML/issues/611)
[\[2\]](https://github.com/spdx/license-list-XML/issues/612).

#### How we measured time

Hardware: Intel Core i7-7500U (2x2 threads), 2x8GB LPDDR3@1867MHz.

```bash
$ cd $(go env GOPATH)/src/gopkg.in/src-d/go-license-detector.v2/licensedb
$ mkdir dataset && cd dataset
$ unzip ../dataset.zip
$ # src-d/go-license-detector
$ time license-detector * \
{'fraction_non_alphanumeric': 0.14456294922327845, 'fraction_numerical': 0.028634361233480177, 'mean_word_length': 4.1566049013747755, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 38, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '190078', 'n_tokens_mistral': 2727, 'n_tokens_neox': 2483, 'n_words': 887}