task_categories:text-classification task_categories:zero-shot-classification task_categories:summarization task_categories:text2text-generation task_categories:sentence-similarity size_categories:1M<n<10M license:mit chemistry biology finance legal music art code climate medical region:us
---
license: mit
task_categories:
- text-classification
- zero-shot-classification
- summarization
- text2text-generation
- sentence-similarity
tags:
- chemistry
- biology
- finance
- legal
- music
- art
- code
- climate
- medical
pretty_name: One million commits
size_categories:
- 1M<n<10M
---

# One million commits

A large variety of git commits pulled from across GitHub.

Created by **William Entriken**, released **2023-09-26**, version **1**.

This composition is licensed under the **MIT license**.

## Intended use

This dataset could be used to train a model concerned with programming tasks:

1. Summarize some programming work
1. Perform work given a description of the work to do
1. Learn-by-example the syntax for all active programming languages and structured data formats

This dataset could be used to train a model concerned with scoring and trend analysis:

1. Identify trends in programming across years
1. Compare popularity and impact of programming topics/languages (using stars, quality as a signal)
1. Score programming work as good or bad (using authors' ratings as a signal)
1. Identify popular project file layouts for each programming language

Because this dataset comprises about a half terabyte of text, it may be useful for a training a general text processing model. Or at least a niche of such a model.

## Factors

An attempt was made at providing a large variety of repositories. To example the potential biases, here is a comparison of our dataset to sampling at random from the list of all commits across GitHub.

1. This dataset should overrepresent repositories with 7+ character words.
1. This dataset should overrepresent repositories with few commits (because only one commit is included per repository).
1. This dataset should overrepresent repositories from authors with many repositories.

## Metrics

* Commits: 1,000,000
* Distinct repositories: 1,000,000
* Distinct authors: xxx
* Bytes: 

## Data format

The files are distributed as SQLite databases with the schema:

```sql
CREATE TABLE commits (repo_full_name TEXT UNIQUE, repo JSON, `commit` JSON, commit_patch TEXT, author JSON);
```

Each repository is included at most once, which is why `repo_full_name` is a unique key. The `commit_patch` is the full `.patch` file for the commit. Other columns `repo`, `commit` and `author` are JSON blobs following the GitHub API schema:

<details>

<summary>repository</summary>

```json
{
  "title": "Repo Search Result Item",
  "description": "Repo Search Result Item",
  "type": "object",
  "properties": {
    "id": {
      "type": "integer"
    },
    "node_id": {
      "type": "string"
    },
    "name": {
      "type": "string"
    },
    "full_name": {
      "type": "string"
    },
    "owner": {
      "anyOf": [
        {
          "type": "null"
        },
        {
          "title": "Simple User",
          "description": "A GitHub user.",
          "type": "object",
          "properties": {
            "name": {
              "type": [
                "string",
                "null"
              ]
            },
            "email": {
              "type": [
                "string",
                "null"
              ]
            },
            "login": {
              "type": "string",
              "examples": [
                "octocat"
              ]
            },
            "id": {
              "type": "integer",
              "examples": [
                1
              ]
            },
            "node_id": {
              "type": "string",
              "examples": [
                "MDQ6VXNlcjE="
              ]
            },
            "avatar_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://github.com/images/error/octocat_happy.gif"
              ]
            },
            "gravatar_id": {
              "type": [
                "string",
                "null"
              ],
              "examples": [
                "41d064eb2195891e12d0413f63227ea7"
              ]
            },
            "url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat"
              ]
            },
            "html_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://github.com/octocat"
              ]
            },
            "followers_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/followers"
              ]
            },
            "following_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/following{/other_user}"
              ]
            },
            "gists_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/gists{/gist_id}"
              ]
            },
            "starred_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/starred{/owner}{/repo}"
              ]
            },
            "subscriptions_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/subscriptions"
              ]
            },
            "organizations_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/orgs"
              ]
            },
            "repos_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/repos"
              ]
            },
            "events_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/events{/privacy}"
              ]
            },
            "received_events_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/received_events"
              ]
            },
            "type": {
              "type": "string",
              "examples": [
                "User"
              ]
            },
            "site_admin": {
              "type": "boolean"
            },
            "starred_at": {
              "type": "string",
              "examples": [
                "\"2020-07-09T00:17:55Z\""
              ]
            }
          },
          "required": [
            "avatar_url",
            "events_url",
            "followers_url",
            "following_url",
            "gists_url",
            "gravatar_id",
            "html_url",
            "id",
            "node_id",
            "login",
            "organizations_url",
            "received_events_url",
            "repos_url",
            "site_admin",
            "starred_url",
            "subscriptions_url",
            "type",
            "url"
          ]
        }
      ]
    },
    "private": {
      "type": "boolean"
    },
    "html_url": {
      "type": "string",
      "format": "uri"
    },
    "description": {
      "type": [
        "string",
        "null"
      ]
    },
    "fork": {
      "type": "boolean"
    },
    "url": {
      "type": "string",
      "format": "uri"
    },
    "created_at": {
      "type": "string",
      "format": "date-time"
    },
    "updated_at": {
      "type": "string",
      "format": "date-time"
    },
    "pushed_at": {
      "type": "string",
      "format": "date-time"
    },
    "homepage": {
      "type": [
        "string",
        "null"
      ],
      "format": "uri"
    },
    "size": {
      "type": "integer"
    },
    "stargazers_count": {
      "type": "integer"
    },
    "watchers_count": {
      "type": "integer"
    },
    "language": {
      "type": [
        "string",
        "null"
      ]
    },
    "forks_count": {
      "type": "integer"
    },
    "open_issues_count": {
      "type": "integer"
    },
    "master_branch": {
      "type": "string"
    },
    "default_branch": {
      "type": "string"
    },
    "score": {
      "type": "number"
    },
    "forks_url": {
      "type": "string",
      "format": "uri"
    },
    "keys_url": {
      "type": "string"
    },
    "collaborators_url": {
      "type": "string"
    },
    "teams_url": {
      "type": "string",
      "format": "uri"
    },
    "hooks_url": {
      "type": "string",
      "format": "uri"
    },
    "issue_events_url": {
      "type": "string"
    },
    "events_url": {
      "type": "string",
      "format": "uri"
    },
    "assignees_url": {
      "type": "string"
    },
    "branches_url": {
      "type": "string"
    },
    "tags_url": {
      "type": "string",
      "format": "uri"
    },
    "blobs_url": {
      "type": "string"
    },
    "git_tags_url": {
      "type": "string"
    },
    "git_refs_url": {
      "type": "string"
    },
    "trees_url": {
      "type": "string"
    },
    "statuses_url": {
      "type": "string"
    },
    "languages_url": {
      "type": "string",
      "format": "uri"
    },
    "stargazers_url": {
      "type": "string",
      "format": "uri"
    },
    "contributors_url": {
      "type": "string",
      "format": "uri"
    },
    "subscribers_url": {
      "type": "string",
      "format": "uri"
    },
    "subscription_url": {
      "type": "string",
      "format": "uri"
    },
    "commits_url": {
      "type": "string"
    },
    "git_commits_url": {
      "type": "string"
    },
    "comments_url": {
      "type": "string"
    },
    "issue_comment_url": {
      "type": "string"
    },
    "contents_url": {
      "type": "string"
    },
    "compare_url": {
      "type": "string"
    },
    "merges_url": {
      "type": "string",
      "format": "uri"
    },
    "archive_url": {
      "type": "string"
    },
    "downloads_url": {
      "type": "string",
      "format": "uri"
    },
    "issues_url": {
      "type": "string"
    },
    "pulls_url": {
      "type": "string"
    },
    "milestones_url": {
      "type": "string"
    },
    "notifications_url": {
      "type": "string"
    },
    "labels_url": {
      "type": "string"
    },
    "releases_url": {
      "type": "string"
    },
    "deployments_url": {
      "type": "string",
      "format": "uri"
    },
    "git_url": {
      "type": "string"
    },
    "ssh_url": {
      "type": "string"
    },
    "clone_url": {
      "type": "string"
    },
    "svn_url": {
      "type": "string",
      "format": "uri"
    },
    "forks": {
      "type": "integer"
    },
    "open_issues": {
      "type": "integer"
    },
    "watchers": {
      "type": "integer"
    },
    "topics": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "mirror_url": {
      "type": [
        "string",
        "null"
      ],
      "format": "uri"
    },
    "has_issues": {
      "type": "boolean"
    },
    "has_projects": {
      "type": "boolean"
    },
    "has_pages": {
      "type": "boolean"
    },
    "has_wiki": {
      "type": "boolean"
    },
    "has_downloads": {
      "type": "boolean"
    },
    "has_discussions": {
      "type": "boolean"
    },
    "archived": {
      "type": "boolean"
    },
    "disabled": {
      "type": "boolean",
      "description": "Returns whether or not this repository disabled."
    },
    "visibility": {
      "description": "The repository visibility: public, private, or internal.",
      "type": "string"
    },
    "license": {
      "anyOf": [
        {
          "type": "null"
        },
        {
          "title": "License Simple",
          "description": "License Simple",
          "type": "object",
          "properties": {
            "key": {
              "type": "string",
              "examples": [
                "mit"
              ]
            },
            "name": {
              "type": "string",
              "examples": [
                "MIT License"
              ]
            },
            "url": {
              "type": [
                "string",
                "null"
              ],
              "format": "uri",
              "examples": [
                "https://api.github.com/licenses/mit"
              ]
            },
            "spdx_id": {
              "type": [
                "string",
                "null"
              ],
              "examples": [
                "MIT"
              ]
            },
            "node_id": {
              "type": "string",
              "examples": [
                "MDc6TGljZW5zZW1pdA=="
              ]
            },
            "html_url": {
              "type": "string",
              "format": "uri"
            }
          },
          "required": [
            "key",
            "name",
            "url",
            "spdx_id",
            "node_id"
          ]
        }
      ]
    },
    "permissions": {
      "type": "object",
      "properties": {
        "admin": {
          "type": "boolean"
        },
        "maintain": {
          "type": "boolean"
        },
        "push": {
          "type": "boolean"
        },
        "triage": {
          "type": "boolean"
        },
        "pull": {
          "type": "boolean"
        }
      },
      "required": [
        "admin",
        "pull",
        "push"
      ]
    },
    "text_matches": {
      "title": "Search Result Text Matches",
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "object_url": {
            "type": "string"
          },
          "object_type": {
            "type": [
              "string",
              "null"
            ]
          },
          "property": {
            "type": "string"
          },
          "fragment": {
            "type": "string"
          },
          "matches": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "text": {
                  "type": "string"
                },
                "indices": {
                  "type": "array",
                  "items": {
                    "type": "integer"
                  }
                }
              }
            }
          }
        }
      }
    },
    "temp_clone_token": {
      "type": "string"
    },
    "allow_merge_commit": {
      "type": "boolean"
    },
    "allow_squash_merge": {
      "type": "boolean"
    },
    "allow_rebase_merge": {
      "type": "boolean"
    },
    "allow_auto_merge": {
      "type": "boolean"
    },
    "delete_branch_on_merge": {
      "type": "boolean"
    },
    "allow_forking": {
      "type": "boolean"
    },
    "is_template": {
      "type": "boolean"
    },
    "web_commit_signoff_required": {
      "type": "boolean",
      "examples": [
        false
      ]
    }
  },
  "required": [
    "archive_url",
    "assignees_url",
    "blobs_url",
    "branches_url",
    "collaborators_url",
    "comments_url",
    "commits_url",
    "compare_url",
    "contents_url",
    "contributors_url",
    "deployments_url",
    "description",
    "downloads_url",
    "events_url",
    "fork",
    "forks_url",
    "full_name",
    "git_commits_url",
    "git_refs_url",
    "git_tags_url",
    "hooks_url",
    "html_url",
    "id",
    "node_id",
    "issue_comment_url",
    "issue_events_url",
    "issues_url",
    "keys_url",
    "labels_url",
    "languages_url",
    "merges_url",
    "milestones_url",
    "name",
    "notifications_url",
    "owner",
    "private",
    "pulls_url",
    "releases_url",
    "stargazers_url",
    "statuses_url",
    "subscribers_url",
    "subscription_url",
    "tags_url",
    "teams_url",
    "trees_url",
    "url",
    "clone_url",
    "default_branch",
    "forks",
    "forks_count",
    "git_url",
    "has_downloads",
    "has_issues",
    "has_projects",
    "has_wiki",
    "has_pages",
    "homepage",
    "language",
    "archived",
    "disabled",
    "mirror_url",
    "open_issues",
    "open_issues_count",
    "license",
    "pushed_at",
    "size",
    "ssh_url",
    "stargazers_count",
    "svn_url",
    "watchers",
    "watchers_count",
    "created_at",
    "updated_at",
    "score"
  ]
}
```

</details>

<details>

<summary>commit</summary>

```json
{
  "title": "Commit",
  "description": "Commit",
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "format": "uri",
      "examples": [
        "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e"
      ]
    },
    "sha": {
      "type": "string",
      "examples": [
        "6dcb09b5b57875f334f61aebed695e2e4193db5e"
      ]
    },
    "node_id": {
      "type": "string",
      "examples": [
        "MDY6Q29tbWl0NmRjYjA5YjViNTc4NzVmMzM0ZjYxYWViZWQ2OTVlMmU0MTkzZGI1ZQ=="
      ]
    },
    "html_url": {
      "type": "string",
      "format": "uri",
      "examples": [
        "https://github.com/octocat/Hello-World/commit/6dcb09b5b57875f334f61aebed695e2e4193db5e"
      ]
    },
    "comments_url": {
      "type": "string",
      "format": "uri",
      "examples": [
        "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e/comments"
      ]
    },
    "commit": {
      "type": "object",
      "properties": {
        "url": {
          "type": "string",
          "format": "uri",
          "examples": [
            "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e"
          ]
        },
        "author": {
          "anyOf": [
            {
              "type": "null"
            },
            {
              "title": "Git User",
              "description": "Metaproperties for Git author/committer information.",
              "type": "object",
              "properties": {
                "name": {
                  "type": "string",
                  "examples": [
                    "\"Chris Wanstrath\""
                  ]
                },
                "email": {
                  "type": "string",
                  "examples": [
                    "\"chris@ozmm.org\""
                  ]
                },
                "date": {
                  "type": "string",
                  "examples": [
                    "\"2007-10-29T02:42:39.000-07:00\""
                  ]
                }
              }
            }
          ]
        },
        "committer": {
          "anyOf": [
            {
              "type": "null"
            },
            {
              "title": "Git User",
              "description": "Metaproperties for Git author/committer information.",
              "type": "object",
              "properties": {
                "name": {
                  "type": "string",
                  "examples": [
                    "\"Chris Wanstrath\""
                  ]
                },
                "email": {
                  "type": "string",
                  "examples": [
                    "\"chris@ozmm.org\""
                  ]
                },
                "date": {
                  "type": "string",
                  "examples": [
                    "\"2007-10-29T02:42:39.000-07:00\""
                  ]
                }
              }
            }
          ]
        },
        "message": {
          "type": "string",
          "examples": [
            "Fix all the bugs"
          ]
        },
        "comment_count": {
          "type": "integer",
          "examples": [
            0
          ]
        },
        "tree": {
          "type": "object",
          "properties": {
            "sha": {
              "type": "string",
              "examples": [
                "827efc6d56897b048c772eb4087f854f46256132"
              ]
            },
            "url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/repos/octocat/Hello-World/tree/827efc6d56897b048c772eb4087f854f46256132"
              ]
            }
          },
          "required": [
            "sha",
            "url"
          ]
        },
        "verification": {
          "title": "Verification",
          "type": "object",
          "properties": {
            "verified": {
              "type": "boolean"
            },
            "reason": {
              "type": "string"
            },
            "payload": {
              "type": [
                "string",
                "null"
              ]
            },
            "signature": {
              "type": [
                "string",
                "null"
              ]
            }
          },
          "required": [
            "verified",
            "reason",
            "payload",
            "signature"
          ]
        }
      },
      "required": [
        "author",
        "committer",
        "comment_count",
        "message",
        "tree",
        "url"
      ]
    },
    "author": {
      "anyOf": [
        {
          "type": "null"
        },
        {
          "title": "Simple User",
          "description": "A GitHub user.",
          "type": "object",
          "properties": {
            "name": {
              "type": [
                "string",
                "null"
              ]
            },
            "email": {
              "type": [
                "string",
                "null"
              ]
            },
            "login": {
              "type": "string",
              "examples": [
                "octocat"
              ]
            },
            "id": {
              "type": "integer",
              "examples": [
                1
              ]
            },
            "node_id": {
              "type": "string",
              "examples": [
                "MDQ6VXNlcjE="
              ]
            },
            "avatar_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://github.com/images/error/octocat_happy.gif"
              ]
            },
            "gravatar_id": {
              "type": [
                "string",
                "null"
              ],
              "examples": [
                "41d064eb2195891e12d0413f63227ea7"
              ]
            },
            "url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat"
              ]
            },
            "html_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://github.com/octocat"
              ]
            },
            "followers_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/followers"
              ]
            },
            "following_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/following{/other_user}"
              ]
            },
            "gists_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/gists{/gist_id}"
              ]
            },
            "starred_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/starred{/owner}{/repo}"
              ]
            },
            "subscriptions_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/subscriptions"
              ]
            },
            "organizations_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/orgs"
              ]
            },
            "repos_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/repos"
              ]
            },
            "events_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/events{/privacy}"
              ]
            },
            "received_events_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/received_events"
              ]
            },
            "type": {
              "type": "string",
              "examples": [
                "User"
              ]
            },
            "site_admin": {
              "type": "boolean"
            },
            "starred_at": {
              "type": "string",
              "examples": [
                "\"2020-07-09T00:17:55Z\""
              ]
            }
          },
          "required": [
            "avatar_url",
            "events_url",
            "followers_url",
            "following_url",
            "gists_url",
            "gravatar_id",
            "html_url",
            "id",
            "node_id",
            "login",
            "organizations_url",
            "received_events_url",
            "repos_url",
            "site_admin",
            "starred_url",
            "subscriptions_url",
            "type",
            "url"
          ]
        }
      ]
    },
    "committer": {
      "anyOf": [
        {
          "type": "null"
        },
        {
          "title": "Simple User",
          "description": "A GitHub user.",
          "type": "object",
          "properties": {
            "name": {
              "type": [
                "string",
                "null"
              ]
            },
            "email": {
              "type": [
                "string",
                "null"
              ]
            },
            "login": {
              "type": "string",
              "examples": [
                "octocat"
              ]
            },
            "id": {
              "type": "integer",
              "examples": [
                1
              ]
            },
            "node_id": {
              "type": "string",
              "examples": [
                "MDQ6VXNlcjE="
              ]
            },
            "avatar_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://github.com/images/error/octocat_happy.gif"
              ]
            },
            "gravatar_id": {
              "type": [
                "string",
                "null"
              ],
              "examples": [
                "41d064eb2195891e12d0413f63227ea7"
              ]
            },
            "url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat"
              ]
            },
            "html_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://github.com/octocat"
              ]
            },
            "followers_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/followers"
              ]
            },
            "following_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/following{/other_user}"
              ]
            },
            "gists_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/gists{/gist_id}"
              ]
            },
            "starred_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/starred{/owner}{/repo}"
              ]
            },
            "subscriptions_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/subscriptions"
              ]
            },
            "organizations_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/orgs"
              ]
            },
            "repos_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/repos"
              ]
            },
            "events_url": {
              "type": "string",
              "examples": [
                "https://api.github.com/users/octocat/events{/privacy}"
              ]
            },
            "received_events_url": {
              "type": "string",
              "format": "uri",
              "examples": [
                "https://api.github.com/users/octocat/received_events"
              ]
            },
            "type": {
              "type": "string",
              "examples": [
                "User"
              ]
            },
            "site_admin": {
              "type": "boolean"
            },
            "starred_at": {
              "type": "string",
              "examples": [
                "\"2020-07-09T00:17:55Z\""
              ]
            }
          },
          "required": [
            "avatar_url",
            "events_url",
            "followers_url",
            "following_url",
            "gists_url",
            "gravatar_id",
            "html_url",
            "id",
            "node_id",
            "login",
            "organizations_url",
            "received_events_url",
            "repos_url",
            "site_admin",
            "starred_url",
            "subscriptions_url",
            "type",
            "url"
          ]
        }
      ]
    },
    "parents": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "sha": {
            "type": "string",
            "examples": [
              "7638417db6d59f3c431d3e1f261cc637155684cd"
            ]
          },
          "url": {
            "type": "string",
            "format": "uri",
            "examples": [
              "https://api.github.com/repos/octocat/Hello-World/commits/7638417db6d59f3c431d3e1f261cc637155684cd"
            ]
          },
          "html_url": {
            "type": "string",
            "format": "uri",
            "examples": [
              "https://github.com/octocat/Hello-World/commit/7638417db6d59f3c431d3e1f261cc637155684cd"
            ]
          }
        },
        "required": [
          "sha",
          "url"
        ]
      }
    },
    "stats": {
      "type": "object",
      "properties": {
        "additions": {
          "type": "integer"
        },
        "deletions": {
          "type": "integer"
        },
        "total": {
          "type": "integer"
        }
      }
    },
    "files": {
      "type": "array",
      "items": {
        "title": "Diff Entry",
        "description": "Diff Entry",
        "type": "object",
        "properties": {
          "sha": {
            "type": "string",
            "examples": [
              "bbcd538c8e72b8c175046e27cc8f907076331401"
            ]
          },
          "filename": {
            "type": "string",
            "examples": [
              "file1.txt"
            ]
          },
          "status": {
            "type": "string",
            "enum": [
              "added",
              "removed",
              "modified",
              "renamed",
              "copied",
              "changed",
              "unchanged"
            ],
            "examples": [
              "added"
            ]
          },
          "additions": {
            "type": "integer",
            "examples": [
              103
            ]
          },
          "deletions": {
            "type": "integer",
            "examples": [
              21
            ]
          },
          "changes": {
            "type": "integer",
            "examples": [
              124
            ]
          },
          "blob_url": {
            "type": "string",
            "format": "uri",
            "examples": [
              "https://github.com/octocat/Hello-World/blob/6dcb09b5b57875f334f61aebed695e2e4193db5e/file1.txt"
            ]
          },
          "raw_url": {
            "type": "string",
            "format": "uri",
            "examples": [
              "https://github.com/octocat/Hello-World/raw/6dcb09b5b57875f334f61aebed695e2e4193db5e/file1.txt"
            ]
          },
          "contents_url": {
            "type": "string",
            "format": "uri",
            "examples": [
              "https://api.github.com/repos/octocat/Hello-World/contents/file1.txt?ref=6dcb09b5b57875f334f61aebed695e2e4193db5e"
            ]
          },
          "patch": {
            "type": "string",
            "examples": [
              "@@ -132,7 +132,7 @@ module Test @@ -1000,7 +1000,7 @@ module Test"
            ]
          },
          "previous_filename": {
            "type": "string",
            "examples": [
              "file.txt"
            ]
          }
        },
        "required": [
          "additions",
          "blob_url",
          "changes",
          "contents_url",
          "deletions",
          "filename",
          "raw_url",
          "sha",
          "status"
        ]
      }
    }
  },
  "required": [
    "url",
    "sha",
    "node_id",
    "html_url",
    "comments_url",
    "commit",
    "author",
    "committer",
    "parents"
  ]
}
```

</details>

<details>

<summary>user</summary>

```json
{
  "oneOf": [
    {
      "title": "Private User",
      "description": "Private User",
      "type": "object",
      "properties": {
        "login": {
          "type": "string",
          "examples": [
            "octocat"
          ]
        },
        "id": {
          "type": "integer",
          "examples": [
            1
          ]
        },
        "node_id": {
          "type": "string",
          "examples": [
            "MDQ6VXNlcjE="
          ]
        },
        "avatar_url": {
          "type": "string",
          "format": "uri",
          "examples": [
            "https://github.com/images/error/octocat_happy.gif"
          ]
        },
        "gravatar_id": {
          "type": [
            "string",
            "null"
          ],
          "examples": [
            "41d064eb2195891e12d0413f63227ea7"
          ]
        },
        "url": {
          "type": "string",
          "format": "uri",
          "examples": [
            "https://api.github.com/users/octocat"
          ]
        },
        "html_url": {
          "type": "string",
          "format": "uri",
          "examples": [
            "https://github.com/octocat"
          ]
        },
        "followers_url": {
          "type": "string",
          "format": "uri",
          "examples": [
            "https://api.github.com/users/octocat/followers"
          ]
        },
        "following_url": {
          "type": "string",
          "examples": [
            "https://api.github.com/users/octocat/following{/other_user}"
          ]
        },
        "gists_url": {
          "type": "string",
          "examples": [
            "https://api.github.com/users/octocat/gists{/gist_id}"
          ]
        },
        "starred_url": {
          "type": "string",
          "examples": [
            "https://api.github.com/users/octocat/starred{/owner}{/repo}"
          ]
        },
        "subscriptions_url": {
          "type": "string",
          "format": "uri",
          "examples": [
            "https://api.github.com/users/octocat/subscriptions"
          ]
        },
        "organizations_url": {
          "type": "string",
          "format": "uri",
          "examples": [
            "https://api.github.com/users/octocat/orgs"
          ]
        },
        "repos_url": {
          "type": "string",
          "format": "uri",
          "examples": [
            "https://api.github.com/users/octocat/repos"
          ]
        },
        "events_url": {
          "type": "string",
          "examples": [
            "https://api.github.com/users/octocat/events{/privacy}"
          ]
        },
        "received_events_url": {
          "type": "string",
          "format": "uri",
          "examples": [
            "https://api.github.com/users/octocat/received_events"
          ]
        },
        "type": {
          "type": "string",
          "examples": [
            "User"
          ]
        },
        "site_admin": {
          "type": "boolean"
        },
        "name": {
          "type": [
            "string",
            "null"
          ],
          "examples": [
            "monalisa octocat"
          ]
        },
        "company": {
          "type": [
            "string",
            "null"
          ],
          "examples": [
            "GitHub"
          ]
        },
        "blog": {
          "type": [
            "string",
            "null"
          ],
          "examples": [
            "https://github.com/blog"
          ]
        },
        "location": {
          "type": [
            "string",
            "null"
          ],
          "examples": [
            "San Francisco"
          ]
        },
        "email": {
          "type": [
            "string",
            "null"
          ],
          "format": "email",
          "examples": [
            "octocat@github.com"
          ]
        },
        "hireable": {
          "type": [
            "boolean",
            "null"
          ]
        },
        "bio": {
          "type": [
            "string",
            "null"
          ],
          "examples": [
            "There once was..."
          ]
        },
        "twitter_username": {
          "type": [
            "string",
            "null"
          ],
          "examples": [
            "monalisa"
          ]
        },
        "public_repos": {
          "type": "integer",
          "examples": [
            2
          ]
        },
        "public_gists": {
          "type": "integer",
          "examples": [
            1
          ]
        },
        "followers": {
          "type": "integer",
          "examples": [
            20
          ]
        },
        "following": {
          "type": "integer",
          "examples": [
            0
          ]
        },
        "created_at": {
          "type": "string",
          "format": "date-time",
          "examples": [
            "2008-01-14T04:33:35Z"
          ]
        },
        "updated_at": {
          "type": "string",
          "format": "date-time",
          "examples": [
            "2008-01-14T04:33:35Z"
          ]
        },
        "private_gists": {
          "type": "integer",
          "examples": [
            81
          ]
        },
        "total_private_repos": {
          "type": "integer",
          "examples": [
            100
          ]
        },
        "owned_private_repos": {
          "type": "integer",
          "examples": [
            100
          ]
        },
        "disk_usage": {
          "type": "integer",
          "examples": [
            10000
          ]
        },
        "collaborators": {
          "type": "integer",
          "examples": [
            8
          ]
        },
        "two_factor_authentication": {
          "type": "boolean",
          "examples": [
            true
          ]
        },
        "plan": {
          "type": "object",
          "properties": {
            "collaborators": {
              "type": "integer"
            },
            "name": {
              "type": "string"
            },
            "space": {
              "type": "integer"
            },
            "private_repos": {
              "type": "integer"
            }
          },
          "required": [
            "collaborators",
            "name",
            "space",
            "private_repos"
          ]
        },
        "suspended_at": {
          "type": [
            "string",
            "null"
          ],
          "format": "date-time"
        },
        "business_plus": {
          "type": "boolean"
        },
        "ldap_dn": {
          "type": "string"
        }
      },
      "required": [
        "avatar_url",
        "events_url",
        "followers_url",
        "following_url",
        "gists_url",
        "gravatar_id",
        "html_url",
        "id",
        "node_id",
        "login",
        "organizations_url",
        "received_events_url",
        "repos_url",
        "site_admin",
        "starred_url",
        "subscriptions_url",
        "type",
        "url",
        "bio",
        "blog",
        "company",
        "email",
        "followers",
        "following",
        "hireable",
        "location",
        "name",
        "public_gists",
        "public_repos",
        "created_at",
        "updated_at",
        "collaborators",
        "disk_usage",
        "owned_private_repos",
        "private_gists",
        "total_private_repos",
        "two_factor_authentication"
      ]
    },
    {
      "title": "Public User",
      "description": "Public User",
      "type": "object",
      "properties": {
        "login": {
          "type": "string"
        },
        "id": {
          "type": "integer"
        },
        "node_id": {
          "type": "string"
        },
        "avatar_url": {
          "type": "string",
          "format": "uri"
        },
        "gravatar_id": {
          "type": [
            "string",
            "null"
          ]
        },
        "url": {
          "type": "string",
          "format": "uri"
        },
        "html_url": {
          "type": "string",
          "format": "uri"
        },
        "followers_url": {
          "type": "string",
          "format": "uri"
        },
        "following_url": {
          "type": "string"
        },
        "gists_url": {
          "type": "string"
        },
        "starred_url": {
          "type": "string"
        },
        "subscriptions_url": {
          "type": "string",
          "format": "uri"
        },
        "organizations_url": {
          "type": "string",
          "format": "uri"
        },
        "repos_url": {
          "type": "string",
          "format": "uri"
        },
        "events_url": {
          "type": "string"
        },
        "received_events_url": {
          "type": "string",
          "format": "uri"
        },
        "type": {
          "type": "string"
        },
        "site_admin": {
          "type": "boolean"
        },
        "name": {
          "type": [
            "string",
            "null"
          ]
        },
        "company": {
          "type": [
            "string",
            "null"
          ]
        },
        "blog": {
          "type": [
            "string",
            "null"
          ]
        },
        "location": {
          "type": [
            "string",
            "null"
          ]
        },
        "email": {
          "type": [
            "string",
            "null"
          ],
          "format": "email"
        },
        "hireable": {
          "type": [
            "boolean",
            "null"
          ]
        },
        "bio": {
          "type": [
            "string",
            "null"
          ]
        },
        "twitter_username": {
          "type": [
            "string",
            "null"
          ]
        },
        "public_repos": {
          "type": "integer"
        },
        "public_gists": {
          "type": "integer"
        },
        "followers": {
          "type": "integer"
        },
        "following": {
          "type": "integer"
        },
        "created_at": {
          "type": "string",
          "format": "date-time"
        },
        "updated_at": {
          "type": "string",
          "format": "date-time"
        },
        "plan": {
          "type": "object",
          "properties": {
            "collaborators": {
              "type": "integer"
            },
            "name": {
              "type": "string"
            },
            "space": {
              "type": "integer"
            },
            "private_repos": {
              "type": "integer"
            }
          },
          "required": [
            "collaborators",
            "name",
            "space",
            "private_repos"
          ]
        },
        "suspended_at": {
          "type": [
            "string",
            "null"
          ],
          "format": "date-time"
        },
        "private_gists": {
          "type": "integer",
          "examples": [
            1
          ]
        },
        "total_private_repos": {
          "type": "integer",
          "examples": [
            2
          ]
        },
        "owned_private_repos": {
          "type": "integer",
          "examples": [
            2
          ]
        },
        "disk_usage": {
          "type": "integer",
          "examples": [
            1
          ]
        },
        "collaborators": {
          "type": "integer",
          "examples": [
            3
          ]
        }
      },
      "required": [
        "avatar_url",
        "events_url",
        "followers_url",
        "following_url",
        "gists_url",
        "gravatar_id",
        "html_url",
        "id",
        "node_id",
        "login",
        "organizations_url",
        "received_events_url",
        "repos_url",
        "site_admin",
        "starred_url",
        "subscriptions_url",
        "type",
        "url",
        "bio",
        "blog",
        "company",
        "email",
        "followers",
        "following",
        "hireable",
        "location",
        "name",
        "public_gists",
        "public_repos",
        "created_at",
        "updated_at"
      ],
      "additionalProperties": false
    }
  ]
}
```

</details>

## Methodology

A set of some random words were started as a seed. Then a search was performed against the GitHub API using one of those words, a random acceptable license, a random sort order and direction. Then from each repository, one of the most recent 1,000 commits was randomly selected. And words over seven characters were added back to the random word set. This process repeated one million times.

The acceptable licenses were chosen to include every license which allows that commit to be included in this dataset without me including one million separate licenses notices. Those are (identified by SPDX identifier):

- `apache-2.0`
- `bsl-1.0`
- `bsd-2-clause`
- `bsd-3-clause`
- `0bsd`
- `cc0-1.0`
- `wtfpl`
- `isc`
- `mit`
- `postgresql`
- `unlicense`
- `zlib`

These licenses are identified in the dataset so you are easily able to filter if needed.

## Ethical considerations

Commits on GitHub include author contact information, which is publicly available to anybody even without a GitHub account. This dataset includes that contact information in each commit. This dataset creates a risk for spam unwanted outreach.

I considered whether it was appropriate to attempt redacting such information from the commit patches and contents, and ultimately decided against this.

Taking my own experience in receiving mail I have found that the vast majority of outreach to my GitHub email address (filtered through Gmail) is seemingly legitimate research requests and genuine interview requests, compared to zero apparent evil phishing or selling attempts. And practically, I understand it would be more effective and trivial for an evil spammer to look up individual author's contact information or to harvest it for a specific topic using the GitHub API than downloading and parsing this terabyte-scale dataset.

So overall, the risk was deemed acceptable.

## Caveats and recommendations

The database is not normalized. So you may find that author information in one commit does not match that same author's information in another commit. This is simply an artifact of the data collection process having taken course over a period of weeks rather than a single snapshot.

## Acknowledgements

Thank you to these people for sharing their GitHub API keys to aid in this research: [Dan Tedesco](https://twitter.com/dtedesco1), [@037](https://twitter.com/037), [Sandeep Pani](https://twitter.com/skcd42), [Axelle Moortgat](https://twitter.com/axellemoortgat)

## Citation

Please cite this dataset as:

```latex
@misc{entriken_2023_onemillioncommits,
  author       = {William Entriken},
  title        = {``One million commits''},
  year         = 2023,
  howpublished = {Hugging Face datasets},
  url          = {https://huggingface.co/datasets/fulldecent/one-million-commits}
}
```