<commit_before>GitHub stores CodeQL databases for over 200,000 repos on GitHub.com, which you can download using the REST API. The list of repos is constantly growing and evolving to make sure that it includes the most interesting codebases for security research.

You can check if a repository has any CodeQL databases available for download using the ``/repos/<owner>/<repo>/code-scanning/codeql/databases`` endpoint.
For example, to check for CodeQL databases using the `GitHub CLI <https://cli.github.com/manual/gh_api>`__ you would run::

   gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/

This command returns information about any CodeQL databases that are available for a repository, including the language the database represents, and when the database was last updated. If no CodeQL databases are available, the response is empty.

When you have confirmed that a CodeQL database exists for the language you are interested in, you can download it using the following command:: 

   gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/<language> -H 'Accept: application/zip' > path/to/local/database.zip

TODO: add link to the REST API docs.<commit_msg>add link and remove todo<commit_after>GitHub stores CodeQL databases for over 200,000 repos on GitHub.com, which you can download using the REST API. The list of repos is constantly growing and evolving to make sure that it includes the most interesting codebases for security research.

You can check if a repository has any CodeQL databases available for download using the ``/repos/<owner>/<repo>/code-scanning/codeql/databases`` endpoint.
For example, to check for CodeQL databases using the `GitHub CLI <https://cli.github.com/manual/gh_api>`__ you would run::

   gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/

This command returns information about any CodeQL databases that are available for a repository, including the language the database represents, and when the database was last updated. If no CodeQL databases are available, the response is empty.

When you have confirmed that a CodeQL database exists for the language you are interested in, you can download it using the following command:: 

   gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/<language> -H 'Accept: application/zip' > path/to/local/database.zip

For more information, see the documentation for the `Get CodeQL database <https://docs.github.com/en/rest/reference/code-scanning#get-codeql-database>`__ endpoint in the GitHub REST API documentation.
{'fraction_non_alphanumeric': 0.07280598189689098, 'fraction_numerical': 0.004722550177095631, 'mean_word_length': 4.790432801822323, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 22, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1210296', 'n_tokens_mistral': 692, 'n_tokens_neox': 630, 'n_words': 316}