<jupyter_start><jupyter_text>Generate Deep Review stats for templated use in the Meta ReviewNote that some statistics are generated from the deep-review git submodule, and correspond to the version specified by `commit`. Other statistics are generated from the GitHub API and reflect the repositories status at runtime, as specified by `creation_time_utc`.<jupyter_code>import collections
import datetime
import json
import pathlib

import pandas
import requests
import yaml
stats = collections.OrderedDict()
stats['creation_time_utc'] = datetime.datetime.utcnow().isoformat()<jupyter_output><empty_output><jupyter_text>Deep Review git repository stats<jupyter_code>commit_df = pandas.read_table('commits.tsv')
commit_df.tail(2)
# State of the git repo (deep-review submodule)
stats['commit'] = commit_df.commit.iloc[-1]
stats['total_commits'] = len(commit_df)
# Number of non-merge commits that modified the manuscript markdown source
writing_commit_df = commit_df.query("(characters_added > 0 or characters_deleted > 0) and merge == 0")
stats['manuscript_commits'] = len(writing_commit_df)<jupyter_output><empty_output><jupyter_text>Number of formal Deep Review authors<jupyter_code>path = pathlib.Path('deep-review/content/metadata.yaml')
with path.open() as read_file:
    metadata = yaml.load(read_file)
authors = metadata['author_info']
stats['deep_review_authors'] = len(authors)<jupyter_output><empty_output><jupyter_text>Authors who first contributed after May 28, 2017 ([date](https://github.com/greenelab/meta-review/issues/23issuecomment-338833911) of first submission) but not after March 12, 2018 (date of [v1.0 release](https://github.com/greenelab/deep-review/releases/tag/v1.0)). Note that author names have not been standardized here.<jupyter_code>pre_submission_authors = set(writing_commit_df.query("authored_datetime <= '2017-05-28'").author_name)
post_submission_authors = set(writing_commit_df.query("'2017-05-28' < authored_datetime <= '2018-03-12'").author_name)
new_authors = post_submission_authors - pre_submission_authors
new_authors
stats['deep_review_post_submission_authors'] = len(new_authors)<jupyter_output><empty_output><jupyter_text>GitHub repo stats<jupyter_code># https://developer.github.com/v3/repos/#get
response = requests.get('https://api.github.com/repos/greenelab/deep-review')
result = response.json()
stats['github_stars'] = result['stargazers_count']
stats['github_forks'] = result['forks_count']<jupyter_output><empty_output><jupyter_text>Number of pull requests<jupyter_code>def github_issue_search(query):
    """
    Search issues and pull requests on GitHub.

    https://developer.github.com/v3/search/#search-issues
    https://help.github.com/articles/searching-issues-and-pull-requests/
    """
    url = 'https://api.github.com/search/issues'
    params = {
        'q': query,
        'sort': 'created',
        'order': 'asc',
    }
    response = requests.get(url, params)
    print(response.url)
    assert response.status_code == 200
    result = response.json()
    assert not result['incomplete_results']
    return result
# Merged PRs
result = github_issue_search('repo:greenelab/deep-review type:pr is:merged')
stats['merged_pull_requests'] = result['total_count']
# Closed PRs that were not merged
result = github_issue_search('repo:greenelab/deep-review type:pr is:unmerged state:closed')
stats['declined_pull_requests'] = result['total_count']
# Open PRs
result = github_issue_search('repo:greenelab/deep-review type:pr state:open')
stats['open_pull_requests'] = result['total_count']
# Open Issues
result = github_issue_search('repo:greenelab/deep-review type:issue state:open')
stats['open_issues'] = result['total_count']
# Closed Issues
result = github_issue_search('repo:greenelab/deep-review type:issue state:closed')
stats['closed_issues'] = result['total_count']<jupyter_output>https://api.github.com/search/issues?q=repo%3Agreenelab%2Fdeep-review+type%3Aissue+state%3Aclosed&sort=created&order=asc<jupyter_text>Write stats<jupyter_code>stats_str = json.dumps(stats, indent=2)
print(stats_str)
path = pathlib.Path('deep-review-stats.json')
path.write_text(stats_str)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12287425149700598, 'fraction_numerical': 0.015808383233532935, 'mean_word_length': 6.93916349809886, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 30, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '326913', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1409, 'n_tokens_neox': 1318, 'n_words': 260, 'path': 'analyses/deep-review-contrib/03.contrib-stats.ipynb', 'repo_name': 'slochower/meta-review', 'size': '4175'}