<jupyter_start><jupyter_text>Mining repos from GitHubFor all available at the moment programming languages:- C- C- C++- Go- Java- JavaScript- Kotlin- Objective-C- PHP- Python- Ruby- Swift- TypeScript<jupyter_code>import os
%load_ext autoreload
%autoreload 2
import pandas as pd
from tqdm import tqdm
pd.options.display.float_format = '{:.2f}'.format

os.chdir('D:\stuff\github_data')<jupyter_output><empty_output><jupyter_text>Выставляем на [GitHub Search](https://seart-ghs.si.usi.ch/) следующие фильтры:* 50+ звезд* 1000+ коммитов* 10+ контрибьюторов* репозитории созданы не позднее 15 сентября 2019 года (чтобы им было 2+ года)* не берем форки* не берем репозитории без лицензии<jupyter_code>ghs_df = pd.read_csv('ghs_results.csv')
ghs_df.shape
ghs_df.head()<jupyter_output><empty_output><jupyter_text>Соберем через GitHub API дополнительную информацию о репозиториях, оставшихся после применения фильтров в GitHub Search.<jupyter_code>from github import Github
from github.GithubException import RateLimitExceededException, UnknownObjectException, GithubException

# using an access token
with open('access_token.txt') as file:
    access_token = file.readline().strip()
    
g = Github(access_token)
import calendar
import time
import datetime


for name in tqdm(ghs_df['Name'][11552:]):
    try:
        repos.append(g.get_repo(name))
    except RateLimitExceededException:
        core_rate_limit = g.get_rate_limit().core
        reset_timestamp = calendar.timegm(core_rate_limit.reset.timetuple())
        sleep_time = reset_timestamp - calendar.timegm(time.gmtime()) + 5
        print(f"{datetime.datetime.now()} Rate Limit exceeded! Sleeping for {sleep_time // 60} minutes")
        time.sleep(sleep_time)
        repos.append(g.get_repo(name))
    except UnknownObjectException:
        print(f"Unknown object exception with repo {name}")
        repos.append(-1)
    except GithubException:
        print(f"GitHub exception with repo {name}")
        repos.append(-1)
full_names = []
git_urls = []
languages = []

for repo in tqdm(repos):
    try:
        full_names.append(repo.full_name)
        git_urls.append(repo.git_url)
        languages.append(repo.language)
    except:
        full_names.append('')
        git_urls.append('')
        languages.append('null')
ghs_df['full_name'] = full_names
ghs_df['git_url'] = git_urls
ghs_df["git_url"] = ghs_df["git_url"].apply(lambda x: "https" + x[3:])
ghs_df['Language'] = languages
ghs_df['License'].unique()
ghs_df['Language'].unique()<jupyter_output><empty_output><jupyter_text>Теперь еще больше пофильтруем репозитории:1. Отбросим репозитории с лицензиями, отличными от Apache, MIT или BSD 3-Clause2. Отбросим совпадения по `full_name` (которые могли появиться из-за переименования/переноса репозиториев)<jupyter_code># leave repos only with Apache / MIT / BSD license
res_df = ghs_df.loc[ghs_df['License'].isin(['Apache License 2.0', 'MIT License',
                                            'BSD 3-Clause New or Revised License'])]
print(f"After dropping repos without Apache/MIT/BSD licences: {len(res_df)}, diff with previous step: {len(ghs_df) - len(res_df)}")
prev_len = len(res_df)

# drop duplicate repos
res_df = res_df.drop_duplicates(subset=['full_name'])
print(f"After dropping full_name duplicates: {len(res_df)}, diff with previous step: {prev_len - len(res_df)}")

res_df = res_df.sort_values(by='Stargazers', ascending=False)
res_df.head()
res_df.groupby("Language").count()["Name"].sort_values()
res_df.groupby("Language").sum()["Commits"].sort_values()
print(f"{len(res_df)} repos and {res_df['Commits'].sum()} commits")
res_df.to_csv('filtered_result.csv')
res_df['Name'].to_csv('repos_names.txt', index=None, header=None)
res_df['git_url'].to_csv('repos_urls.txt', index=None, header=None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12733000787608295, 'fraction_numerical': 0.008926227356261486, 'mean_word_length': 5.0, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '443094', 'license': 'Apache-2.0', 'n_tokens_mistral': 1382, 'n_tokens_neox': 1386, 'n_words': 282, 'path': 'notebooks/multilang/choosing_repos.ipynb', 'repo_name': 'saridormi/commits_dataset', 'size': '3809'}