<jupyter_start><jupyter_text>Analysis of user collectionIn this notebook, the retrieved results of the user collection is analyzed.<jupyter_code>import glob
import time
import sys
from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Configure modules like pandas and plotting libraries.<jupyter_code>plt.rcParams['figure.figsize'] = [12, 8]
pd.set_option('display.min_rows', 24)<jupyter_output><empty_output><jupyter_text>Folders and constants<jupyter_code>fp_figs = Path("figs")

annotated_users = Path("results", "users_enriched_summer2021.xlsx")<jupyter_output><empty_output><jupyter_text>DatasetIn this section, output data of the various methods is collected and combined.<jupyter_code>data_files = glob.glob("methods/*/results/*.csv")

for i, d in enumerate(data_files):
    print(i+1, d)<jupyter_output>1 methods/papers_with_code/results/paperswithcode.csv
2 methods/github_search/results/github_search_topic.csv
3 methods/github_search/results/github_search_users.csv
4 methods/github_search/results/github_search_repos.csv
5 methods/profile_pages/results/profile_page_uu.csv
6 methods/pure/results/pure.csv<jupyter_text>All files is loaded and combined in a single dataset. The dataset contains the variables:- `source` The source is the method used to find the user, e.g. paperswithcode, github_search_users, github_search_topics. The name is derived from the file name of the results file.- `service` The service the user profile is found, e.g. github.com, github.warwick.ac.uk, gitlab.- `date` The date of collection. This can be useful when updating results.- `user_id` The user handle of the found user.<jupyter_code>df_user_names = pd.concat(
    [pd.read_csv(fp) for fp in data_files], 
    axis=0, 
    keys=data_files, 
    names=["source", "row"]
) \
    .reset_index("source") \
    .reset_index(drop=True)

df_user_names["source"] = df_user_names["source"].apply(lambda x: Path(x).stem)
df_user_names<jupyter_output><empty_output><jupyter_text>FindabilityThe following results shows how many times user handles are found with each retrieval method. If a user is found with multiple methods, the findability of the user can be consider to be better.   Cross table user and methodCreate a cross table with service and user on one axis and the extraction method on the other axis.<jupyter_code>df_name_method_crosstab = df_user_names \
    .drop("date", axis=1) \
    .groupby(["service", "user_id", "source"]) \
    .size() \
    .unstack("source") \
    .fillna(0) \
    .applymap(int)

df_name_method_crosstab<jupyter_output><empty_output><jupyter_text>The following example can be uncommented to find the findabilty of a specific project.<jupyter_code># df_name_method_crosstab.loc[('github.com', 'asreview')]<jupyter_output><empty_output><jupyter_text>Findability per userThe following table show which users are found with most collection strategies. The count indicates the number of methods, and the relative score the number relative to the total number of collection strategies.<jupyter_code># compute the count
df_name_findability = (df_name_method_crosstab > 0) \
    .astype(int) \
    .sum(axis=1) \
    .sort_values(ascending=False) \
    .to_frame(name="count")

# compute relative score
df_name_findability["relative"] = df_name_findability["count"] / len(data_files)

df_name_findability

retrieval_count = df_name_findability["count"].value_counts().to_frame(name="Retrieval count")

sns.barplot(
    x=retrieval_count.index.astype(str), 
    y=retrieval_count["Retrieval count"]
)
plt.title("Retrieval count for each user in dataset")
plt.savefig(Path(fp_figs, 'user_collection_user_findability.png'))<jupyter_output><empty_output><jupyter_text>Findability per methodThis table indicates how succeful a method is collecting users.<jupyter_code>df_method_findability = (df_name_method_crosstab > 0) \
    .astype(int) \
    .sum(axis=0) \
    .sort_values(ascending=False) \
    .to_frame(name="count")

df_method_findability
df_method_findability
sns.barplot(
    x=df_method_findability.index, 
    y=df_method_findability["count"]
)
plt.title("Retrieval count for each method in dataset")
plt.savefig(Path(fp_figs, 'user_collection_method_findability.png'))<jupyter_output><empty_output><jupyter_text>Filtering of usersNot all users collected in the previous steps are relevant to the analysis. Some are not (or no longer) part of the organisation, or excluded for other reasons (for example students).<jupyter_code>df_annotated_users = pd.read_excel(annotated_users)

df_annotated_users[
    ["user_id", 
     "is_student", 
     "is_employee", 
     "is_currently_employed", 
     "is_research_group", 
     "final_decision"
    ]
]
df_annotated_users_included = df_annotated_users[df_annotated_users["final_decision"] == 1]

print("The number of included users is", len(df_annotated_users_included))<jupyter_output>The number of included users is 143<jupyter_text>Properties of users<jupyter_code>df_annotated_users_included \
    .loc[df_annotated_users_included["final_decision"] == 1, "is_research_group"] \
    .fillna(0) \
    .value_counts()
# create a boxplot with swarm
ax = sns.boxplot(x='public_repos', data=df_annotated_users_included)
ax = sns.swarmplot(x='public_repos', data=df_annotated_users_included, color=".25")

# output users with most public repos
df_annotated_users_included[['user_id', 'public_repos']].sort_values('public_repos', ascending=False).head(10)
sns.boxplot(x='public_gists', data=df_annotated_users_included)
df_annotated_users_included[['user_id', 'public_gists']].sort_values('public_gists', ascending=False).head(10)
ax = sns.boxplot(x='followers', data=df_annotated_users_included, showfliers=False)

df_annotated_users_included[['user_id', 'followers']].sort_values('followers', ascending=False).head(10)
sns.boxplot(x=df_annotated_users_included['following'], showfliers = False)
df_annotated_users_included[['user_id', 'following']].sort_values('following', ascending=False).head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10543603218919363, 'fraction_numerical': 0.006404992609623912, 'mean_word_length': 6.481572481572481, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '634684', 'license': 'MIT', 'n_tokens_mistral': 2078, 'n_tokens_neox': 1984, 'n_words': 455, 'path': 'collect_users/analyze_users.ipynb', 'repo_name': 'UtrechtUniversity/SWORDS-UU', 'size': '6089'}