<jupyter_start><jupyter_text>Raw Data Summary<jupyter_code>top_repos_raw = pd.read_csv("../data/scraped/top-repos.csv", parse_dates=["created"])
user_data_raw = pd.read_csv("../data/scraped/user-data.csv", parse_dates=["created"])
top_user_repos_raw = pd.read_csv("../data/scraped/top-user-repos.csv", parse_dates=["created"])
top_org_repos_raw = pd.read_csv("../data/scraped/top-org-repos.csv", parse_dates=["created"])
top_repos_raw.info()
user_data_raw.info()
top_user_repos_raw.info()
top_org_repos_raw.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 6252 entries, 0 to 6251
Data columns (total 13 columns):
 #   Column       Non-Null Count  Dtype         
---  ------       --------------  -----         
 0   id           6252 non-null   int64         
 1   repo_name    6252 non-null   object        
 2   full_name    6252 non-null   object        
 3   description  5696 non-null   object        
 4   created      6252 non-null   datetime64[ns]
 5   language     5532 non-null   object        
 6   type         6252 non-null   object        
 7   username     6252 non-null   object        
 8   stars        6252 non-null   int64         
 9   forks        6252 non-null   int64         
 10  subscribers  6252 non-null   int64         
 11  open_issues  6252 non-null   int64         
 12  topics       6252 non-null   object        
dtypes: datetime64[ns](1), int64(5), object(7)
memory usage: 635.1+ KB<jupyter_text>Cleaned Data Summary<jupyter_code>top_repos = pd.read_csv("../data/cleaned/top-repos.csv", parse_dates=["created"])
user_data = pd.read_csv("../data/cleaned/user-data.csv", parse_dates=["created"])
top_user_repos = pd.read_csv("../data/cleaned/top-user-repos.csv", parse_dates=["created"])
top_org_repos = pd.read_csv("../data/cleaned/top-org-repos.csv", parse_dates=["created"])
location_df = pd.read_csv("../data/cleaned/user-location-data.csv", parse_dates=["created"])
top_repos.info()
user_data.info()
top_user_repos.info()
top_org_repos.info()
location_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 921 entries, 0 to 920
Data columns (total 20 columns):
 #   Column        Non-Null Count  Dtype         
---  ------        --------------  -----         
 0   id            921 non-null    int64         
 1   username      921 non-null    object        
 2   name          889 non-null    object        
 3   type          921 non-null    object        
 4   bio           681 non-null    object        
 5   created       921 non-null    datetime64[ns]
 6   company       476 non-null    object        
 7   email         525 non-null    object        
 8   location      921 non-null    object        
 9   hireable      246 non-null    object        
 10  followers     921 non-null    int64         
 11  following     921 non-null    int64         
 12  public_gists  921 non-null    int64         
 13  public_repos  921 non-null    int64         
 14  subject       921 non-null    object        
 15  geo-location  921 non-null    object    [...]
{'fraction_non_alphanumeric': 0.11629434954007885, 'fraction_numerical': 0.06340341655716163, 'mean_word_length': 1.9678362573099415, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '220566', 'license': 'MIT', 'n_tokens_mistral': 1124, 'n_tokens_neox': 988, 'n_words': 224, 'path': 'results/data_summary.ipynb', 'repo_name': 'nicovandenhooff/top-repo-analysis', 'size': '3044'}