<jupyter_start><jupyter_text>Data Wrangling  Data Flow ![Imgur](http://i.imgur.com/W5hfGVo.png)Mining data directly from GitHub, `Viz` is powered by the [GitHub API](https://developer.github.com/v3/) and leverages the following:* [`github3.py`](https://github.com/sigmavirus24/github3.py) to access the GitHub API through Python.* [`pandas`](https://github.com/pydata/pandas) in the following [IPython Notebook](https://github.com/donnemartin/viz/blob/master/githubstats/data_wrangling.ipynb) for data wrangling.* [Google Maps API](https://developers.google.com/maps/?hl=en) through [`geocoder`](https://github.com/DenisCarriere/geocoder) for location data.* [Tableau Public](https://public.tableau.com/s/) for visualizations.*In the future, [Google BigQuery](https://cloud.google.com/bigquery/) along with [GitHub Archive](https://www.githubarchive.org/) could also supplement the GitHub API.  Imports<jupyter_code>import re

import pandas as pd<jupyter_output><empty_output><jupyter_text>Prepare Repo Data Load the repos data and drop duplicates:<jupyter_code>repos = pd.read_csv("data/2017/repos-dump.csv", quotechar='"', skipinitialspace=True)
print('Shape before dropping duplicates', repos.shape)
repos = repos.drop_duplicates(subset='full_name', keep='last')
print('Shape after  dropping duplicates', repos.shape)
repos.head()<jupyter_output>Shape before dropping duplicates (8697, 5)
Shape after  dropping duplicates (8697, 5)<jupyter_text>Separate out the `user` and `repo` from `full_name` into new columns:<jupyter_code>def extract_user(line):
    return line.split('/')[0]

def extract_repo(line):
    return line.split('/')[1]

repos['user'] = repos['full_name'].str[:].apply(extract_user)
repos['repo'] = repos['full_name'].str[:].apply(extract_repo)
print(repos.shape)
repos.head()<jupyter_output>(8697, 7)<jupyter_text>Prepare User Data Load the users data and drop duplicates:<jupyter_code>users = pd.read_csv("data/2017/user-geocodes-dump.csv", quotechar='"', skipinitialspace=True)
print('Shape before dropping duplicates', users.shape)
users = users.drop_duplicates(subset='id', keep='last')
print('Shape after  dropping duplicates', users.shape)
users.head()<jupyter_output>Shape before dropping duplicates (6426, 8)
Shape after  dropping duplicates (6426, 8)<jupyter_text>Rename column `id` to `user`:<jupyter_code>users.rename(columns={'id': 'user'}, inplace=True)
users.head()<jupyter_output><empty_output><jupyter_text>Merge Repo and User Data Left join repos and users:<jupyter_code>repos_users = pd.merge(repos, users, on='user', how='left')
print('Shape repos:', repos.shape)
print('Shape users:', users.shape)
print('Shape repos_users:', repos_users.shape)
repos_users.head()<jupyter_output>Shape repos: (8697, 7)
Shape users: (6426, 8)
Shape repos_users: (8697, 14)<jupyter_text>Tidy Up Repo and User Data Re-order the columns:<jupyter_code>repos_users = repos_users.reindex_axis(['full_name',
                                        'repo',
                                        'description',
                                        'stars',
                                        'forks',
                                        'language',
                                        'user',
                                        'name',
                                        'type',
                                        'location',
                                        'lat',
                                        'long',
                                        'city',
                                        'country'], axis=1)
print(repos_users.shape)
repos_users.head()<jupyter_output>(8697, 14)<jupyter_text>Add Overall Ranks Rank each element based on number of stars:<jupyter_code>repos_users['rank'] = repos_users['stars'].rank(ascending=False)
print(repos_users.shape)
repos_users.head()<jupyter_output>(8697, 15)<jupyter_text>Verify Results: Users Equivalent [GitHub search query](https://github.com/search?utf8=%E2%9C%93&q=created%3A2017-01-01..2017-12-31+stars%3A%3E%3D100+user%3Adonnemartin&type=Repositories&ref=searchresults): `created:2017-01-01..2017-12-31 stars:>=100 user:donnemartin`*Note: The data might be slightly off, as the search query will take into account data up to when the query was executed.  Data in this notebook was mined on January 1, 2017 to 'freeze' the results for the year 2017.  The longer you run the search from January 1, 2017, the larger the discrepancy.*<jupyter_code>repos_users[repos_users['user'] == 'donnemartin']<jupyter_output><empty_output><jupyter_text>Verify Results: Python Repos Equivalent [GitHub search query](https://github.com/search?utf8=%E2%9C%93&q=created%3A2017-01-01..2017-12-31+stars%3A%3E%3D100+language%3Apython&type=Repositories&ref=searchresults): `created:2017-01-01..2017-12-31 stars:>=100 language:python`*Note: The data might be slightly off, as the search query will take into account data up to when the query was executed.  Data in this notebook was mined on January 1, 2017 to 'freeze' the results for the year 2017.  The longer you run the search from January 1, 2017, the larger the discrepancy.*<jupyter_code>print(repos_users[repos_users['language'] == 'Python'].shape)
repos_users[repos_users['language'] == 'Python'].head()<jupyter_output>(1357, 15)<jupyter_text>Verify Results: Overall Repos Equivalent [GitHub search query](https://github.com/search?utf8=%E2%9C%93&q=created%3A2017-01-01..2017-12-31+stars%3A%3E%3D100&type=Repositories&ref=searchresults): `created:2017-01-01..2017-12-31 stars:>=100`*Note: The data might be slightly off, as the search query will take into account data up to when the query was executed.  Data in this notebook was mined on January 1, 2017 to 'freeze' the results for the year 2017.  The longer you run the search from January 1, 2017, the larger the discrepancy.*<jupyter_code>print(repos_users.shape)
repos_users.head()<jupyter_output>(8697, 15)<jupyter_text>Output Results Write out the results to csv to visualize in Tableau:<jupyter_code>users.to_csv('data/2017/users.csv', index=False)
repos_users.to_csv('data/2017/repos-users-geocodes.csv', index=False)
repos_users.to_csv('data/2017/repos-users.csv', index=False)
repos_rank = repos_users.reindex_axis(['full_name', 'rank'], axis=1)
repos_rank.to_csv('data/2017/repos-ranks.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13009922822491732, 'fraction_numerical': 0.0447314537722476, 'mean_word_length': 4.226337448559671, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 44, 'https://': 12, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '150324', 'license': 'Apache-2.0', 'n_tokens_mistral': 2237, 'n_tokens_neox': 1958, 'n_words': 467, 'path': 'githubstats/data_wrangling.ipynb', 'repo_name': 'tigefa4u/viz', 'size': '6349'}