<jupyter_start><jupyter_text>This notebook will help create a visualization of english wikipedia user traffic There are three major steps: 1. Retrieve Pagecount Data using the API and store this output in JSON files 2. Merge all data and store this as a CSV file 3. Generate and save the visualization  1. Retrieve Pagecount Data Set up the API callsAPI code borrowed from http://paws-public.wmflabs.org/paws-public/User:Jtmorgan/data512_a1_example.ipynbUses the Wikipedia API https://wikimedia.org/api/rest_v1/!/Legacy_data/get_metrics_legacy_pagecounts_aggregate_project_access_site_granularity_start_end<jupyter_code>import json
import requests
endpoint_legacy = 'https://wikimedia.org/api/rest_v1/metrics/legacy/pagecounts/aggregate/{project}/{access-site}/{granularity}/{start}/{end}'

endpoint_pageviews = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/{project}/{access}/{agent}/{granularity}/{start}/{end}'
# SAMPLE parameters for getting aggregated legacy view data 
# see: https://wikimedia.org/api/rest_v1/#!/Legacy_data/get_metrics_legacy_pagecounts_aggregate_project_access_site_granularity_start_end
params_legacy = {"project" : "en.wikipedia.org",
                 "access-site" : "desktop-site",
                 "granularity" : "monthly",
                 "start" : "2001010100",
                # for end use 1st day of month following final month of data
                 "end" : "2016070100"
                    }

params_legacy_mobile = {"project" : "en.wikipedia.org",
                 "access-site" : "mobile-site",
                 "granularity" : "monthly",
                 "start" : "2001010100",
                # for end use 1st day of month following final month of data
                 "end" : "2016070100"
                    }

params_pageviews = {"project" : "en.wikipedia.org",
                    "access" : "desktop",
                    "agent" : "user",
                    "granularity" : "monthly",
                    "start" : "2001010100",
                    # for end use 1st day of month following final month of data
                    "end" : '2020100100'
                        }

params_pageviews_mobile_app = {"project" : "en.wikipedia.org",
                    "access" : "mobile-app",
                    "agent" : "user",
                    "granularity" : "monthly",
                    "start" : "2001010100",
                    # for end use 1st day of month following final month of data
                    "end" : '2020100100'
                        }

params_pageviews_mobile_web = {"project" : "en.wikipedia.org",
                    "access" : "mobile-web",
                    "agent" : "user",
                    "granularity" : "monthly",
                    "start" : "2001010100",
                    # for end use 1st day of month following final month of data
                    "end" : '2020100100'
                        }

# Customize these with your own information
headers = {
    'User-Agent': 'https://github.com/jfm888',
    'From': '<EMAIL>'
}
def api_call(endpoint,parameters):
    call = requests.get(endpoint.format(**parameters), headers=headers)
    response = call.json()
    
    return response<jupyter_output><empty_output><jupyter_text>Run the API calls Generates the traffic data from api calls to the wiki api<jupyter_code>monthly_legacy_desktop = api_call(endpoint_legacy, params_legacy)
monthly_legacy_mobile = api_call(endpoint_legacy, params_legacy_mobile)
monthly_pageviews_desktop = api_call(endpoint_pageviews, params_pageviews)
monthly_pageviews_mobile_app = api_call(endpoint_pageviews, params_pageviews_mobile_app)
monthly_pageviews_mobile_web = api_call(endpoint_pageviews, params_pageviews_mobile_web)<jupyter_output><empty_output><jupyter_text>Explore the API data to see if it makes senseFor brevity I've only shown the first and last entry in each API call but we can see the date ranges<jupyter_code>print(monthly_legacy_desktop['items'][0])
print(monthly_legacy_desktop['items'][-1])
print(monthly_legacy_mobile['items'][0])
print(monthly_legacy_mobile['items'][-1])
print(monthly_pageviews_desktop['items'][0])
print(monthly_pageviews_desktop['items'][-1])
print(monthly_pageviews_mobile_app['items'][0])
print(monthly_pageviews_mobile_app['items'][-1])
print(monthly_pageviews_mobile_web['items'][0])
print(monthly_pageviews_mobile_web['items'][-1])<jupyter_output>{'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015070100', 'views': 3179131148}
{'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2020090100', 'views': 4345048664}<jupyter_text>Save the API call data in JSON filesWe will save the API call data in JSON files as a backupFormat for the saved files is "apiname_accesstype_firstmonth-lastmonth.json" using "with open" will close our files automatically after we are done writing, very nice<jupyter_code>with open("pagecounts_desktop-site_200712-201608.json", "w") as outfile: 
    json.dump(monthly_legacy_desktop, outfile)
with open("pagecounts_mobile-site_201410-201608.json", "w") as outfile: 
    json.dump(monthly_legacy_mobile, outfile) 
with open("pageviews_desktop_201507-202009.json", "w") as outfile: 
    json.dump(monthly_pageviews_desktop, outfile)
with open("pageviews_mobile-app_201507-202009.json", "w") as outfile: 
    json.dump(monthly_pageviews_mobile_app, outfile)
with open("pageviews_mobile-web_201507-202009.json", "w") as outfile: 
    json.dump(monthly_pageviews_mobile_web, outfile)<jupyter_output><empty_output><jupyter_text>2. Merge all API data Create Pandas DataFrame objects for each API call import the Pandas package and alias it as pd<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>create the pagecount_desktop dataframe<jupyter_code>df_pagecount_desktop = pd.DataFrame(monthly_legacy_desktop['items']).sort_values(by=['timestamp'])
df_pagecount_desktop.head()<jupyter_output><empty_output><jupyter_text>create the pagecount_mobile dataframe<jupyter_code>df_pagecount_mobile = pd.DataFrame(monthly_legacy_mobile['items']).sort_values(by=['timestamp'])
df_pagecount_mobile.head()<jupyter_output><empty_output><jupyter_text>create the pageviews_mobile_desktop dataframe<jupyter_code>df_pageviews_desktop = pd.DataFrame(monthly_pageviews_desktop['items']).sort_values(by=['timestamp'])
df_pageviews_desktop.head()<jupyter_output><empty_output><jupyter_text>create the pageviews_mobile_app dataframe<jupyter_code>df_pageviews_mobile_app = pd.DataFrame(monthly_pageviews_mobile_app['items']).sort_values(by=['timestamp'])
df_pageviews_mobile_app.head()<jupyter_output><empty_output><jupyter_text>create the pageviews_mobile_web dataframe<jupyter_code>df_pageviews_mobile_web = pd.DataFrame(monthly_pageviews_mobile_web['items']).sort_values(by=['timestamp'])
df_pageviews_mobile_web.head()<jupyter_output><empty_output><jupyter_text>Inital formatting of the dataframesWe want to change the views column to reflect type of view either desktop or mobile and date column to be two columns: year, and month. We will drop the other columns.<jupyter_code>df_pagecount_desktop.head()<jupyter_output><empty_output><jupyter_text>drop 'project', 'access-site', and 'granularity' columns<jupyter_code>df_pagecount_desktop = df_pagecount_desktop.drop(['project', 'access-site', 'granularity'], axis=1)
df_pagecount_mobile = df_pagecount_mobile.drop(['project', 'access-site', 'granularity'], axis=1)
df_pageviews_desktop.head()<jupyter_output><empty_output><jupyter_text>filter pageview dataframes so that the only agents are users<jupyter_code>only_user_desktop = df_pageviews_desktop['agent']=='user'
df_pageviews_desktop = df_pageviews_desktop[only_user_desktop]

only_user_mobile_app = df_pageviews_mobile_app['agent']=='user'
df_pageviews_mobile_app = df_pageviews_mobile_app[only_user_mobile_app]

only_user_mobile_web = df_pageviews_mobile_web['agent']=='user'
df_pageviews_mobile_web = df_pageviews_mobile_web[only_user_mobile_web]<jupyter_output><empty_output><jupyter_text>from pageview dataframes drop 'project', 'access', 'agent', and 'granularity' columns<jupyter_code>df_pageviews_desktop = df_pageviews_desktop.drop(['project', 'access', 'agent','granularity'], axis=1)
df_pageviews_mobile_app = df_pageviews_mobile_app.drop(['project', 'access', 'agent','granularity'], axis=1)
df_pageviews_mobile_web = df_pageviews_mobile_web.drop(['project', 'access', 'agent','granularity'], axis=1)<jupyter_output><empty_output><jupyter_text>rename views and counts to more descriptive names for such as "pageview_desktop_views"...<jupyter_code>df_pagecount_desktop = df_pagecount_desktop.rename(columns={'timestamp':'timestamp', 'count':'pagecount_desktop_views'
                                                           })
df_pagecount_mobile = df_pagecount_mobile.rename(columns={'timestamp':'timestamp', 'count':'pagecount_mobile_views'
                                                           })
df_pageviews_desktop = df_pageviews_desktop.rename(columns={'timestamp':'timestamp', 'views':'pageview_desktop_views'
                                                           })
df_pageviews_mobile_app = df_pageviews_mobile_app.rename(columns={'timestamp':'timestamp', 'views':'pageview_mobile_views'
                                                           })
df_pageviews_mobile_web = df_pageviews_mobile_web.rename(columns={'timestamp':'timestamp', 'views':'pageview_mobile_views'
                                                           })<jupyter_output><empty_output><jupyter_text>Inspect the data<jupyter_code>print(df_pagecount_desktop.head())
print(df_pagecount_mobile.head())<jupyter_output>timestamp  pagecount_desktop_views
0  2007120100               2998331524
1  2008010100               4930902570
2  2008020100               4818393763
3  2008030100               4955405809
4  2008040100               5159162183
    timestamp  pagecount_mobile_views
0  2014100100              3091546685
1  2014110100              3027489668
2  2014120100              3278950021
3  2015010100              3485302091
4  2015020100              3091534479<jupyter_text>Inspect the data<jupyter_code>print(df_pageviews_desktop.head())
print(df_pageviews_mobile_app.head())
print(df_pageviews_mobile_web.head())<jupyter_output>timestamp  pageview_desktop_views
0  2015070100              4376666686
1  2015080100              4332482183
2  2015090100              4485491704
3  2015100100              4477532755
4  2015110100              4287720220
    timestamp  pageview_mobile_views
0  2015070100              109624146
1  2015080100              109669149
2  2015090100               96221684
3  2015100100               94523777
4  2015110100               94353925
    timestamp  pageview_mobile_views
0  2015070100             3179131148
1  2015080100             3192663889
2  2015090100             3073981649
3  2015100100             3173975355
4  2015110100             3142247145<jupyter_text>merge pagecount dataframesneed to use a left join as desktop date range is wider than mobile date range<jupyter_code>df_pagecounts = df_pagecount_desktop.merge(df_pagecount_mobile, how='left', left_on='timestamp', right_on='timestamp')<jupyter_output><empty_output><jupyter_text>replace the 'nan' in our data frame with zeros<jupyter_code>df_pagecounts = df_pagecounts.fillna(0)<jupyter_output><empty_output><jupyter_text>Let's inspect our data<jupyter_code>df_pagecounts.head()
df_pagecounts.tail()
df_pagecounts['pagecount_all_views'] = df_pagecounts.sum(axis=1)
df_pagecounts.head()<jupyter_output><empty_output><jupyter_text>merge pageview dataframespageview api data all covers same date range so we dont need a left join and can use a merge or inner join<jupyter_code>df_pageviews_all_mobile = df_pageviews_mobile_app.merge(df_pageviews_mobile_web, left_on='timestamp', right_on='timestamp')
df_pageviews_all_mobile.head()<jupyter_output><empty_output><jupyter_text>Create the "all views" column<jupyter_code>df_pageviews_all_mobile['pageview_mobile_views'] = df_pageviews_all_mobile.sum(axis=1)<jupyter_output><empty_output><jupyter_text>drop the mobile app and web data i.e. the x and y data<jupyter_code>df_pageviews_all_mobile = df_pageviews_all_mobile.drop(['pageview_mobile_views_x', 'pageview_mobile_views_y'], axis=1)<jupyter_output><empty_output><jupyter_text>inspect our dataframe<jupyter_code>df_pageviews_all_mobile.head()<jupyter_output><empty_output><jupyter_text>We are merging the mobile app and web data<jupyter_code>df_pageviews = df_pageviews_all_mobile.merge(df_pageviews_desktop, left_on='timestamp', right_on='timestamp')<jupyter_output><empty_output><jupyter_text>let's see our dataframe<jupyter_code>df_pageviews.head()<jupyter_output><empty_output><jupyter_text>Create the "all views" column<jupyter_code>df_pageviews['pageview_all_views'] = df_pageviews.sum(axis=1)<jupyter_output><empty_output><jupyter_text>Let's see our dataframe<jupyter_code>df_pageviews.head()<jupyter_output><empty_output><jupyter_text>merge pageview and pagecount data framesThese dataframes cover different time ranges so we need an outer join<jupyter_code>df_traffic = df_pagecounts.merge(df_pageviews, how='outer')<jupyter_output><empty_output><jupyter_text>replace the 'nan' with zeros<jupyter_code>df_traffic = df_traffic.fillna(0)<jupyter_output><empty_output><jupyter_text>let's inspect our dataframe<jupyter_code>df_traffic.head()<jupyter_output><empty_output><jupyter_text>change timestamp to datetime object so we can easily extract year and month data<jupyter_code>df_traffic['timestamp'] = pd.to_datetime(df_traffic['timestamp'], format='%Y%m%d%H')<jupyter_output><empty_output><jupyter_text>create the year column<jupyter_code>df_traffic['year'] = pd.DatetimeIndex(df_traffic['timestamp']).year<jupyter_output><empty_output><jupyter_text>create the month column<jupyter_code>df_traffic['month'] = pd.DatetimeIndex(df_traffic['timestamp']).month<jupyter_output><empty_output><jupyter_text>save this dataframe for latter as it will make visualization easier if we have a datatime col<jupyter_code>df_traffic_timestamp = df_traffic #save for latter as it will make visualization easier if we have a datatime col<jupyter_output><empty_output><jupyter_text>drop the 'timestamp' column<jupyter_code>df_traffic = df_traffic.drop(['timestamp'], axis=1)<jupyter_output><empty_output><jupyter_text>make the data int64 type<jupyter_code>df_traffic = df_traffic.astype('int64')<jupyter_output><empty_output><jupyter_text>to make zero padded months<jupyter_code>df_traffic["month"] = df_traffic['month'].map("{:02}".format)<jupyter_output><empty_output><jupyter_text>inspect the dataframe<jupyter_code>df_traffic<jupyter_output><empty_output><jupyter_text>Create CSV file from our combined dataframe<jupyter_code>df_traffic.to_csv(r'en-wikipedia_traffic_200712-202009.csv', index = False)<jupyter_output><empty_output><jupyter_text>Generate and save the visualization<jupyter_code>import matplotlib.pyplot as plt
import numpy as np<jupyter_output><empty_output><jupyter_text>make values smaller to increase legibility of visualization<jupyter_code>df_traffic_timestamp[['pagecount_desktop_views', 'pagecount_mobile_views',
       'pagecount_all_views', 'pageview_mobile_views',
       'pageview_desktop_views', 'pageview_all_views']] = df_traffic_timestamp[['pagecount_desktop_views', 'pagecount_mobile_views',
       'pagecount_all_views', 'pageview_mobile_views',
       'pageview_desktop_views', 'pageview_all_views']].div(1000000)<jupyter_output><empty_output><jupyter_text>convert data types to integers<jupyter_code>df_traffic_timestamp = df_traffic_timestamp.astype({'pagecount_desktop_views':'int64', 'pagecount_mobile_views':'int64',
       'pagecount_all_views':'int64', 'pageview_mobile_views':'int64',
       'pageview_desktop_views':'int64', 'pageview_all_views':'int64'})<jupyter_output><empty_output><jupyter_text>remove zero values to clean up visualization<jupyter_code>drop_rows = df_traffic_timestamp['pagecount_desktop_views']!=0
df_traffic_timestamp_pagecount_desktop = df_traffic_timestamp[drop_rows]
df_traffic_timestamp_pagecount_desktop.head()<jupyter_output><empty_output><jupyter_text>remove zero values to clean up visualization<jupyter_code>drop_rows = df_traffic_timestamp['pagecount_mobile_views']!=0
df_traffic_timestamp_pagecount_mobile = df_traffic_timestamp[drop_rows]
df_traffic_timestamp_pagecount_mobile.head()<jupyter_output><empty_output><jupyter_text>remove zero values to clean up visualization<jupyter_code>drop_rows = df_traffic_timestamp['pagecount_all_views']!=0
df_traffic_timestamp_pagecount_all = df_traffic_timestamp[drop_rows]
df_traffic_timestamp_pagecount_all.head()<jupyter_output><empty_output><jupyter_text>remove zero values to clean up visualization<jupyter_code>drop_rows = df_traffic_timestamp['pageview_all_views']!=0
df_traffic_timestamp_pageview_all = df_traffic_timestamp[drop_rows]
df_traffic_timestamp_pageview_all.head()<jupyter_output><empty_output><jupyter_text>remove zero values to clean up visualization<jupyter_code>drop_rows = df_traffic_timestamp['pageview_mobile_views']!=0
df_traffic_timestamp_pageview_mobile = df_traffic_timestamp[drop_rows]
df_traffic_timestamp_pageview_mobile.head()<jupyter_output><empty_output><jupyter_text>remove zero values to clean up visualization<jupyter_code>drop_rows = df_traffic_timestamp['pageview_desktop_views']!=0
df_traffic_timestamp_pageview_desktop = df_traffic_timestamp[drop_rows]
df_traffic_timestamp_pageview_desktop.head()<jupyter_output><empty_output><jupyter_text>Plot and save the visulaization<jupyter_code>plt.figure(num=None, figsize=(16, 6), dpi=80, facecolor='w', edgecolor='k')
plt.plot( 'timestamp', 'pagecount_desktop_views', data=df_traffic_timestamp_pagecount_desktop, marker='', color='g', linewidth=2, linestyle='--')
plt.plot( 'timestamp', 'pagecount_mobile_views', data=df_traffic_timestamp_pagecount_mobile, marker='', color='b', linewidth=2, linestyle='--')
plt.plot( 'timestamp', 'pagecount_all_views', data=df_traffic_timestamp_pagecount_all, marker='', color='k', linewidth=2, linestyle='--')
plt.plot( 'timestamp', 'pageview_desktop_views', data=df_traffic_timestamp_pageview_desktop, marker='', color='g', linewidth=2, linestyle='-')
plt.plot( 'timestamp', 'pageview_mobile_views', data=df_traffic_timestamp_pageview_mobile, marker='', color='b', linewidth=2, linestyle='-')
plt.plot( 'timestamp', 'pageview_all_views', data=df_traffic_timestamp_pageview_all, marker='', color='k', linewidth=2, linestyle='-')
plt.yticks(np.arange(0, 13000, 2000))
plt.xlabel('May 2015: a new pageview deffinition took effect, which eliminated all crawler taffic. Solid lines mark new deffintion i.e. "pageview" rather than "pagecount"', color='red')  
plt.title("Page Views on English Wikipedia (x 1,000,000)")
plt.legend()

plt.savefig('data512_a1_viz.png')<jupyter_output>/Users/Jack/opt/anaconda3/lib/python3.7/site-packages/pandas/plotting/_matplotlib/converter.py:103: FutureWarning: Using an implicitly registered datetime converter for a matplotlib plotting method. The converter was registered by pandas on import. Future versions of pandas will require you to explicitly register matplotlib converters.

To register the converters:
	>>> from pandas.plotting import register_matplotlib_converters
	>>> register_matplotlib_converters()
  warnings.warn(msg, FutureWarning)
{'fraction_non_alphanumeric': 0.103480183005192, 'fraction_numerical': 0.043438030123888347, 'mean_word_length': 5.783124128312413, 'pattern_counts': {'":': 0, '<': 198, '<?xml version=': 0, '>': 204, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '50', 'dir': 'jupyter-structured-clean-dedup', 'id': '166998', 'license': 'MIT', 'n_tokens_mistral': 6953, 'n_tokens_neox': 6255, 'n_words': 1094, 'path': 'data-512-a1/data-512-a1.ipynb', 'repo_name': 'jfm888/data-512', 'size': '19458'}