<jupyter_start><jupyter_text>Step 1: Data acquisition - In order to measure Wikipedia traffic from 2008-2019, we will need to collect data from two different API endpoints, the Legacy Pagecounts API (December 2007 through July 2016) and the Pageviews API(July 2015 through September 2019).- For each API, we will need to collect data for all months where data is available and then save the raw results into 5 separate JSON source data files (one file per API query type).- The Pageview API (but not the Pagecount API) allows us to filter by agent=user to filter our web crawlers.<jupyter_code># Importing Libraries
import json
import requests
import csv
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# APIs

# pagecount
endpoint_legacy = 'https://wikimedia.org/api/rest_v1/metrics/legacy/pagecounts/aggregate/{project}/{access-site}/{granularity}/{start}/{end}'
# pageview
endpoint_pageviews = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/{project}/{access}/{agent}/{granularity}/{start}/{end}'
"""
    Pagecount API
    parameters for getting aggregated legacy view data
    see: https://wikimedia.org/api/rest_v1/#/ for details 
"""

#all-sites
example_params_legacy1 = {"project" : "en.wikipedia.org",
                 "access-site" : "all-sites",
                 "granularity" : "monthly",
                 "start" : "2007120100",
                # for end use 1st day of month following final month of data
                 "end" : "2016070100"
                    }
#mobile-site
example_params_legacy2 = {"project" : "en.wikipedia.org",
                 "access-site" : "mobile-site",
                 "granularity" : "monthly",
                 "start" : "2007120100",
                # for end use 1st day of month following final month of data
                 "end" : "2016070100"
                    }

#desktop-site
example_params_legacy3 = {"project" : "en.wikipedia.org",
                 "access-site" : "desktop-site",
                 "granularity" : "monthly",
                 "start" : "2007120100",
                # for end use 1st day of month following final month of data
                 "end" : "2016070100"
                    }

"""
  Pageview API
  see: https://wikimedia.org/api/rest_v1/#/ for details
"""
#mobile-app
example_params_pageviews4 = {"project" : "en.wikipedia.org",
                    "access" : "mobile-app",
                    "agent" : "user",
                    "granularity" : "monthly",
                    "start" : "2015070100",
                    # for end use 1st day of month following final month of data
                    "end" : '2019091000'
                        }

#mobile-web
example_params_pageviews5 = {"project" : "en.wikipedia.org",
                    "access" : "mobile-web",
                    "agent" : "user",
                    "granularity" : "monthly",
                    "start" : "2015070100",
                    # for end use 1st day of month following final month of data
                    "end" : '2019091000'
                        }

# Customized headers with my information
headers = {
    'User-Agent': 'https://github.com/ishna',
    'From': '<EMAIL>'
}
# Function to make the API call
def api_call(endpoint,parameters):
    call = requests.get(endpoint.format(**parameters), headers=headers)
    response = call.json()
    # loads the json file
    return response<jupyter_output><empty_output><jupyter_text>Making the 5 json file calls and storing them in given format<jupyter_code># All monthly pagecount data
all_monthly_legacy = api_call(endpoint_legacy, example_params_legacy1)
with open("pagecounts_all-sites_200712-201607.json", "w") as f:
    #Write it to file
    json.dump(all_monthly_legacy, f)
# Monthly mobile pagecount data
mobile_monthly_legacy = api_call(endpoint_legacy, example_params_legacy2)
with open("pagecounts_mobile-site_200712-201607.json", "w") as f:
    #Write it to file
    json.dump(mobile_monthly_legacy, f)
# Monthly desktop pagecount data
desktop_monthly_legacy = api_call(endpoint_legacy, example_params_legacy3)
with open("pagecounts_desktop-site_200712-201607.json", "w") as f:
    #Write it to file
    json.dump(desktop_monthly_legacy, f)
# Monthly mobile pageview data
mobileapp_monthly_pageview = api_call(endpoint_pageviews, example_params_pageviews4)
with open("pageview_mobile-app_201507-201909.json", "w") as f:
    #Write it to file
    json.dump(mobileapp_monthly_pageview, f)
# Monthly desktop mobile pageview data
mobileweb_monthly_pageview = api_call(endpoint_pageviews, example_params_pageviews5)
with open("pageview_mobile-web_201507-201909.json", "w") as f:
    #Write it to file
    json.dump(mobileweb_monthly_pageview, f)<jupyter_output><empty_output><jupyter_text>Step 2: Data processing We will need to perform a series of processing steps on these data files in order to prepare them for analysis. - For data collected from the Pageviews API, we combine the monthly values for mobile-app and mobile-web to create a total mobile traffic count for each month.- For all data, separate the value of timestamp into four-digit year (YYYY) and two-digit month (MM) and discard values for day and hour (DDHH).- For all months with 0 pageviews for a given access method (e.g. desktop-site, mobile-app), that value for that (column, month) should be listed as 0- Combine all data into a single CSV file   Converting the json to pandas dataframe<jupyter_code>def convert_to_pd(json_file):
    items = json_file['items']
    testdf = pd.DataFrame(items)
    # Seperating month and year 
    testdf["month"] = testdf['timestamp'].astype(str).str[4:6]
    testdf["year"] = testdf['timestamp'].astype(str).str[0:4]
    return testdf
# All legacy monthly data
all_monthly_legacy_pd = convert_to_pd(all_monthly_legacy)
all_monthly_legacy_pd = all_monthly_legacy_pd.rename(columns={'count': 'pagecount_all_views'})
all_monthly_legacy_pd = all_monthly_legacy_pd.drop(['access-site', 'granularity', "project", "timestamp"], axis=1)

# All desktop monthly data
desktop_monthly_legacy_pd = convert_to_pd(desktop_monthly_legacy)
desktop_monthly_legacy_pd = desktop_monthly_legacy_pd.rename(columns={'count': 'pagecount_desktop_views'})
desktop_monthly_legacy_pd = desktop_monthly_legacy_pd.drop(['access-site', 'granularity', "project", "timestamp"], axis=1)

# All mobile monthly data
mobile_monthly_legacy_pd = convert_to_pd(mobile_monthly_legacy)
mobile_monthly_legacy_pd = mobile_monthly_legacy_pd.rename(columns={'count': 'pagecount_mobile_views'})
mobile_monthly_legacy_pd = mobile_monthly_legacy_pd.drop(['access-site', 'granularity', "project", "timestamp"], axis=1)<jupyter_output><empty_output><jupyter_text>Merging data from legacy counts<jupyter_code>legacy_count1 = pd.merge(all_monthly_legacy_pd , desktop_monthly_legacy_pd, how='left', on=['year', 'month'])
legacy_count = pd.merge(legacy_count1 , mobile_monthly_legacy_pd, how='left', on=['year', 'month'])
# All page count data
legacy_count<jupyter_output><empty_output><jupyter_text>Merging data from pageviews API<jupyter_code># Mobile-app monthly data
mobileapp_monthly_pageview_pd = convert_to_pd(mobileapp_monthly_pageview)
mobileapp_monthly_pageview_pd = mobileapp_monthly_pageview_pd.rename(columns={'views': 'pageview_mobile_views'})
mobileapp_monthly_pageview_pd = mobileapp_monthly_pageview_pd.drop(['agent', 'access', 'granularity', "project", "timestamp"], axis=1)

# Mobile-web monthly data
mobileweb_monthly_pageview_pd = convert_to_pd(mobileweb_monthly_pageview)
mobileweb_monthly_pageview_pd = mobileweb_monthly_pageview_pd.rename(columns={'views': 'pageview_desktop_views'})
mobileweb_monthly_pageview_pd = mobileweb_monthly_pageview_pd.drop(['agent','access', 'granularity', "project", "timestamp"], axis=1)<jupyter_output><empty_output><jupyter_text>Merging data from pageview API<jupyter_code>page_view = pd.merge(mobileapp_monthly_pageview_pd , mobileweb_monthly_pageview_pd, how='left', on=['year', 'month'])
# Total views = sum of mobile + site
page_view['pageview_all_views'] = page_view['pageview_mobile_views'] + page_view['pageview_desktop_views']

# All pageview data
page_view<jupyter_output><empty_output><jupyter_text>Combining all data<jupyter_code># Outerjoin on combined tables
overall = pd.merge(legacy_count , page_view, how='outer', on=['year', 'month'])
overall = overall.fillna(0)
overall<jupyter_output><empty_output><jupyter_text>Pushing this to csv file:<jupyter_code>overall.to_csv('en-wikipedia_traffic_200712-201809.csv')<jupyter_output><empty_output><jupyter_text>Step 3: Analysis Visualize the dataset you have created as a time series graph.Our visualization will track three traffic metrics: mobile traffic, desktop traffic, and all traffic (mobile + desktop).<jupyter_code># Adding columns for total, mobile and desktop views
overall["mobile"] = overall["pagecount_mobile_views"] + overall ["pageview_mobile_views"]
overall ["desktop"] = overall["pagecount_desktop_views"] + overall ["pageview_desktop_views"]
overall ["total"] = overall["mobile"] + overall ["desktop"]
overall<jupyter_output><empty_output><jupyter_text>Plotting the graphs and saving the visualization<jupyter_code># Make a data frame
df=pd.DataFrame({'x': range(1,11), 'y1': np.random.randn(10), 'y2': np.random.randn(10)+range(1,11), 'y3': np.random.randn(10)+range(11,21), 'y4': np.random.randn(10)+range(6,16), 'y5': np.random.randn(10)+range(4,14)+(0,0,0,0,0,0,0,-3,-8,-6), 'y6': np.random.randn(10)+range(2,12), 'y7': np.random.randn(10)+range(5,15), 'y8': np.random.randn(10)+range(4,14), 'y9': np.random.randn(10)+range(4,14), 'y10': np.random.randn(10)+range(2,12) })
 
# style
plt.style.use('seaborn-darkgrid')
 
# create a color palette
palette = plt.get_cmap('Set1')
df = overall[overall.columns[-3:]]

# multiple line plot
num=0
for column in df:
    num+=1
    plt.plot(overall['year'], df[column], marker='', color=palette(num), linewidth=1, alpha=0.9, label=column)
    # Add legend
    plt.legend(loc=2, ncol=2)
 
    # Add titles
    plt.title("mobile and desktop total views", loc='left', fontsize=12, fontweight=0, color='orange')
    plt.xlabel("year")
    plt.ylabel("count/view")
    plt.savefig('testplot.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11838889432007038, 'fraction_numerical': 0.031088082901554404, 'mean_word_length': 4.494092373791622, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '611106', 'license': 'MIT', 'n_tokens_mistral': 3534, 'n_tokens_neox': 3281, 'n_words': 789, 'path': 'hcds-a1-data-curation.ipynb', 'repo_name': 'ishna/data-512-a1', 'size': '10235'}