<jupyter_start><jupyter_text>Human-Centrerd Data Science ([HCDS](https://www.mi.fu-berlin.de/en/inf/groups/hcc/teaching/winter_term_2020_21/course_human_centered_data_science.html)) - Winter Term 2020/21 - [HCC](https://www.mi.fu-berlin.de/en/inf/groups/hcc/index.html) | [Freie Universität Berlin](https://www.fu-berlin.de/)*** A2 - Reproducibility Workflow  Step 1️⃣: Data acquisitionIn order to measure Wikipedia traffic from January 2008 until October 2020, you will need to collect data from two different APIs:1. The **Legacy Pagecounts API** ([documentation](https://wikitech.wikimedia.org/wiki/Analytics/AQS/Legacy_Pagecounts), [endpoint](https://wikimedia.org/api/rest_v1/!/Pagecounts_data_(legacy)/get_metrics_legacy_pagecounts_aggregate_project_access_site_granularity_start_end)) provides access to desktop and mobile traffic data from December 2007 through July 2016.1. The **Pageviews API** ([documentation](https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews), [endpoint](https://wikimedia.org/api/rest_v1/!/Pageviews_data/get_metrics_pageviews_aggregate_project_access_agent_granularity_start_end)) provides access to desktop, mobile web, and mobile app traffic data from July 2015 through last month.For each API, you will need to collect data for all months where data is available and then save the raw results into five (3+2) separate `JSON`files (one file per API query type) before continuing to step 2. The first step includes to prepare the queries, call the API endpoints and write the JSON files to the folder "data_raw"<jupyter_code>import json
import requests

legacy_desktop = 'https://wikimedia.org/api/rest_v1/metrics/legacy/pagecounts/aggregate/en.wikipedia.org/desktop-site/monthly/2007120100/2016070100'
legacy_mobile = 'https://wikimedia.org/api/rest_v1/metrics/legacy/pagecounts/aggregate/en.wikipedia.org/mobile-site/monthly/2007120100/2016070100'
pageviews_desktop = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/en.wikipedia.org/desktop/user/monthly/2015070100/2020111400'
pageviews_m_web = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/en.wikipedia.org/mobile-web/user/monthly/2015070100/2020110100'
pageviews_m_app = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/en.wikipedia.org/mobile-app/user/monthly/2015070100/2020110100'

headers = {
    'User-Agent': 'https://github.com/Francosinus',
    'From': '<EMAIL>'
}

def api_call(endpoint):
    call = requests.get(endpoint, headers=headers)
    response = call.json()
    
    return response

def write_to_folder(file,name):
    with open(str(name), 'w') as f:
        json.dump(file, f)
list_website = [legacy_desktop,legacy_mobile,pageviews_desktop,pageviews_m_web,pageviews_m_app]<jupyter_output><empty_output><jupyter_text>Save json files into folder "raw_data"<jupyter_code>path ='data_raw/'
names = ["legacy_desktop-site_200712-201607", "legacy_mobile-site_200712-201607","pagecounts_desktop-site_20150701-20201101","pagecounts_mobile-web_20150701-20201101","pagecounts_mobile-app_20150701-20201101"]


for url,name in zip(list_website,names):
    file = api_call(url)
    write_to_folder(file,path+name+".json")<jupyter_output><empty_output><jupyter_text>Step 2: Data processingYou will need to perform a series of processing steps on these data files in order to prepare them for analysis. These steps must be followed exactly in order to prepare the data for analysis. At the end of this step, you will have a single `CSV`-formatted data file `en-wikipedia_traffic_200712-202010.csv` that can be used in your analysis (step 3) with no significant additional processing.* For data collected from the Pageviews API, combine the monthly values for `mobile-app` and `mobile-web` to create a total mobile traffic count for each month.* For all data, separate the value of `timestamp` into four-digit year (`YYYY`) and two-digit month (`MM`) and discard values for day and hour (`DDHH`).Combine all data into a single CSV file with the following headers:| year | month |pagecount_all_views|pagecount_desktop_views|pagecount_mobile_views|pageview_all_views|pageview_desktop_views|pageview_mobile_views||------| ------|-------------------|-----------------------|----------------------|------------------|----------------------|---------------------|| YYYY | MM    |num_views          |num_views              |num_views             |num_views         |num_views             |num_views            |  The prepared JSON files are again loaded from the data_raw folder and converted to a pandas dataframe to do further preprocessing steps. For that the files are loaded in loop and afterwards converted to dataframes which are written into a list. All dataframes are combined to a single dataframe.<jupyter_code>import pandas as pd
import numpy as np
dfs = [] # an empty list to store the data frames
for file in names:
    f = open(path+file+".json")
    d = json.load(f)
    data = pd.json_normalize(d, 'items') # read data frame from json file
    dfs.append(data) # append the data frame to the list

df = pd.concat(dfs,ignore_index=True)
df<jupyter_output><empty_output><jupyter_text>Since the two wikipedia APIs use different naming, we have to rename the columns and combine them afterwards.<jupyter_code>df.columns = df.columns.str.replace('access-site', 'access')
df.columns = df.columns.str.replace('count', 'views')
s=df.stack()
df = s.unstack()
df['access'] = df['access'].str.replace('mobile-web','mobile-new')
df['access'] = df['access'].str.replace('mobile-app','mobile-new')
df<jupyter_output><empty_output><jupyter_text>Since the timestamp is not in a valid format, the next step is to convert it to pandas timestamp.<jupyter_code>df['timestamp'] =  pd.to_datetime(df['timestamp'], format='%Y%m%d%H')
df['month'] = df['timestamp'].dt.month
df['year'] = df['timestamp'].dt.year
df=df.drop(['timestamp'], axis=1)
df<jupyter_output><empty_output><jupyter_text>Since we want to know how many pageviews from which device were made, we have to pivot the table. This makes it possible to group by year and month and sum up the number of views for the different access devices (legacy and new).<jupyter_code>tm=pd.pivot_table(df,index=["year","month"],values=["views"],
               columns=["access"],aggfunc=[np.sum])
tm<jupyter_output><empty_output><jupyter_text>The table has now a multilevel index and multicolumn levels. We drop two column levels and reset the index to have a simple dataframe again.<jupyter_code>tm.columns = tm.columns.droplevel(0)
tm.columns = tm.columns.droplevel(0)
tm.columns.name = None               

tm.reset_index(inplace=True)  
tm<jupyter_output><empty_output><jupyter_text>Now it's time to sum up the total pageviews.<jupyter_code>tm['pagecount_all_views'] = tm.loc[:,['desktop-site','mobile-site']].sum(axis=1)
tm['pageviews_all_views'] = tm.loc[:,['desktop','mobile-new']].sum(axis=1)
tm
tm = tm.rename(columns={'desktop': 'pageviews_desktop_views', 'mobile-new': 'pageviews_mobile_views', 'desktop-site':'pagecount_desktop_views','mobile-site':'pagecount_mobile_views'})
tm
tm.to_csv("data_clean/en-wikipedia_traffic_200712-202010.csv")<jupyter_output><empty_output><jupyter_text>Step 3: AnalysisThe data is nicely preprocessed and can now be plotted.<jupyter_code>import matplotlib.pyplot as plt

import seaborn as sns
pl = tm
pl.set_index(pd.to_datetime({
    'day':1,
    'month': pl.pop('month'),
    'year': pl.pop('year')
}), inplace=True)

cols = ["pageviews_all_views","pagecount_all_views"]
pl[cols] = pl[cols].replace({0:np.nan})
fig, ax = plt.subplots(figsize=(20, 10))


desk,=ax.plot(pl.index.values,
       pl['pageviews_desktop_views'],
       color='blue')
mobile,=ax.plot(pl.index.values,
       pl['pageviews_mobile_views'],
       color='red')
alle,=ax.plot(pl.index.values,
       pl['pageviews_all_views'],
       color='black')
ax.plot(pl.index.values,
       pl['pagecount_mobile_views'],
       color='red',
       linestyle="--")
ax.plot(pl.index.values,
       pl['pagecount_desktop_views'],
       color='blue',
       linestyle="--")
ax.plot(pl.index.values,
       pl['pagecount_all_views'],
       color='black',
       linestyle='--')



ax.set(xlabel="Year",
       ylabel="Views (Billion)",
       title="Page Views on English Wikipedia")
plt.legend([desk,mobile,alle],['Desktop Traffic','Mobile Traffic','All Traffic'])
plt.savefig("wiki.png")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13910667617011166, 'fraction_numerical': 0.030411023996198622, 'mean_word_length': 4.854659248956884, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 13, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '273153', 'license': 'MIT', 'n_tokens_mistral': 2898, 'n_tokens_neox': 2658, 'n_words': 646, 'path': 'A2_ReproducibilityWorkflow.ipynb', 'repo_name': 'Francosinus/A2-hcds-hcc', 'size': '8436'}