<jupyter_start><jupyter_text>Import Packages and DataWe first import the relevant packages as described in the README.<jupyter_code>import requests
import json
import pandas as pd
import os
import csv
import numpy as np

# The following code is motivated from the Stack Overflow post by user emunsing:
#        https://stackoverflow.com/a/29665452/3905509
from IPython.display import display, HTML<jupyter_output><empty_output><jupyter_text>Next, we import the 2 datasets as pandas dataframes:* one representing population estimates from many nations (*data_population*)* one representing political Wikipeia articles from many nations (*data_wiki*)Downstream in this notebook, we will use the ORES API from Wikipedia. To prevent access issues (and more generally to be polite), we partition *data_wiki* into chunks of 100 articles in the list of dataframes *data_wiki_partition*.<jupyter_code># Import WPDS_2018_data.csv
# The following dataset comes from https://www.dropbox.com/s/5u7sy1xt7g0oi2c/WPDS_2018_data.csv?dl=0
data_population = pd.read_csv('WPDS_2018_data.csv', thousands = ',')

# Import page_data.csv
# The following dataset is produced by the R project found here: https://figshare.com/articles/Untitled_Item/5513449
data_wiki = pd.read_csv('page_data.csv')

# Partition data_wiki into small enough chunks to not get blocked by ORES
# The following creates a list of data_wiki partitions. Each list element has 100 records.
data_wiki_partition = np.array_split(data_wiki, data_wiki.shape[0] // 100)<jupyter_output><empty_output><jupyter_text>Scoring Wikipedia Articles with ORESIn order to estimate the quality of Wikipedia articles, we use the ORES API. In the following code block, relevant headers and an api call function, *get_ores_data*, are defined. The scored ORES data is stored in the list of json objects called *data_wiki_ores_json*.To make *data_wiki_ores_json* more usable, we restate it as a dataframe in the object *data_wiki_ores_df*. Lastly, the predicted article qualities are appended to the original *data_wiki* object into the dataframe *data_wiki_ores*.<jupyter_code># The code in this block is reused & adapted from Os Keyes:
#        https://github.com/Ironholds/data-512-a2/blob/master/hcds-a2-bias_demo.ipynb

headers = {'User-Agent' : 'https://github.com/OO00OO00', 'From' : '<EMAIL>'}

# Score Wiki articles with ORES
def get_ores_data(revision_ids, headers):
    
    # Define the endpoint
    endpoint = 'https://ores.wikimedia.org/v3/scores/{project}/?models={model}&revids={revids}'
    
    # Specify the parameters - smushing all the revision IDs together separated by | marks.    
    params = {'project' : 'enwiki',
              'model'   : 'wp10',
              'revids'  : '|'.join(str(x) for x in revision_ids)
              }
    api_call = requests.get(endpoint.format(**params))
    response = api_call.json()
    return response

# Score the chunks of 100 articles in ORES. Return as a list of json output for each partition.
data_wiki_ores_json = [get_ores_data(i['rev_id'].tolist(), headers) for i in data_wiki_partition]

# For each batch and each revision id contained therein (i.e., the keys of batch['enwiki']['scores']):
#     1) Restate the wp10 dictionary value as a pandas dataframe
#     2) Add a column reflecting the revision id
#     3) Extract the relevant prediction field, i.e., the 1st row of the resulting dataframe
#     4) Lastly, concatenate the list of dataframes as a new dataframe and transpose the result
data_wiki_ores_df = pd.concat([pd.DataFrame.from_dict(batch['enwiki']['scores'][revID]['wp10']).assign(rev_id = int(revID)).iloc[0,:] 
                               for batch in data_wiki_ores_json 
                               for revID in batch['enwiki']['scores'].keys()
                              ], axis = 1).transpose()

# Append ORES prediction to data_wiki
data_wiki_ores = pd.merge(data_wiki, data_wiki_ores_df, left_on = 'rev_id', right_on = 'rev_id', how = 'left')<jupyter_output><empty_output><jupyter_text>Combine DataNext, we combine the ORES-scored article dataframe with the population dataframe into a new dataframe called *data*. Some minor changes are performed on it to clarify its operations and analysis downstream in this notebook. Lastly, its results are exported to CSV in the file *wikipedia_political_article_bias_2018.csv*<jupyter_code># Merge dataframes
data = pd.merge(data_population, data_wiki_ores, left_on = 'Geography', right_on = 'country', how = 'inner')

# Rename columns:
data.rename(columns = {'page': 'article_name'}, inplace = True)
data.rename(columns = {'rev_id': 'revision_id'}, inplace = True)
data.rename(columns = {'score': 'article_quality'}, inplace = True)
data.rename(columns = {'Population mid-2018 (millions)': 'population'}, inplace = True)

# Remove duplicate column
data.drop(columns = 'Geography')

# Reorder columns:
data = data.reindex(columns = ['country', 'article_name', 'revision_id', 'article_quality', 'population'])

# Export data to CSV
data.to_csv('wikipedia_political_article_bias_2018.csv', index = False)<jupyter_output><empty_output><jupyter_text>Since we want to calculate the proportion of high quality political articles via the FA and GA ORES values, the following code reshapes the *data* dataframe to expand these levels as new columns and stores the result in the dataframe *data_pivot*. Next, *data_pivot* is aggregated at the country level and additional summary data is calculated. The result is called *data_aggregated*. Lastly, *data_aggregated* is modified to more easily create derived fields downstream in this notebook.<jupyter_code># Expand levels of article_quality column into new columns themselves
#    Motivated by the Stack Overflow response from user DYZ:
#        https://stackoverflow.com/a/42708606
# 1) Use pivot_table on data to dcast levels of article_quality as new columns
# 2) Restate pivot to numpy record array
# 3) Restate numpy record array back to a pandas dataframe
data_pivot = pd.DataFrame(data.pivot_table(index = ['country', 'article_name', 'revision_id', 'population'], columns = 'article_quality', aggfunc=len).to_records())

# Aggregate data to country-level and include relevant summary data
data_aggregated = data_pivot.groupby('country').agg({
    'population': {'max'},
    'revision_id': {'count'},
    'B': {'count'},
    'C': {'count'},
    'FA': {'count'},
    'GA': {'count'},
    'Start': {'count'},
    'Stub': {'count'}
})

# Flatten the multi-index dataframe for ease in deriving new columns
# The following code is motivated by the Stack Overflow response from user <NAME>:
#        https://stackoverflow.com/a/14508355
data_aggregated.columns = data_aggregated.columns.get_level_values(0)

# For clarity, renaming revision_id to article_count
data_aggregated.rename(columns = {'revision_id' : 'article_count'}, inplace = True)<jupyter_output><empty_output><jupyter_text>Two fields are derived:* articles_per_population: the proportion of articles-per-population for each nation* pct_high_quality_articles: the proportion of high quality articles for each nation, where "high quality" means proportion of articles that have the ORES prediction quality of *FA* or *GA*.<jupyter_code># Create derived fields
data_aggregated['articles_per_population'] = data_aggregated['article_count'] / (data_aggregated['population'] * 1e6)
data_aggregated['pct_high_quality_articles'] = (data_aggregated['FA'] + data_aggregated['GA']) / data_aggregated['article_count']<jupyter_output><empty_output><jupyter_text>AnalysisLastly, for this notebook, we create 4 tables:* 10 highest-ranked countries in terms of number of politician articles as a proportion of country population* 10 lowest-ranked countries in terms of number of politician articles as a proportion of country population* 10 highest-ranked countries in terms of number of GA and FA-quality articles as a proportion of all articles about politicians from that country* 10 lowest-ranked countries in terms of number of GA and FA-quality articles as a proportion of all articles about politicians from that countryNote: The wording for the tables is taken from here: https://wiki.communitydata.cc/Human_Centered_Data_Science_(Fall_2018)/AssignmentsA2:_Bias_in_data<jupyter_code>print('10 highest-ranked countries in terms of number of politician articles as a proportion of country population:')
display(data_aggregated.sort_values(by = ['articles_per_population'], ascending = False).iloc[0:10,[0, 1, -2]])

print('10 lowest-ranked countries in terms of number of politician articles as a proportion of country population:')
display(data_aggregated.sort_values(by = ['articles_per_population'], ascending = True).iloc[0:10,[0, 1, -2]])

print('10 highest-ranked countries in terms of number of GA and FA-quality articles as a proportion of all articles about politicians from that country:')
display(data_aggregated.sort_values(by = ['pct_high_quality_articles'], ascending = False).iloc[0:10,[1, 4, 5, -1]])

print('10 lowest-ranked countries in terms of number of GA and FA-quality articles as a proportion of all articles about politicians from that country:')
display(data_aggregated.sort_values(by = ['pct_high_quality_articles', 'article_count', 'population'], ascending = True).iloc[0:10,[0, 1, 4, 5, -1]])<jupyter_output>10 highest-ranked countries in terms of number of politician articles as a proportion of country population:
{'fraction_non_alphanumeric': 0.09529374269627111, 'fraction_numerical': 0.01731647721236588, 'mean_word_length': 5.213861386138614, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 9, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '302620', 'license': 'MIT', 'n_tokens_mistral': 2845, 'n_tokens_neox': 2659, 'n_words': 974, 'path': 'hcds-a2-bias.ipynb', 'repo_name': 'OO00OO00/data-512-a2', 'size': '9424'}