<jupyter_start><jupyter_text>Assignment 2: Bias in Data The goal of this assignment is to explore the concept of bias through data on Wikipedia articles - specifically, articles on political figures from a variety of countries. For this assignment, we will combine a dataset of Wikipedia articles with a dataset of country populations, and use a machine learning service called ORES to estimate the quality of each article.We will perform an analysis on how the coverage of politicians on Wikipedia and the quality of articles about politicians varies between countries. The analysis will consist of a series of tables that show:- the countries with the greatest and least coverage of politicians on Wikipedia compared to their population.- the countries with the highest and lowest proportion of high quality articles about politicians.- a ranking of geographic regions by articles-per-person and proportion of high quality articles.  Import Libraries<jupyter_code>import os
import requests
from urllib.parse import urlencode

import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Step 1: Getting the Article and Population data The first step is getting the data, which lives in several different places. The Wikipedia politicians by country dataset can be found on Figshare. - download the zipped folder manually from the link https://figshare.com/articles/dataset/Untitled_Item/5513449 - unzip it in our data/source folderThe population data is available in CSV format as WPDS_2020_data.csv. This dataset is drawn from the world population data sheet published by the Population Reference Bureau.<jupyter_code>df_cd = pd.read_csv('../data/source/page_data.csv')
df_wpd = pd.read_csv('../data/source/WPDS_2020_data.csv')
df_cd.head(5)
df_wpd.head(5)<jupyter_output><empty_output><jupyter_text>Step 2: Data Cleaning Both page_data.csv and WPDS_2020_data.csv contain some rows that need to be filtered out:page_data.csv:- the dataset contains some page names that start with the string "Template:". These pages are not Wikipedia articles, and should not be included in the analysisWPDS_2020_data.csv:- contains some rows that provide cumulative regional population counts, rather than country-level counts. These rows are distinguished by having ALL CAPS values in the 'geography' field (e.g. AFRICA, OCEANIA)<jupyter_code>df_cd = df_cd[df_cd["page"].str.contains("Template:")==False]

df_wpd_country = df_wpd[df_wpd['Name'].str.isupper() == False] # Country-level 
df_wpd_region = df_wpd[df_wpd['Name'].str.isupper()] # Cumulative region level
df_cd.head(5)
df_wpd_country.head(5)
df_wpd_region.head(5)<jupyter_output><empty_output><jupyter_text>Step 3: Getting Article Quality Predictions Now we need to get the predicted quality scores for each article in the Wikipedia dataset. We're using a machine learning system called ORES. This was originally an acronym for "Objective Revision Evaluation Service" but was simply renamed “ORES”. ORES is a machine learning tool that can provide estimates of Wikipedia article quality. The article quality estimates are, from best to worst:- FA - Featured article- GA - Good article- B - B-class article- C - C-class article- Start - Start-class article- Stub - Stub-class articleThese were learned based on articles in Wikipedia that were peer-reviewed using the Wikipedia content assessment procedures. These quality classes are a sub-set of quality assessment categories developed by Wikipedia editors. In order to get article predictions for each article in the Wikipedia dataset, we will first need to read page_data.csv and then read through the dataset line by line, using the value of the rev_id column to make an API query.<jupyter_code>headers = {
    'User-Agent': 'https://github.com/KrishaMehta98',
    'From': '<EMAIL>'
}
def api_call(rev_ids):
    endpoint = 'https://ores.wikimedia.org/v3/scores/{project}/?models={model}&revids={revids}'
    
    parameters ={'project' : 'enwiki',
                 'model'   : 'articlequality',
                 'revids'   : '|'.join(str(x) for x in rev_ids)
                }
    call = requests.get(endpoint.format(**parameters), headers = headers)
    response = call.json()
    return response
rev_ids = df_cd['rev_id'].tolist()
revids = []
scores = []
missing_revids = []

for i in range(0, len(rev_ids), 50):
    
    if i+50>len(rev_ids):
        ids = rev_ids[i:len(rev_ids)]
    else:
        ids = rev_ids[i:i+50]
    
    response = api_call(ids)
    
    for revid in ids:
        if response['enwiki']['scores'][str(revid)]['articlequality'].get('score') is not None:
            revids.append(revid)
            scores.append(response['enwiki']['scores'][str(revid)]['articlequality']['score']['prediction'])
        else:
            missing_revids.append(revid)
revid_score = pd.DataFrame([revids, scores]).T
revid_score.columns = ['rev_id', 'prediction']
revid_score.revision_id = revid_score.rev_id.astype(int)
revid_score.to_csv('../data/final/wikipedia-politician-article-quality.csv', index=False)


revid_missing = pd.DataFrame([missing_revids]).T
revid_missing.columns = ['revid']
revid_missing.to_csv('../data/final/revids_missing.csv', index=False)<jupyter_output><ipython-input-15-579e4e19dae0>:3: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
  revid_score.revision_id = revid_score.rev_id.astype(int)<jupyter_text>Step 4: Combining the data sets Some processing of the data needs to be done. In particular, we'll have to - after retrieving and including the ORES data for each article - merge the wikipedia data and population data together. Both have fields containing country names for just that purpose. After merging the data, we'll invariably run into entries which cannot be merged. Either the population dataset does not have an entry for the equivalent Wikipedia country, or vise versa.Removing any rows that do not have matching data, and output them to a CSV file called: wp_wpds_countries-no_match.csvConsolidate the remaining data into a single CSV file called: wp_wpds_politicians_by_country.csv<jupyter_code>wpd_cd = df_cd.merge(df_wpd, how='outer', left_on='country', right_on='Name')
wpd_cd.head(5)
missing_wpd_cd = wpd_cd.loc[(wpd_cd['country'].isnull() | wpd_cd['Name'].isnull())]
missing_wpd_cd.to_csv('../data/final/wp_wpds_countries-no_match.csv', index = False)
wp_wpds  = df_cd.merge(df_wpd, how='inner', left_on='country', right_on='Name')
wp_wpds = wp_wpds.merge(revid_score, on='rev_id')
wp_wpds = wp_wpds.drop(columns=['FIPS', 'Name', 'Type', 'TimeFrame', 'Data (M)'])
wp_wpds = wp_wpds.rename(index=str, columns={'page': 'article_name', 'rev_id': 'revision_id', 'Population':'population', 'prediction':'article_quality_est'})
wp_wpds.to_csv('../data/final/wp_wpds_politicians_by_country.csv', index = False)
wp_wpds.head(5)<jupyter_output><empty_output><jupyter_text>Step 5: Analysis The analysis will consist of calculating the proportion (as a percentage) of articles-per-population and high-quality articles for each country AND for each geographic region. By "high quality" articles, in this case we mean the number of articles about politicians in a given country that ORES predicted would be in either the "FA" (featured article) or "GA" (good article) classes.<jupyter_code>wp_wpds_per_country = pd.pivot_table(wp_wpds, fill_value=0,columns=['article_quality_est'],aggfunc={'article_quality_est': len,},index=['country'])
wp_wpds_per_country.columns = wp_wpds_per_country.columns.droplevel() 
wp_wpds_per_country = wp_wpds_per_country.reset_index()
wp_wpds_per_country.columns.name = None
wp_wpds_per_country = wp_wpds_per_country.merge(wp_wpds.groupby(['country'])['population'].mean(), left_on='country', right_index=True)
wp_wpds_per_country['total_articles'] = wp_wpds_per_country['FA'] + wp_wpds_per_country['GA'] + wp_wpds_per_country['B'] + wp_wpds_per_country['C'] + wp_wpds_per_country['Stub'] + wp_wpds_per_country['Start']
wp_wpds_per_country['total_high_quality_articles'] = wp_wpds_per_country['FA'] + wp_wpds_per_country['GA']
wp_wpds_per_country['articles_per_population_percent'] = (wp_wpds_per_country['total_articles'] / wp_wpds_per_country['population']) * 100
wp_wpds_per_country['high_quality_articles_percent'] = (wp_wpds_per_country['total_high_quality_articles'] / wp_wpds_per_country['total_articles']) * 100

wp_wpds_per_country.head(5)
region = "NORTHERN AFRICA"

regions = ['WORLD', 'AFRICA', 'NORTHERN AFRICA']
for i in range(3, len(df_wpd)):
    if df_wpd.iloc[i]['Type'] == 'Sub-Region':
        region = df_wpd.iloc[i]['Name']
    regions.append(region)

df_wpd['Region'] = regions
wp_wpds_per_region = pd.merge(left=wp_wpds,right=df_wpd,left_on='country',right_on='Name',how='left')
wp_wpds_per_region = pd.merge(left=wp_wpds_per_region,right=df_wpd_region,left_on='Region',right_on='Name',how='left')
wp_wpds_per_region = wp_wpds_per_region[['Region', 'country', 'article_name', 'revision_id', 'article_quality_est', 'Population_y', 'population']]
wp_wpds_per_region.rename(columns={'Region': 'region', 'Population_y': 'region_population','population': 'country_population'}, inplace=True)
wp_wpds_per_region.dropna(subset=['region_population'], inplace=True)
wp_wpds_per_region.head(5)
wp_wpds_per_region_pivot = pd.pivot_table(wp_wpds_per_region,fill_value=0,columns=['article_quality_est'],aggfunc={'article_quality_est': len, },index=['region'])
wp_wpds_per_region_pivot.columns = wp_wpds_per_region_pivot.columns.droplevel() #clean up multilevel index
wp_wpds_per_region_pivot = wp_wpds_per_region_pivot.reset_index()
wp_wpds_per_region_pivot.columns.name = None
wp_wpds_per_region_pivot = pd.merge(left=wp_wpds_per_region_pivot,right=wp_wpds_per_region.groupby(['region'])['region_population'].mean(), left_on='region', right_index=True)
wp_wpds_per_region_pivot['total_articles'] = wp_wpds_per_region_pivot['FA'] + wp_wpds_per_region_pivot['GA'] + wp_wpds_per_region_pivot['B'] + wp_wpds_per_region_pivot['C'] + wp_wpds_per_region_pivot['Stub'] + wp_wpds_per_region_pivot['Start']
wp_wpds_per_region_pivot['total_high_quality_articles'] = wp_wpds_per_region_pivot['FA'] + wp_wpds_per_region_pivot['GA']
wp_wpds_per_region_pivot['articles_per_population_percent'] = (wp_wpds_per_region_pivot['total_articles'] / wp_wpds_per_region_pivot['region_population']) * 100
wp_wpds_per_region_pivot['high_quality_articles_percent'] = (wp_wpds_per_region_pivot['total_high_quality_articles'] / wp_wpds_per_region_pivot['total_articles']) * 100
wp_wpds_per_region_pivot.head(5)<jupyter_output><empty_output><jupyter_text>Step 6: Results  1. Top 10 countries by coverage: 10 highest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>wp_wpds_per_country.sort_values(by=['articles_per_population_percent'], ascending=False).head(10)[['country', 'articles_per_population_percent']]<jupyter_output><empty_output><jupyter_text>2. Bottom 10 countries by coverage: 10 lowest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>wp_wpds_per_country.sort_values(by=['articles_per_population_percent'], ascending=True).head(10)[['country', 'articles_per_population_percent']]<jupyter_output><empty_output><jupyter_text>3. Top 10 countries by relative quality: 10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>wp_wpds_per_country.sort_values(by=['high_quality_articles_percent'], ascending=False).head(10)[['country', 'high_quality_articles_percent']]<jupyter_output><empty_output><jupyter_text>4. Bottom 10 countries by relative quality: 10 lowest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>wp_wpds_per_country.sort_values(by=['high_quality_articles_percent'], ascending=True).head(10)[['country', 'high_quality_articles_percent']]<jupyter_output><empty_output><jupyter_text>5. Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the total count of politician a<jupyter_code>wp_wpds_per_region_pivot.sort_values(by=['articles_per_population_percent'], ascending=False)[['region', 'articles_per_population_percent']]<jupyter_output><empty_output><jupyter_text>6. Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-quality<jupyter_code>wp_wpds_per_region_pivot.sort_values(by=['high_quality_articles_percent'], ascending=False)[['region', 'articles_per_population_percent']]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10513141426783479, 'fraction_numerical': 0.008213391739674593, 'mean_word_length': 6.945929148539466, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 51, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '123153', 'license': 'MIT', 'n_tokens_mistral': 4160, 'n_tokens_neox': 3997, 'n_words': 1037, 'path': 'code/hcds-a2-bias.ipynb', 'repo_name': 'KrishaMehta98/data-512-a2', 'size': '12792'}