<jupyter_start><jupyter_text>Wikipedia politician pages, and bias in data University of Washington, DATA 512 Autumn 2019, Assignment 2 <NAME>In this assignment, we analyze the number and quality of English Wikipedia politician pages relative to country and region population. The two main data sources we use are the Politicians by Country from the English-language Wikipedia dataset and the 2018 World Population Data from the Population Reference Bureau. We process this data, then use the Wikipedia ORES API to estimate the quality of each Wikipedia politician article. Finally, we merge all the data and analyze the trends in relative coverage and quality of articles by country and region, then discuss our findings along with potential sources of bias in the data.First, we will import all the needed packages.<jupyter_code>import csv
import urllib.request
from urllib.request import urlopen
import codecs
import pandas as pd
from io import BytesIO
from zipfile import ZipFile
import oresapi
import numpy as np<jupyter_output><empty_output><jupyter_text>Part 1: Data AcquisitionIn this first step, we download and clean all the necessary input data.The first dataset we read in is the [Politicians by Country from the English-language Wikipedia](https://figshare.com/articles/Untitled_Item/5513449) dataset. This file contains over 400,000 observations, and the variables:   `page`: Wikipedia page title, containing the name of a politician      `country`: Name of the country that the politician worked in      `rev_id`: edit ID of the last edit to the page    We save out a copy of the raw data, then clean it by removing any entries starting with the string "Template:", since these entries are not Wikipedia articles.<jupyter_code># Read in Politicians by Country dataset from website, and parse the resulting csv file
resp = urlopen("https://ndownloader.figshare.com/files/9614893")
zipfile = ZipFile(BytesIO(resp.read()))
file = zipfile.open("country/data/page_data.csv")
csvfile = csv.reader(codecs.iterdecode(file, 'utf-8'))
header = next(csvfile)
page_data = pd.DataFrame(csvfile)
page_data.columns = header

# Save out raw data
page_data.to_csv('../data_raw/page_data.csv', index = False) 

# Drop pages starting with "Template:", which are not Wikipedia articles
page_data = page_data[page_data['page'].str.startswith('Template:') != True]
page_data = page_data.reset_index(drop = True)

page_data.head()<jupyter_output><empty_output><jupyter_text>Next, we read in the Population Reference Bureau's 2018 World Population Data Sheet. This dataset contains two variables:    `Geography`: Country or region measured    `Population mid-2018 (millions)`: Population of the respective location in mid-2018, in millions of people    We downloaded this dataset manually from the [Canvas course site](https://canvas.uw.edu/files/58607571/download?download_frd=1) to the `data_raw` directory. We then remove any entries of locations given in all capital letters, because these are regions rather than countries.Note that there is an updated set of population estimates is available from the [Population Reference Bureau's 2019 World Population Data Sheet](https://www.prb.org/international/indicator/population/table/), which was not used in this assignment. I have included some unused code in the commented out section below that will instead read this dataset from the website, keep only the geographical information and population variables, then save out the formatted raw data.<jupyter_code># Read in 2018 World Population data
populations = pd.read_csv('../data_raw/WPDS_2018_data.csv')
populations.columns = ['country', 'Population mid-2018 (millions)']

## Alternately, read in 2019 World Population data
# ftpstream = urllib.request.urlopen("https://datacenter.prb.org/download/international/indicator/population/csv")
# csvfile = csv.reader(codecs.iterdecode(ftpstream, 'utf-8'))
# populations = pd.DataFrame(csvfile)
#
## Keep only needed rows and columns
# populations = populations.drop([0, 1, 2, 3, 4])
# populations = populations.drop(populations.columns[[0, 2, 3]], axis = 1)
#
## Rename columns
# populations.columns = ['Geography', 'Population mid-2019 (millions)']
#
# populations.to_csv('../data_raw/WPDS_2019_data.csv', index = False) 

# Remove uppercased rows that contain region-level data
populations_country = populations[populations.country != populations.country.str.upper()]
populations_country = populations_country.reset_index(drop = True)

populations_country.head()<jupyter_output><empty_output><jupyter_text>Part 2: Data ProcessingIn this section, we process the population dataset to map each country to its respective region. Then, we use the ORES API to obtain article quality data, and finally merge all our datasets together in preparation for analysis.First, let's create a dataset of countries together with their corresponding region.<jupyter_code># First, extract just the regions
populations_region = populations[populations.country == populations.country.str.upper()]
regions_list = []
regs = populations_region['country'].tolist()

# Extract the indices of the regions dataset, and subtract them to get the number of times each region should be repeated
indices = populations_region.index
last_element = indices[-1]
indices = [(indices[i - 1] - indices[i]) * -1 for i in range(1, len(indices))]
indices.append(len(populations.index) + 1 - last_element)

# Then loop over the regions, and append each to a list a number of times equal to the number of rows belonging to that region in the original populations dataset
for i in range(len(regs)): 
    j = 1
    while j <= indices[i]:
        regions_list.append(regs[i])
        j += 1

# Finally, convert the regions list to a DataFrame, and merge the region data on to the country populations data
regions_df = pd.DataFrame.from_dict(regions_list)
regions_df.columns = ["region"]
populations = populations.merge(regions_df, left_index = True, right_index = True)<jupyter_output><empty_output><jupyter_text>Next, we query the [ORES client](https://github.com/wikimedia/ores), from Wikimedia Foundation and authors <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, available under an MIT License. Documentation pages for the API are available [here](https://www.mediawiki.org/wiki/ORES). We obtain predictions of article quality for each Wikipedia politician page. Note that there are six possible predicted values of article quality: FA (Featured article), GA (Good article), B (B-class article), C (C-class article), Start (Start-class article), Stub (Stub-class article). For the purposes of this assignment, we only consider FA and GA as corresponding to high-quality articles.Note that I was not able to install the `ores` package due to package compatibility errors, but was able to instead use the [`oresapi` package](https://github.com/halfak/oresapi), available from <NAME>, 2019, under an MIT License.<jupyter_code># Start an ORES API session
# Provide this useragent argument for the class to help the ORES team track requests
ores_session = oresapi.Session("https://ores.wikimedia.org", "Class project <<EMAIL>>")

# Obtain predictions of article quality for each revision ID in the politicians dataset
results = ores_session.score("enwiki", ["articlequality"], page_data['rev_id'])
gen_lst = list(results)

# Convert outputs to a dataframe
results2 = pd.DataFrame(columns = ['articlequality'])
for i in gen_lst:
    results2 = results2.append(i, ignore_index = True)
        
# Extract just the predicted article quality
results2['articlequality'] = results2['articlequality'].astype(str).str.slice(start = 26).str.rsplit("'", expand = True)

results2.head()<jupyter_output><empty_output><jupyter_text>Finally, merge all the datasets together, and save out a cleaned dataset for analysis.<jupyter_code># First merge article quality predictions onto politicians pages dataset
all_data = page_data.merge(results2, left_index = True, right_index = True)

# Remove and save out entries for which the ORES API query did not return article quality scores
all_data_noscores = all_data[~all_data['articlequality'].isin(['B', 'C', 'FA', 'GA', 'Stub', 'Start'])]
all_data_noscores.to_csv('../data_clean/wp_wpds_politicians_noscores.csv', index = False)
all_data = all_data[all_data['articlequality'].isin(['B', 'C', 'FA', 'GA', 'Stub', 'Start'])]

# Now merge on country population data
all_data = all_data.merge(populations_country, how = 'outer', indicator = True)

# Output a dataset containing the data that failed to merge - either no country population data, or no politician data
all_data_nomerge = all_data[all_data['_merge'].isin(['left_only', 'right_only'])]
all_data_nomerge.to_csv('../data_clean/wp_wpds_countries-no_match.csv', index = False)

# Save out the final cleaned dataset with all the entries that merged
all_data_fin = all_data[all_data['_merge'] == "both"]
all_data_fin = all_data_fin.rename(columns = {"page": "article_name", "rev_id": "revision_id", "articlequality": 
                                           "article_quality", "Population mid-2018 (millions)": "population"})
all_data_fin = all_data_fin[["country", "article_name", "revision_id", "article_quality", "population"]]
all_data_fin.to_csv('../data_clean/wp_wpds_politicians_by_country.csv', index = False)


# Merge on region data
all_data_fin = all_data_fin.merge(populations)

# Merge on region populations
populations_region.columns = ['region', 'region_population']
all_data_fin = all_data_fin.merge(populations_region)

# Create indicator for whether the article quality is considered high-quality (GA or FA)
all_data_fin['high_quality'] = np.where((all_data_fin['article_quality'] == "GA") | 
                                        (all_data_fin['article_quality'] == "FA"), 1, 0)

# Convert population data type from string to integer
all_data_fin['population'] = pd.to_numeric(all_data_fin['population'].str.replace(',', ''))
all_data_fin['region_population'] = all_data_fin['region_population'].str.replace(',', '').astype(int)

all_data_fin.head()<jupyter_output><empty_output><jupyter_text>Step 3: AnalysisIn this section, we create six tables comparing the relative quantity and quality of Wikipedia polititcal pages relative to population across countries and regions. Finally, we conclude with a writeup of the findings and potential sources of bias in the data.<jupyter_code># TABLE 1: Top 10 countries by coverage
# "10 highest-ranked countries in terms of number of politician articles as a proportion of country population"
articles_country = all_data_fin.groupby(['country', 'population'], as_index = False).count()
articles_country['proportion'] = articles_country['article_name'] / articles_country['population']
articles_country = articles_country.sort_values('proportion', ascending = False)
articles_country2 = articles_country[['country', 'proportion']]
articles_country2.head(10)
# TABLE 2: Bottom 10 countries by coverage
# "10 lowest-ranked countries in terms of number of politician articles as a proportion of country population"
articles_country2 = articles_country2.sort_values('proportion')
articles_country2.head(10)
# TABLE 3: Top 10 countries by relative quality
# "10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality"
articles_qual = all_data_fin.groupby(['country'], as_index = False).sum()
articles_qual = articles_qual.merge(articles_country[['country', 'article_name']])
articles_qual['proportion'] = articles_qual['high_quality'] / articles_qual['article_name']
articles_qual = articles_qual.sort_values('proportion', ascending = False)
articles_qual = articles_qual[['country', 'proportion']]
articles_qual.head(10)
# TABLE 4: Bottom 10 countries by relative quality
# "10 lowest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality"
articles_qual = articles_qual.sort_values('proportion')
articles_qual.head(10)
# TABLE 5: Geographic regions by coverage 
# "Ranking of geographic regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population"
articles_region = all_data_fin.groupby(['region', 'region_population'], as_index = False).count()
articles_region['proportion'] = articles_region['article_name'] / articles_region['region_population']
articles_region = articles_region.sort_values('proportion', ascending = False)
articles_region2 = articles_region[['region', 'proportion']]
articles_region2
# TABLE 6: Geographic regions by coverage
# "Ranking of geographic regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-quality"
region_qual = all_data_fin.groupby(['region'], as_index = False).sum()
region_qual = region_qual.merge(articles_region[['region', 'article_name']])
region_qual['proportion'] = region_qual['high_quality'] / region_qual['article_name']
region_qual = region_qual.sort_values('proportion', ascending = False)
region_qual = region_qual[['region', 'proportion']]
region_qual<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.096580547112462, 'fraction_numerical': 0.012006079027355622, 'mean_word_length': 5.923198316675434, 'pattern_counts': {'":': 4, '<': 39, '<?xml version=': 0, '>': 38, 'https://': 9, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '324325', 'license': 'MIT', 'n_tokens_mistral': 3762, 'n_tokens_neox': 3505, 'n_words': 1327, 'path': 'src/hcds-a2-bias.ipynb', 'repo_name': 'zlavogb/data-512-a2', 'size': '13217'}