<jupyter_start><jupyter_text>Assignment 2  Part 1: AcquisitionThe goal of the following assignment is to explore the concept of bias through data on Wikipedia articles about political figures from a variety of countries. In this first step, we acquire all data needed to perform the analysis. The first data import is for a file "page_data.csv" that contains a list of articles on wikipedia for world politicians, their associated countries, and ids to lookup the articles. Next is an import of "WPDS_2018.data.csv" that has the population data for countries of the world in 2018 as well as for overall world regions.<jupyter_code># Import packages for dataframe manipulation and viewing all code output in-notebook.
import pandas as pd
import numpy as np
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
# Import data and peek at heads for sanity check.
page_data = pd.read_csv('page_data.csv')
pop_data = pd.read_csv('WPDS_2018_data.csv')
page_data.head()
pop_data.head()<jupyter_output><empty_output><jupyter_text>In the next few cells, I filter out rows from both dataframes that don't match up with valid politician articles from valid countries.<jupyter_code># Filter out pages that have "Template:" in the title.
page_filtered = page_data[~page_data['page'].str.contains("Template")]
page_filtered.head()
# Filter out population rows for geographic regions listed in uppercase (as opposed to countries).
pop_filtered = pop_data[~pop_data['Geography'].str.isupper()]
pop_filtered.head()<jupyter_output><empty_output><jupyter_text>In the following code I import the ORES API which I'll use to query for article quality values.  Beforehand I ran the command "pip install oresapi" in my OS terminal which enabled import of this package. After import, I used the session header provided in the assignment instructions to call the score function on all politician page ids post-filtering.<jupyter_code># Get ORES data for each article.
import oresapi
ores_session = oresapi.Session("https://ores.wikimedia.org", "Class project <<EMAIL>>")
page_scores = ores_session.score("enwiki", ["articlequality"], page_filtered['rev_id'])<jupyter_output><empty_output><jupyter_text>Part 2: ProcessingThe main goal of the processing step is to merge the three data sources into a single coherent dataframe that can be exported and used for further analysis. This dataframe should have page name, country, country population, and article quality information in each row.<jupyter_code># Create dataframe with empty article qualities and rev_ids so we can merge with page dataframe
# Note that I do it in this maybe strange way because I wasn't able to place the popped quality values right into
# the page_filtered dataframe because the indices had been modified post-filtering.
n_articles = page_filtered.shape[0]
score_columns = ['rev_id','article_quality']
scores_temp = pd.DataFrame(index=range(0,n_articles),columns=score_columns)
scores_temp.head()
# Copy over rev_ids.
scores_temp['rev_id'] = page_filtered['rev_id']
# Pop page quality values into rows whenever there is a rev_id at each index.
import math
row = 0
for score in page_scores:
    if row == n_articles:
        break
    while math.isnan(scores_temp.loc[row,'rev_id']):
        row = row + 1
    if row == n_articles:
        break
    else:
        scores_temp.loc[row, 'article_quality'] = score.pop('articlequality').pop('score',{'prediction':'n/a'}).pop('prediction','n/a')
        row = row + 1
# Sanity check.
scores_temp[0:20]<jupyter_output><empty_output><jupyter_text>Here's where I start doing actual data frame merges. After populating the page quality values into a dataframe and getting it merge-ready, I merge it with the filtered page list using page ids as the lookup.<jupyter_code># Merge article quality with page counts.
page_filtered = pd.DataFrame.merge(page_filtered,scores_temp,how='left',on='rev_id')
# Rename the Geography column so that we can do a merge on the country column name.
pop_filtered.rename(columns={'Geography':'country'}, inplace=True)
pop_filtered.head()<jupyter_output>/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py:3781: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  return super(DataFrame, self).rename(**kwargs)<jupyter_text>Here's where I do my second merge to add in population data. Because there are some listed countries without population data and some countries with population data with no articles, I perform an outer merge here as the assignment instructions specify exporting any rows with no matches later.<jupyter_code># Keep all the rows because we want the NaNs for later export and then do a sanity check.
merged_pages = pd.DataFrame.merge(page_filtered,pop_filtered,how='outer',on='country')
merged_pages.head()
merged_pages[20000:20010]
# Rename columns before export.
merged_pages.columns=['article_name','country','revision_id','article_quality','population']
merged_pages.head()
# Export any rows that have a lack of a match anywhere to a csv file.
no_match = merged_pages[merged_pages.isnull().any(axis=1)]
no_match.to_csv('wp_wpds_countries-no_match.csv',index=False)
# Export all complete merged data to a separate csv file.
matches = merged_pages[~merged_pages.isnull().any(axis=1)]
matches.to_csv('wp_wpds_politicians_by_country.csv',index=False)<jupyter_output><empty_output><jupyter_text>Part 3: AnalysisIn this step, I perform all calculations and dataframe manipulations necessary to produce six data tables. These will be as follows: - Top 10 countries by coverage: 10 highest-ranked countries in terms of number of politician articles as a proportion of country population- Bottom 10 countries by coverage: 10 lowest-ranked countries in terms of number of politician articles as a proportion of country population- Top 10 countries by relative quality: 10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality- Bottom 10 countries by relative quality: 10 lowest-ranked countries in terms of the relative proportion of - politician articles that are of GA and FA-quality- Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population- Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-quality<jupyter_code># Get list of unique countries in matches and initialize a dataframe that we'll use to storethe calculated values.
country_data = pd.DataFrame(matches['country'].unique())
country_data.insert(1,'population',0)
country_data.insert(2,'n_articles',0)
country_data.insert(3,'n_quality_articles',0)
country_data.insert(4,'coverage',0)
country_data.insert(5,'relative_quality',0)
# Rename columns and sanity check.
country_data.columns = ['country','population','n_articles','n_quality_articles','coverage','relative_quality']
country_data.head()
# Pump in population data and sanity check.
n_countries = country_data.shape[0]
for i in range(n_countries):
    pop_index = matches[matches.country == country_data.loc[i,'country']].index[0]
    country_data.loc[i,'population'] = matches.loc[pop_index,'population']
country_data.head()
# Count number of articles for each country by how many rows in the list of matches there are for each country.
for i in range(n_countries):
    country_data.loc[i,'n_articles'] = len(matches[matches.country == country_data.loc[i,'country']])
country_data.head()
# Manually set China and India's populations so they don't have commas in them. 
country_data.loc[6,'population'] = 1371.3
country_data.loc[22,'population'] = 1393.8

# Compute coverage of each country as number of politician articles divided by country population.
for i in range(n_countries):
    country_data.loc[i,'coverage'] = country_data.loc[i,'n_articles']/float(country_data.loc[i,'population'])
country_data.head()<jupyter_output><empty_output><jupyter_text>After counting the number of articles in each country and computing coverage by dividing number of articles by the country population, I sort the dataframe by coverage to see which countries are the top 10 and which are the bottom 10 by coverage. These are the first two deliverable data tables.<jupyter_code># Order data frame by coverage in ascending and descending order to get top and bottom countries by coverage.
top_by_coverage = country_data.sort_values(by='coverage', ascending=False)
bottom_by_coverage = country_data.sort_values(by='coverage')

# Display top 10 and bottom 10 countries by coverage.
top_by_coverage['country'][0:10]
bottom_by_coverage['country'][0:10]<jupyter_output><empty_output><jupyter_text>A quality article is defined as one that the ORES API defined as most like a Featured Article (FA) or a Good Article (GA). Total number of quality articles for each country is the sum of articles that match each label.<jupyter_code># Count number of quality articles for each country.
for i in range(n_countries):
    country_matches = matches[matches.country == country_data.loc[i,'country']]
    country_data.loc[i,'n_quality_articles'] = len(country_matches[country_matches.article_quality == 'FA']) + len(country_matches[country_matches.article_quality == 'GA'])
country_data.head()<jupyter_output><empty_output><jupyter_text>Relative quality is computed as the number of quality articles for a country divided by the total number of articles for a country.<jupyter_code># Compute relative quality.
for i in range(n_countries):
    country_data.loc[i,'relative_quality'] = country_data.loc[i,'n_quality_articles']/float(country_data.loc[i,'n_articles'])
country_data.head()<jupyter_output><empty_output><jupyter_text>After computing relative quality for each country, I sort the dataframe by relative quality to see which countries are the top 10 and which are the bottom 10. These are the third and fourth deliverable data tables.<jupyter_code># Get top 10 and bottom 10 countries by quality.
top_by_quality = country_data.sort_values(by='relative_quality', ascending=False)
bottom_by_quality = country_data.sort_values(by='relative_quality')

top_by_quality['country'][0:10]
bottom_by_quality['country'][0:10]<jupyter_output><empty_output><jupyter_text>In the next few cells, I start putting together a data frame that computes the same coverage and relative quality values, but rolled up by geographic region. Region names and country member information is available in the original WPDS_2018_data.csv file for population data. Region names are listed in all caps and every successive country in the table belongs to that region until encountering a new region name. I take advantage of this behavior in the following code.<jupyter_code># Gather region names from the geography rows that have capital letters and add new columns that will be 
# used to compute final data tables.
regions = pop_data[pop_data['Geography'].str.isupper()]
regions.insert(2, "n_articles",0.0)
regions.insert(3, "coverage",0.0)
regions.insert(4, "relative_quality",0.0)
regions<jupyter_output><empty_output><jupyter_text>This section was a pain to get working due to getting my head around indices. It may not be very efficient, but it hopefully works correctly. In the below code, I step through every country in the full population data file. If the country does in fact exist in the processed dataframe (if it has any articles), I add the number of articles and quality articles to that region's total count. If I've instead hit upon a region, I move to the next region and add subsequent article counts to that region.<jupyter_code># Compute coverage and relative quality by region.
import copy
i = 0
while i < pop_data.shape[0]:
    country = pop_data.loc[i,'Geography']
    if country.isupper():
        region = copy.copy(country)
        i += 1
    else:
        country_index = country_data[country_data.country == country].index
        if country_index.shape[0] > 0:
            country_index = country_index[0]
            regions.loc[regions['Geography'] == region,'n_articles'] += country_data.loc[country_index,'n_articles']
            regions.loc[regions['Geography'] == region,'relative_quality'] += country_data.loc[country_index,'n_quality_articles']
        i += 1
# Clean up the region dataframe a bit. Reset indices to avoid some further headache, shorten the name of
# the population column, and remove commas from all numbers.
regions.reset_index(drop=True, inplace=True)
regions.rename(columns={'Population mid-2018 (millions)':'population'}, inplace=True)
regions.loc[0,'population'] = 1284
regions.loc[3,'population'] = 4536
regions<jupyter_output><empty_output><jupyter_text>In the below code, I compute coverage and relative quality for each region in the same way as was done for each country: coverage is number of articles divided by population and relative quality is number of quality articles divided by total number of articles.<jupyter_code># Compute region coverage and relative_quality from article counts and population. 
for i in range(regions.shape[0]):
    regions.loc[i,'coverage'] = regions.loc[i,'n_articles']/float(regions.loc[i,'population'])
    regions.loc[i,'relative_quality'] /= regions.loc[i,'n_articles']
regions<jupyter_output><empty_output><jupyter_text>After computing coverage and relative quality by geographic region. I sort the entire dataframe by each  These are the final deliverable data tables.<jupyter_code># Display regions sorted by coverage. 
regions.sort_values(by='coverage', ascending=False)
# Display regions sorted by relative quality.
regions.sort_values(by='relative_quality', ascending=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08292857142857143, 'fraction_numerical': 0.0105, 'mean_word_length': 5.903846153846154, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '640587', 'license': 'MIT', 'n_tokens_mistral': 4012, 'n_tokens_neox': 3793, 'n_words': 1507, 'path': 'data-512-a2/hcds-a2-bias.ipynb', 'repo_name': 'zacharyfgarcia/DATA512', 'size': '14014'}