<jupyter_start><jupyter_text>Bias in Data <NAME> An exploration of bias in data on Wikipedia, originally completed for University of Washington's DATA 512 class in Autumn 2018 The purpose of this notebook is to explore bias in data by looking at English Wikipedia pages on politicians from a variety of countries. I combine the article data with data on the countries' populations and a prediction of the quality of the article garnered from ORES (more information on this below), and explore whether these have an impact on the number of articles on politicians overall and the number of higher quality articles.This notebook is used to process the downloaded data, get article quality prediction data from ORES, munge the data from the different sources, and complete an analysis of the combined data. A discussion of the results can be found in the ReadMe of this repository.<jupyter_code>#Import necessary packages and initialize desired notebook settings
import json
import numpy as np
import pandas as pd
import requests

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_inateractivity = "all"

from IPython.display import display, HTML<jupyter_output><empty_output><jupyter_text>Getting Data Two sources of data were used for this analysis; both are described in separate sections below. For both the data is publically available and easily downloaded.  Article DataThe first data set contains data about English Wikipedia articles on politicians, by country. It can be found at this link: https://figshare.com/articles/Untitled_Item/5513449This data is released under the CC-BY-SA 4.0 license, and so can be included here in this repo. From this page you can download a zip file containing the data, as well as R code used to generate the data. For the purpose of this analysis, only the page_data.csv is needed, so I have extracted that from the download and included it in the 'raw_data' folder of this repository.<jupyter_code>#Read page data csv from local data and look at first couple of rows
pages = pd.read_csv('data_raw/page_data.csv')
pages.head()<jupyter_output><empty_output><jupyter_text>Consistent with what is described on the figshare page for this data, page_data.csv contains the following columns: - page: the name of the English Wikipedia page (not cleaned) - country: the name of the country the politician was from - rev_id: the id of the last revision made to the page  Population DataThe second data set used for this analysis contains world population by country, in millions of people, as of 2018. The data set can be found here: https://www.dropbox.com/s/5u7sy1xt7g0oi2c/WPDS_2018_data.csv?dl=0The origin of this data is from the Population Reference Bureau (PRB); more information on the data can be found on their website, here: https://www.prb.org/2018-world-population-data-sheet-with-focus-on-changing-age-structures/Since this data is not licensed, and everything on the PRB website is copyrighted I have not included it in this repository. The code below loads the data locally. If you want to replicate this analysis, you can download the data from the link above and update the filepath used to read in the csv in the code below.Note that here I also do some minor data processing to make the analysis below easier, namely renaming the columns and converting the population to the full value rather than millions.<jupyter_code>#Read population data from local csv and review first couple of rows
pop = pd.read_csv('C:/Users/kentd/OneDrive/Documents/School-Grad/DATA_512/WPDS_2018_data.csv')

#Rename columns to make future analysis easier
pop.rename(columns={'Geography': 'country', 'Population mid-2018 (millions)': 'Population'}, inplace=True)

#Convert population to int, and then to raw value
pop['Population'] = pd.to_numeric(pop['Population'], errors='coerce')
pop['Population'] = pop['Population'] * 1e6
pop.head()<jupyter_output><empty_output><jupyter_text>Article Quality Predictions Using ORES For this analysis, I get the prediction of article quality from ORES, an API for machine learning developed by Wikimedia. See the following link for more information:https://www.mediawiki.org/wiki/ORESORES takes a Wikipedia article ID and assigns a probability that the quality of the article falls into one of six categories. The highest probability is the category assigned to the article. The categories are (from best to worst quality): - FA - Featured article - GA - Good article - B - B-class article - C - C-class article - Start - Start-class article - Stub - Stub-class article For this analysis, I consider articles classified as 'FA' and 'GA' to be "high quality" articles.To get the predictions, I feed the pages dataset to the ORES system using the code below. Since the API documentation recommends batching 50 revision IDs per request, I have split the data into chunks of 50 rather than sending the entire dataframe. The code for the function 'ores' I used to make the API requests is based on the example provided here: https://github.com/Ironholds/data-512-a2/blob/master/hcds-a2-bias_demo.ipynb<jupyter_code>#Start with some housekeeping - define headers & API endpoint
headers = {'User-Agent' : 'https://github.com/kentdanas', 'From' : '<EMAIL>'}
endpoint = 'https://ores.wikimedia.org/v3/scores/{project}/?models={model}&revids={revids}'

#Now define function to make the API call for a given list of revision ids
def ores(rev_ids, headers=headers, endpoint=endpoint):
    params = {'project' : 'enwiki',
              'model'   : 'wp10',
              'revids'  : '|'.join(str(x) for x in rev_ids)
              }
    api_call = requests.get(endpoint.format(**params))
    response = api_call.json()
    return response
#Use the ORES function above to get predictions for each article by looping through 50 articles at a time. 
# Note, this chunk will take a few minutes to run; this is a good time to grab a cup of coffee!

#Define start indexes
index_start = 0
index_stop = 49

#Create empty dataframe to store results
page_quality = pd.DataFrame(columns=('rev_id', 'article_quality'))

while index_start < len(pages):
    #Pull chunk of revision ids
    rev_ids = list(pages['rev_id'][index_start:index_stop])

    #Feed revision ids to ORES and get predictions
    response = ores(rev_ids)

    #pull out prediction for each rev_id from resulting json dump. Note that if the article was not found by ORES 
    # it will not have a prediction, hence the try/except statement. For articles with no prediction I filled the
    # results with 'nan'
    for rev_id in rev_ids:
        rev_id = str(rev_id)
        
        try:
            prediction = response['enwiki']['scores'][rev_id]['wp10']['score']['prediction']

        except:
            prediction = np.nan

        #append results to dataframe
        page_quality = page_quality.append({'rev_id':rev_id, 'article_quality':prediction}, ignore_index=True) 

    #Redefine indexes
    index_start += 50
    index_stop = min(index_stop+50, len(pages))<jupyter_output><empty_output><jupyter_text>Combining DatasetsNow that I have the article quality prediction data from ORES, I combine the three data sets into one so they can be used for the analysis. The final dataframe has the following columns: - article_name: the (uncleaned) article name - country: the country of the politician in the article - revision_id: the id of the last revision of the article - article_quality: the ORES prediction of the article quality - population: the population of the country<jupyter_code>#First combine the ORES predictions and the page data (note that I need to convert the rev_id type to an int first)
page_quality['rev_id'] = pd.to_numeric(page_quality['rev_id'], errors='coerce')
final_dataset = pages.merge(page_quality, how='left', on=['rev_id'])

#Next, merge with population data. This is done as an inner join since I only want to keep articles for which 
#there is a population, and I don't need the population for any countries without any articles
final_dataset = final_dataset.merge(pop, how='inner', on=['country'])

#Rename a couple of columns
final_dataset.rename(columns={'page': 'article_name', 'rev_id': 'revision_id'}, inplace=True)

#Save to CSV
final_dataset.to_csv('data_clean/final_article_dataset.csv', index = False)

#Take a look at the resulting dataframe
final_dataset.head()<jupyter_output><empty_output><jupyter_text>AnalysisThe analysis on this data set consists of a calculation of the proportion of articles per population for each country, and proportion of high quality articles for each country. A detailed discussion of the results of this analysis can be found in the ReadMe of this repository.  Politician Articles as a Proportion of Country PopulationFirst, for each country I counted the total number of articles and then calculated the number of articles per capita as a percentage. The final tables show the top 10 and bottom 10 countries for this metric respectively.<jupyter_code>#Create a new dataframe with count of articles by country
articles_per_cap = final_dataset.groupby(['country', 'Population']).count().reset_index()
articles_per_cap.rename(columns={'article_name': 'number_of_articles'}, inplace=True)
articles_per_cap = articles_per_cap.drop(['revision_id', 'article_quality'], 1)

#Calculate number of articles as proportion of country population as a percentage
articles_per_cap['articles_per_capita_percentage'] = articles_per_cap['number_of_articles'] / articles_per_cap['Population'] * 100<jupyter_output><empty_output><jupyter_text>Now that I have the needed data, I can show the highest and lowest ranked countries:  Lowest<jupyter_code>#Pull first 10 rows of ascending-sorted dataframe
low_articles_per_cap = articles_per_cap.sort_values(by=['articles_per_capita_percentage'])[0:10]
display(HTML(low_articles_per_cap.to_html(index=False)))<jupyter_output><empty_output><jupyter_text>It looks like most of the countries with very low percentages of articles per capita do not have low populations, but may be countries where media access is restricted, or dictatorships that may not have as many politicians. A more detailed discussion of these results can be found in the ReadMe.  Highest<jupyter_code>#Pull first 10 rows of ascending-sorted dataframe
high_articles_per_cap = articles_per_cap.sort_values(by=['articles_per_capita_percentage'], ascending=False)[0:10]
display(HTML(high_articles_per_cap.to_html(index=False)))<jupyter_output><empty_output><jupyter_text>It looks like most of the countries with high percentages of articles per capita have very small populations. Again, a more detailed discussion of this analysis can be found in the ReadMe for this repository.  Proportion of High Quality Articles About PoliticiansIn the last portion of this analysis I look at the proportion of articles per country that are considered "high quality". For this analysis, "high quality" is defined as having an ORES prediction of either FA (featured article) or GA (good article).<jupyter_code>#Create new dataframe with number of high quality articles per country
quality_articles = final_dataset[(final_dataset['article_quality']=='FA')|(final_dataset['article_quality']=='GA')]
quality_articles = quality_articles.groupby(['country']).count().reset_index()
quality_articles.rename(columns={'article_name': 'high_quality_articles'}, inplace=True)

#Merge this new dataframe back to the dataframe created in the section above with total articles per country
#Note that this is a left join because some countries do not have any high quality articles and so do not appear
#in the dataframe I created directly before this. I fill those with zeros
percent_quality_articles = articles_per_cap.merge(quality_articles, on=['country'], how='left').fillna(0)

#Remove unneeded columns
percent_quality_articles = percent_quality_articles.drop(['Population_x', 
                                                          'articles_per_capita_percentage', 
                                                          'revision_id', 
                                                          'article_quality', 
                                                          'Population_y'], 1)
#Calculate percentage of high quality articles
percent_quality_articles['percent_high_quality_articles'] = percent_quality_articles['high_quality_articles']/percent_quality_articles['number_of_articles']*100

#Take a look at resulting dataframe
percent_quality_articles.head()<jupyter_output><empty_output><jupyter_text>Now that I have the needed data I can show the highest and lowest ranked countries:  Lowest<jupyter_code>#Pull first 10 rows of ascending-sorted dataframe
low_percent_quality = percent_quality_articles.sort_values(by=['percent_high_quality_articles'])[0:10]
display(HTML(low_percent_quality.to_html(index=False)))<jupyter_output><empty_output><jupyter_text>All of the bottom 10 countries by percentage of high quality articles had no articles in the "FA" or "GA" category, but these may not be the only countries this is true for. Since this is the case, using a metric of "bottom 10" doesn't really make sense. Therefore, below I also created a table for all countries that had no high quality articles.<jupyter_code>#Create and view new dataframe with all countries that had no high quality articles
no_high_quality = percent_quality_articles[percent_quality_articles['percent_high_quality_articles']==0]
display(HTML(no_high_quality.to_html(index=False)))<jupyter_output><empty_output><jupyter_text>It appears there are actually 37 countries that had no articles predicted to be high quality. More discussion of these results can be found in the ReadMe.  Highest<jupyter_code>#Pull first 10 rows of descending-sorted dataframe
high_percent_quality = percent_quality_articles.sort_values(by=['percent_high_quality_articles'], ascending=False)[0:10]
display(HTML(high_percent_quality.to_html(index=False)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06993407853253081, 'fraction_numerical': 0.007953568357695616, 'mean_word_length': 5.023737591713423, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 51, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '567552', 'license': 'MIT', 'n_tokens_mistral': 3797, 'n_tokens_neox': 3608, 'n_words': 1597, 'path': 'hcds-a2-bias.ipynb', 'repo_name': 'kentdanas/data-512-a2', 'size': '13968'}