<jupyter_start><jupyter_text>Assignment 2 : Bias in Data AssignmentThe goal of this assignment is to explore the concept of bias through data on Wikipedia articles - specifically, articles on political figures from a variety of countries. For this assignment, you will combine a dataset of Wikipedia articles with a dataset of country populations, and use a machine learning service called ORES to estimate the quality of each article.<jupyter_code>import pandas as pd
import requests
import numpy as np
import json<jupyter_output><empty_output><jupyter_text>Data cleaning<jupyter_code># load dataframes
page_data_df = pd.read_csv("data/page_data.csv")
wpds_data_df = pd.read_csv("data/wpds_data.csv")<jupyter_output><empty_output><jupyter_text>We first need to get rid of rows started with `template:` in the `page_data` dataset as these pages are not Wikipedia articles.<jupyter_code>page_data_df = page_data_df[~(page_data_df["page"].str.startswith("Template:"))]
page_data_df<jupyter_output><empty_output><jupyter_text>Similarly, we need to exclude rows in all caps for column `Name` in the `wpds_data` dataset as these rows provide cumulative regional population counts rather than country-level counts. Side note: we still want to retain the cumulative rows in a separate dataFrame.<jupyter_code>wpds_data_cumulative_df = wpds_data_df[(wpds_data_df['Name'].str.isupper())] # we still want to retain them
wpds_data_df = wpds_data_df[~(wpds_data_df['Name'].str.isupper())]
wpds_data_df<jupyter_output><empty_output><jupyter_text>Requesting ORES to gather article quality predictionsWe are using [ORES](https://ores.wikimedia.org/v3//) to get the predicted quality score for each article in the `page_data` dataset. **ATTENTION**: it took a while to fetch the 46k+ entries through the ORES scoring API. Data has been pre-processed and saved in `data/ores_data.csv` in order to avoid re-scoring the article.<jupyter_code># first define our headers and create function call to call the ORES apis
req_headers = {
    "User-Agent": "https://github.com/jell0wed",
    "From": "jepoisso<EMAIL>"
}

def api_call(endpoint, params):
    call = requests.get(endpoint.format(**params), headers=req_headers)
    resp = call.json()
    return resp
cache = {}
ores_articlequality_endpoint = "https://ores.wikimedia.org/v3/scores/{context}/{revid}/{model}"
ores_articlequality_params = {
    "context": "enwiki",
    "revid": None,
    "model": "articlequality"
}

def fetch_ores_score(rev_id):
    ores_articlequality_params["revid"] = rev_id
    if rev_id in cache:
        return json.dumps(cache[rev_id])
    resp = api_call(ores_articlequality_endpoint, ores_articlequality_params)
    article_quality_dict = resp['enwiki']['scores'][str(rev_id)]['articlequality']
    cache[rev_id] = {}
    if 'score' in article_quality_dict:
        cache[rev_id] = article_quality_dict['score']['probability']
        return json.dumps(article_quality_dict['score']['probability'])
    print("processed %d" % (len(cache)))

# do not uncomment the following lines, this took 50m to run, pre-processed data can be found in in `data/ores_data.csv`
#page_data_df["ores_score"] = page_data_df["rev_id"]
#page_data_df["ores_score"] = page_data_df["ores_score"].apply(fetch_ores_score)
#page_data_df.to_csv("data/ores_data.csv")
page_data_df = pd.read_csv("data/ores_data.csv", index_col=0)
# filter out rows for which the prediction was not available and log them as a separate file
page_data_df_excl_df = page_data_df[page_data_df["ores_score"].isnull()]
page_data_df_excl_df.to_csv("data/ores_data_excluded.csv")

page_data_df = page_data_df[~(page_data_df["ores_score"].isnull())]
page_data_df.to_csv("data/ores_data_filtered.csv")<jupyter_output><empty_output><jupyter_text>ORES predictions for some rows was unavailable. Those rows have been filtered out into the `data/ores_data_excluded.csv` file.  Merging datasetsNow that we have gathered all the ORES data and flagged entries for which no prediction is available, we can merge the wikipedia data as well as the population data together.We first need to merge the Wikipedia pages w/ ORES prediction data with the population data. Wikipedia page entries for which no suitable country mapping was found have been filtered out and can be found in `data/wp_wpds_countries-no_match.csv`.The rest of the merged entries of wikipedia pages w/ country population has been re-shaped to the following schema:  - country - article_name - revision_id - article_quality_est. - populationand can be found in `data/wp_wpds_politicians_by_country.csv`.<jupyter_code>page_data_df
wpds_data_df['country'] = wpds_data_df['Name']
# merge both the wikipedia page dataset w/ countries
merged_df = pd.merge(page_data_df, wpds_data_df, how="left", on="country")

# remove entries for which there was no country match
merged_no_match_df = merged_df[merged_df['Name'].isnull()]
merged_no_match_df.to_csv("data/wp_wpds_countries-no_match.csv")

# completed merge dataset, re-shape the data according to schema
merged_df = merged_df[~(merged_df['Name'].isnull())]
# next we need to extract the best prediction of article quality from the `ores_score` column.
# the article quality prediction is the prediction which has the highest probability
def extract_article_quality(ores_json_score):
    ores_preds = json.loads(ores_json_score)
    quality = max(ores_preds, key=ores_preds.get)
    return quality

merged_df["article_quality_est"] = merged_df["ores_score"].apply(extract_article_quality)
# reshape the data to the correct schema
merged_df = merged_df.rename(columns={
    "page": "article_name",
    "rev_id": "revision_id",
    "Population": "population"
})
merged_df = merged_df.drop(columns=['ores_score', 'FIPS', 'Name', 'Type', 'TimeFrame', 'Data (M)'])
merged_df = merged_df[['country', 'article_name', 'revision_id', 'article_quality_est', 'population']]
merged_df.to_csv("data_clean/wp_wpds_politicians_by_country.csv")<jupyter_output><empty_output><jupyter_text>AnalysisWe will first do the necessary aggregations to prepare for the analysis. Mainly, we are interested in the following insights: 1. Top 10 countries by coverage: 10 highest-ranked countries in terms of number of politician articles as a proportion of country population2. Bottom 10 countries by coverage: 10 lowest-ranked countries in terms of number of politician articles as a proportion of country population3. Top 10 countries by relative quality: 10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality4. Bottom 10 countries by relative quality: 10 lowest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality5. Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population6. Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-quality<jupyter_code># coverage; we are interested in the proportion of the number of politician wikipedia articles as a proportion of the 
# country population
coverage_pop_df = merged_df.groupby("country").agg(
    article_count=pd.NamedAgg(column="article_name", aggfunc="count"),
    population=pd.NamedAgg(column="population", aggfunc="first")
)
coverage_pop_df = coverage_pop_df.reset_index()
coverage_pop_df['proportion'] = coverage_pop_df['article_count'] / coverage_pop_df['population']
coverage_pop_df
# high quality article; we are interested in the proportion of the number of high quality (FA, GA) politician 
# wikipedia articles as a proportion of the country population
high_quality_article_df = merged_df[merged_df['article_quality_est'].isin(["FA", "GA"])]
hq_pop_df = high_quality_article_df.groupby("country").agg(
    hq_count=pd.NamedAgg(column="article_name", aggfunc="count"),
    population=pd.NamedAgg(column="population", aggfunc="first")
)
hq_pop_df = hq_pop_df.reset_index()
hq_pop_df['proportion'] = hq_pop_df['hq_count'] / hq_pop_df['population']
hq_pop_df
# trick from https://stackoverflow.com/questions/68714674/select-all-below-rows-till-we-get-next-match-in-column-pandas-issue-resolved-bu
df = pd.read_csv("data/wpds_data.csv")
df = df[df['Type'].isin(["Sub-Region", "Country"])]
sub_regions = len(df.loc[df.Type == 'Sub-Region'])

# assign new ID to rows below until we find the next sub-region
df.loc[df.Type == 'Sub-Region', 'new_id'] = [n for n in range(1, sub_regions+1)]
df.fillna(method='ffill', inplace=True)

sub_region_mapping_df = df.groupby("new_id").agg(
    sub_region=pd.NamedAgg(column="Name", aggfunc="first"),
    sub_region_population=pd.NamedAgg("Population", aggfunc="first")
)
sub_region_mapping = sub_region_mapping_df.to_dict()

sub_region_mapping

# map the subregion to each country & sub-region population
df['sub_region'] = df['new_id'].apply(lambda x: sub_region_mapping['sub_region'][x])
df['sub_region_population'] = df['new_id'].apply(lambda x: sub_region_mapping['sub_region_population'][x])

# now merge with the wikipedia articles
merged_df_subregion = pd.merge(merged_df, df, left_on="country", right_on="Name")
merged_df_subregion = merged_df_subregion.drop(columns=["FIPS", "Name", "Type", "TimeFrame", "Data (M)", "Population", "new_id"])
# coverage
coverage_pop_subregion_df = merged_df_subregion.groupby("sub_region").agg(
    article_count=pd.NamedAgg(column="article_name", aggfunc="count"),
    sub_region_population=pd.NamedAgg(column="sub_region_population", aggfunc="first")
)
coverage_pop_subregion_df['proportion'] = coverage_pop_subregion_df['article_count'] / coverage_pop_subregion_df['sub_region_population']

# hq articles
hqarticle_pop_subregion_df = merged_df_subregion[merged_df_subregion['article_quality_est'].isin(["FA", "GA"])].groupby("sub_region").agg(
    article_count=pd.NamedAgg(column="article_name", aggfunc="count"),
    sub_region_population=pd.NamedAgg(column="sub_region_population", aggfunc="first")
)
hqarticle_pop_subregion_df['proportion'] = coverage_pop_subregion_df['article_count'] / coverage_pop_subregion_df['sub_region_population']<jupyter_output><empty_output><jupyter_text>Table 1 : Top 10 countries by coverage: 10 highest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>coverage_pop_df.sort_values(by=['proportion'], ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Table 2 : Bottom 10 countries by coverage: 10 highest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>coverage_pop_df.sort_values(by=['proportion'], ascending=True).head(10)<jupyter_output><empty_output><jupyter_text>Table 3 : Top 10 countries by relative quality: 10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>hq_pop_df.sort_values(by=['proportion'], ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Table 4 : Bottom 10 countries by relative quality: 10 lowest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>hq_pop_df.sort_values(by=['proportion'], ascending=True).head(10)<jupyter_output><empty_output><jupyter_text>Table 5 : Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population<jupyter_code>coverage_pop_subregion_df.sort_values(by=['proportion'], ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Table 6 : Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-quality<jupyter_code>hqarticle_pop_subregion_df.sort_values(by=['proportion'], ascending=False).head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10123456790123457, 'fraction_numerical': 0.006172839506172839, 'mean_word_length': 6.926288323548597, 'pattern_counts': {'":': 8, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '122395', 'license': 'MIT', 'n_tokens_mistral': 3838, 'n_tokens_neox': 3660, 'n_words': 1055, 'path': 'data512-a2-bias.ipynb', 'repo_name': 'jell0wed/data-512-a2', 'size': '12150'}