<jupyter_start><jupyter_text>A2-Bias in Data The goal of this assignment is to explore the concept of bias through data on Wikipedia articles - specifically, articles on political figures from a variety of countries. We combine a dataset of Wikipedia articles with a dataset of country populations, and use a machine learning service called ORES to estimate the quality of each article.We then perform an analysis of how the coverage of politicians on Wikipedia and the quality of articles about politicians varies between countries. The analysis will consist of a series of tables that show:the countries with the greatest and least coverage of politicians on Wikipedia compared to their population.the countries with the highest and lowest proportion of high quality articles about politicians.a ranking of geographic regions by articles-per-person and proportion of high quality articles.  Getting the Article and Population Data<jupyter_code>import numpy as np
import pandas as pd
df_pagedata=pd.read_csv("page_data.csv")
df_pagedata.head()
df_populationdata=pd.read_csv("WPDS_2020_data - WPDS_2020_data.csv")
df_populationdata.head()
df_pagedata=df_pagedata[~df_pagedata['page'].str.contains("Template:")]
df_pagedata.head()
df_countrydata = df_populationdata[df_populationdata['Type']=='Country']
df_countrydata.head()
df_regiondata = df_populationdata[df_populationdata['Type']=='Sub-Region']
df_regiondata.head()<jupyter_output><empty_output><jupyter_text>Getting article quality predictions Now we need to get the predicted quality scores for each article in the Wikipedia dataset. We're using a machine learning system called ORES. This was originally an acronym for "Objective Revision Evaluation Service" but was simply renamed “ORES”.ORES is a machine learning tool that can provide estimates of Wikipedia article quality. The article quality estimates are, from best to worst:FA - Featured articleGA - Good articleB - B-class articleC - C-class articleStart - Start-class articleStub - Stub-class articleThese were learned based on articles in Wikipedia that were peer-reviewed using the Wikipedia content assessment procedures.These quality classes are a sub-set of quality assessment categories developed by Wikipedia editors.We use a REST API to obtain the information for each article.<jupyter_code>import requests
import json
headers = {'User-Agent': 'https://github.com/Minerva-Lan', 'From' :'<EMAIL>'}
def api_call(rev_ids, headers):
    endpoint="https://ores.wikimedia.org/v3/scores/{context}/?revids={revids}&models={model}"
    params={"context": "enwiki",
            "revids"  : '|'.join(str(x) for x in rev_ids),
            "model"  : "articlequality"
            }
    call = requests.get(endpoint.format(**params), headers=headers)
    response = call.json()
    return response
def extract_class_labels(response): 
    for rev_id, val in response.items():
        try:
            rev_scores[rev_id] = val["articlequality"]["score"]["prediction"]
        except:
            pass
page_data_rev_ids=df_pagedata['rev_id'].tolist()
step = 50
rev_scores={}
for i in range(0, len(page_data_rev_ids), step):
    response = api_call(page_data_rev_ids[i: i+step], headers)
    extract_class_labels(response['enwiki']['scores'])
df_article_quality = pd.DataFrame(list(rev_scores.items()), columns=['rev_id', 'quality'])
df_article_quality.head()
df_article_quality['rev_id'] = df_article_quality['rev_id'].astype('int')
page_article_quality = pd.merge(df_pagedata, df_article_quality, on='rev_id', how='left')
page_article_quality.head()
page_no_match = page_article_quality[page_article_quality['quality'].isnull()]
page_no_match.head()
page_no_match.to_csv('article_without_quality.csv', index=False)
page_with_article_quality=page_article_quality[~page_article_quality['quality'].isnull()]
page_with_article_quality.head()
page_with_article_quality.to_csv('page_with_article_quality.csv', index=False)<jupyter_output><empty_output><jupyter_text>Combining the Datasets<jupyter_code>wp_wpds_politicians_by_country=pd.merge(left=page_with_article_quality, right =df_countrydata, 
                                  left_on='country', right_on='Name')
wp_wpds_politicians_by_country.head()
wp_wpds_politicians_by_country=wp_wpds_politicians_by_country[['page', 'country', 'rev_id', 'quality', 'Population']]
wp_wpds_politicians_by_country.rename(columns={'page' : 'article_name',
                                          'rev_id' : 'revision_id',
                                          'quality' : 'article_quality_est',
                                          'Population' : 'population'}, inplace = True)
wp_wpds_politicians_by_country.head()
wp_wpds_politicians_by_country.to_csv('wp_wpds_politicians_by_country.csv', index=False)
wp_wpds_politicians_by_country_all=pd.merge(left=page_with_article_quality, right =df_countrydata, 
                                  left_on='country', right_on='Name', how='outer')
no_country_match = wp_wpds_politicians_by_country_all[wp_wpds_politicians_by_country_all['Population'].isnull()]
no_country_match.to_csv('wp_wpds_countries-no_match.csv', index = False)<jupyter_output><empty_output><jupyter_text>Analysis<jupyter_code>pivot_quality = pd.pivot_table(wp_wpds_politicians_by_country, index=['country'], columns=['article_quality_est'],
              aggfunc={'article_quality_est' : len}, fill_value =0)
pivot_quality.columns=pivot_quality.columns.droplevel()
pivot_quality.reset_index(inplace=True)
pivot_quality.columns.name=None
pivot_quality.head()
df_country_analysis=pivot_quality.merge(wp_wpds_politicians_by_country.groupby(['country'])['population'].mean(),
                    left_on='country', right_index=True)
df_country_analysis['total_articles']=df_country_analysis['B']+df_country_analysis['C']+df_country_analysis['FA']+df_country_analysis['GA']+df_country_analysis['Start']+df_country_analysis['Stub']
df_country_analysis['high_quality_article'] = df_country_analysis['FA'] + df_country_analysis['GA']
df_country_analysis['relative_quality']=(df_country_analysis['high_quality_article']/df_country_analysis['total_articles'])*100
df_country_analysis['coverage']=(df_country_analysis['total_articles']/df_country_analysis['population'])*100
df_country_analysis.drop(columns=['B', 'C', 'FA', 'GA', 'Start', 'Stub'], inplace=True)
df_country_analysis.head()
region = "NORTHERN AFRICA"

regions = ['WORLD', 'AFRICA', 'NORTHERN AFRICA']
for i in range(3, len(df_populationdata)):
    if df_populationdata.iloc[i]['Type']=='Sub-Region':
        region = df_populationdata.iloc[i]['Name']
    regions.append(region)
df_populationdata['Region'] = regions
df_country_region_analysis = pd.merge(left=df_country_analysis, right =df_populationdata[['Name', 'Region']], 
                                  left_on='country', right_on='Name', how='left')
df_country_region_analysis.drop(columns='Name', inplace = True)
df_country_region_analysis.head()
df_region_analysis=df_country_region_analysis.groupby('Region').agg({'total_articles':'sum', 'high_quality_article': 'sum'})
df_region_analysis= df_region_analysis.merge(df_regiondata[['Name', 'Population']],left_index=True, right_on='Name')
df_region_analysis.head()
df_region_analysis['coverage'] = df_region_analysis['total_articles']/df_region_analysis['Population']*100
df_region_analysis['relative_quality'] = df_region_analysis['high_quality_article']/df_region_analysis['total_articles']*100
df_region_analysis=df_region_analysis[['Name', 'coverage', 'relative_quality']]
df_region_analysis.rename(columns={'Name':'Region'}, inplace=True)
df_region_analysis.head()<jupyter_output><empty_output><jupyter_text>Results  Top 10 countries by coverage: 10 highest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>df_country_analysis[['country', 'coverage']].sort_values(by=['coverage'], ascending=False)[0:10].reset_index().drop('index', axis=1)<jupyter_output><empty_output><jupyter_text>Bottom 10 countries by coverage: 10 lowest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>df_country_analysis[['country','coverage']].sort_values(by=['coverage'])[0:10].reset_index().drop('index', axis=1)<jupyter_output><empty_output><jupyter_text>Top 10 countries by relative quality: 10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>df_country_analysis[['country','relative_quality']].sort_values(by=['relative_quality'], ascending=False)[0:10].reset_index().drop('index', axis=1)<jupyter_output><empty_output><jupyter_text>Bottom 10 countries by relative quality: 10 lowest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>df_country_analysis[['country','relative_quality']].sort_values(by=['relative_quality'])[0:10].reset_index().drop('index', axis=1)<jupyter_output><empty_output><jupyter_text>Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population<jupyter_code>df_region_analysis[['Region','coverage']].sort_values(by=['coverage'], ascending =False)<jupyter_output><empty_output><jupyter_text>Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-quality<jupyter_code>df_region_analysis[['Region','relative_quality']].sort_values(by=['relative_quality'], ascending =False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11306947021232736, 'fraction_numerical': 0.006081220366934653, 'mean_word_length': 7.092577147623019, 'pattern_counts': {'":': 1, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '418580', 'license': 'MIT', 'n_tokens_mistral': 2946, 'n_tokens_neox': 2850, 'n_words': 624, 'path': 'A2-Bias in Data.ipynb', 'repo_name': 'Minerva-Lan/Data-512-A2', 'size': '9707'}