<jupyter_start><jupyter_text>Course Human-Centered Data Science ([HCDS](https://www.mi.fu-berlin.de/en/inf/groups/hcc/teaching/winter_term_2020_21/course_human_centered_data_science.html)) - Winter Term 2020/21 - [HCC](https://www.mi.fu-berlin.de/en/inf/groups/hcc/index.html) | [Freie Universität Berlin](https://www.fu-berlin.de/)*** A2 - Wikipedia, ORES, and Bias in DataPlease follow the reproducability workflow as practiced during the last exercise.  Step 1⃣ | Data acquisitionYou will use two data sources: (1) Wikipedia articles of politicians and (2) world population data.**Wikipedia articles -**The Wikipedia articles can be found on [Figshare](https://figshare.com/articles/Untitled_Item/5513449). It contains politiciaans by country from the English-language wikipedia. Please read through the documentation for this repository, then download and unzip it to extract the data file, which is called `page_data.csv`.**Population data -**The population data is available in `CSV` format in the `_data` folder. The file is named `export_2019.csv`. This dataset is drawn from the [world population datasheet](https://www.prb.org/international/indicator/population/table/) published by the Population Reference Bureau (downloaded 2020-11-13 10:14 AM). I have edited the dataset to make it easier to use in this assignment. The population per country is given in millions!<jupyter_code>import pandas as pd
import numpy as np
import os
import os.path
from os import makedirs, path
import altair as alt<jupyter_output><empty_output><jupyter_text>Step 2⃣ | Data processing and cleaningThe data in `page_data.csv` contain some rows that you will need to filter out. It contains some page names that start with the string `"Template:"`. These pages are not Wikipedia articles, and should not be included in your analysis. The data in `export_2019.csv` does not need any cleaning.***| | `page_data.csv` | | ||-|------|---------|--------|| | **page** | **country** | **rev_id** ||0|	Template:ZambiaProvincialMinisters | Zambia | 235107991 ||1|	<NAME> | Chad | 355319463 |***| | `export_2019.csv` | | ||-|------|---------|--------|| | **country** | **population** | **region** ||0|	Algeria | 44.357 | AFRICA ||1|	Egypt | 100.803 | 355319463 |***  Loading the data<jupyter_code>pages_df = df = pd.read_csv(
  'data_raw\pages\csv\page_data.csv',
  sep = ','
)
population_df = df = pd.read_csv(
  'data_raw\population\csv\export_2019.csv',
  sep = ';'
)<jupyter_output><empty_output><jupyter_text>Exploring the data<jupyter_code>pages_df.head() 
# Checking for Nan or null values
pages_df.info()
print(len(pd.unique(pages_df['country'])),' countries')
population_df.iloc[0]['population']
# Checking for Nan or null values
population_df.info()
population_df['population'].dtype
pd.unique(population_df['region'])<jupyter_output><empty_output><jupyter_text>Getting article quality predictions with ORESNow you need to get the predicted quality scores for each article in the Wikipedia dataset. We're using a machine learning system called [**ORES**](https://www.mediawiki.org/wiki/ORES) ("Objective Revision Evaluation Service"). ORES estimates the quality of an article (at a particular point in time), and assigns a series of probabilities that the article is in one of the six quality categories. The options are, from best to worst:| ID | Quality Category |  Explanation ||----|------------------|----------|| 1 | FA    | Featured article || 2 | GA    | Good article || 3 | B     | B-class article || 4 | C     | C-class article || 5 | Start | Start-class article || 6 | Stub  | Stub-class article |For context, these quality classes are a sub-set of quality assessment categories developed by Wikipedia editors. If you're curious, you can [read more](https://en.wikipedia.org/wiki/Wikipedia:Content_assessmentGrades) about what these assessment classes mean on English Wikipedia. For this assignment, you only need to know that these categories exist, and that ORES will assign one of these six categories to any `rev_id`. You need to extract all `rev_id`s in the `page_data.csv` file and use the ORES API to get the predicted quality score for that specific article revision.  ORES REST API endpointThe [ORES REST API](https://ores.wikimedia.org/v3/!/scoring/get_v3_scores_context_revid_model) is configured fairly similarly to the pageviews API we used for the last assignment. It expects the following parameters:* **project** --> `enwiki`* **revid** --> e.g. `235107991` or multiple ids e.g.: `235107991|355319463` (batch)* **model** --> `wp10` - The name of a model to use when scoring.**❗Note on batch processing:** Please read the documentation about [API usage](https://www.mediawiki.org/wiki/ORESAPI_usage) if you want to query a large number of revisions (batches). You will notice that ORES returns a prediction value that contains the name of one category (e.g. `Start`), as well as probability values for each of the six quality categories. For this assignment, you only need to capture and use the value for prediction.**❗Note:** It's possible that you will be unable to get a score for a particular article. If that happens, make sure to maintain a log of articles for which you were not able to retrieve an ORES score. This log should be saved as a separate file named `ORES_no_scores.csv` and should include the `page`, `country`, and `rev_id` (just as in `page_data.csv`).You can use the following **samle code for API calls**: Sending one request for each `rev_id` might take some time. If you want to send batches you can use `'|'.join(str(x) for x in revision_ids` to put your ids together. Please make sure to deal with [exception handling](https://www.w3schools.com/python/python_try_except.asp) of the `KeyError` exception, when extracting the `prediction` from the `JSON` response.<jupyter_code>import requests
import json

# Customize these with your own information
headers = {
    'User-Agent': 'https://github.com/Alioio',
    'From': '<EMAIL>'
}

def get_ores_data(rev_id, headers):
    
    # Define the endpoint
    # https://ores.wikimedia.org/scores/enwiki/?models=wp10&revids=807420979|807422778
    endpoint = 'https://ores.wikimedia.org/v3/scores/{project}/?models={model}&revids={revids}'

    params = {'project' : 'enwiki',
              'model'   : 'wp10',
              'revids'  : rev_id
              }

    api_call = requests.get(endpoint.format(**params))
    response = api_call.json()
    data = json.dumps(response)

    return data
#rev_ids = pages_df['rev_id'][:200]
rev_ids = pages_df['rev_id']
def make_df(keys, subset):
    quality_df = {'rev_id': [],
        'quality': []
        }

    quality_df = pd.DataFrame(quality_df, columns = ['rev_id', 'quality'])

    for key in keys:
        try:
            new_row = pd.Series(data={'rev_id':key, 'quality':subset[key]['wp10']['score']['prediction']})
            quality_df = quality_df.append(new_row, ignore_index=True)
            #print(subset[key]['wp10']['score']['prediction'])
        except KeyError:
            try:
                new_row = pd.Series(data={'rev_id':key, 'quality':subset[key]['wp10']['error']['message']})
                quality_df = quality_df.append(new_row, ignore_index=True)
            except KeyError:
                new_row = pd.Series(data={'rev_id':key, 'quality':'unknown error'})
                quality_df = quality_df.append(new_row, ignore_index=True)
    return quality_df
def run_batch(id1, id2, batchids):
    bucket = batchids[id1: id2]
    bucket = ('|'.join(str(x) for x in bucket))
    output = get_ores_data(bucket, headers)
    output = json.loads(output)
    subset = output['enwiki']['scores']
    keys   = subset.keys()
    df = make_df(keys, subset)  
    return df
if not os.path.isfile('data_clean/csv/ORES_scores.csv'):

    final_df = pd.DataFrame({'rev_id': [], 'quality': [] }, columns = ['rev_id', 'quality'])

    last = 0
    for bucket in range(0, len(rev_ids)-50, 50):
        print('new batch: ',bucket,': ',bucket+50)
        df = run_batch(bucket, bucket+50, rev_ids)
        final_df = pd.concat([final_df, df], axis= 0)
        last = bucket

    if( (len(rev_ids) - len(final_df)) > 0):
        print('new batch: ',len(final_df),': ',len(rev_ids))
        df = run_batch(len(final_df), len(rev_ids), rev_ids)
        final_df = pd.concat([final_df, df], axis= 0)
        
    pages_df['rev_id'] = pages_df['rev_id'].astype(str)
    final_df['rev_id'] = final_df['rev_id'].astype(str)
    merged_df = pd.merge(pages_df, final_df, on='rev_id', how='outer')
else:
    merged_df = pd.read_csv('data_clean/csv/ORES_scores.csv') 

print('Number of revisions not found by ORES API: ',len(merged_df[merged_df['quality'].str.contains('RevisionNotFound') | merged_df['quality'].str.contains('unknown error')]))
def store_preprocessed(df, name):
    
    if not os.path.isdir('data_clean/csv'):
        makedirs('pre_processed_data/csv')
        
    df.to_csv('data_clean/csv/'+name+'.csv') 
if not os.path.isfile('data_clean/csv/ORES_scores.csv'):
    store_preprocessed(merged_df, 'ORES_scores')
no_scores_df = merged_df[merged_df['quality'].str.contains('RevisionNotFound:')]
no_scores_df
merged_df[merged_df['country'] == 'Hungary']
store_preprocessed(no_scores_df, 'ORES_no_scores')<jupyter_output><empty_output><jupyter_text>Combining the datasetsNow you need to combine both dataset: (1) the wikipedia articles and its ORES quality scores and (2) the population data. Both have columns named `country`. After merging the data, you'll invariably run into entries which cannot be merged. Either the population dataset does not have an entry for the equivalent Wikipedia country, or vis versa.Please remove any rows that do not have matching data, and output them to a `CSV` file called `countries-no_match.csv`. Consolidate the remaining data into a single `CSV` file called `politicians_by_country.csv`.The schema for that file should look like the following table:| article_name | country | region | revision_id | article_quality | population ||--------------|---------|--------|-------------|-----------------|------------|| Bir I of Kanem | Chad  | AFRICA | 807422778 | Stub | 16877000 |  Some exploration of matching and not matching countries<jupyter_code>contries_pop = pd.unique(population_df['country'])
print('# of countries in the population dataset: ',len(contries_pop))
countries_wiki = pd.unique(merged_df['country'])
print('# of countries in the article quality dataset: ',len(countries_wiki))
print('Countries which are present in the population df but has no matching entry in the article quality df:')
contries_pop[np.invert(np.in1d(contries_pop, countries_wiki))]
print('Countries which are present in the article quality df but has no matching entry in the population df:')
countries_wiki[np.invert(np.in1d( countries_wiki, contries_pop))]<jupyter_output>Countries which are present in the article quality df but has no matching entry in the population df:<jupyter_text>It is already visible when comparing the arrays above. Therea are countries where the matching counterpart exists with diffent writing. Checked for some random examples if countries are not mathich because of different writing. Found for example Hondura and Congo, Dem. Rep<jupyter_code>merged_df[merged_df['country'].str.contains('Hondu')]
population_df[population_df['country'].str.contains('Hondu')]
merged_df[merged_df['country'].str.contains('Congo')]
population_df[population_df['country'].str.contains('Congo')]<jupyter_output><empty_output><jupyter_text>Merging the dataframes<jupyter_code>merged_df_pop = pd.merge(merged_df, population_df, on='country', how='inner')
merged_df_outer = pd.merge(merged_df, population_df, on='country', how='left')
no_match_df = merged_df_outer[merged_df_outer['population'].isna()]
#small plausibility check. 
(len(merged_df) - len(merged_df_pop)) - len(no_match_df)
merged_df_pop
store_preprocessed(merged_df_pop, 'politicians_by_country')
store_preprocessed(no_match_df, 'countries-no_match')<jupyter_output><empty_output><jupyter_text>Step 3⃣ | AnalysisYour analysis will consist of calculating the proportion (as a percentage) of articles-per-population (we can also call it `coverage`) and high-quality articles (we can also call it `relative-quality`)for **each country** and for **each region**. By `"high quality"` arcticle we mean an article that ORES predicted as `FA` (featured article) or `GA` (good article).**Examples:*** if a country has a population of `10,000` people, and you found `10` articles about politicians from that country, then the percentage of `articles-per-population` would be `0.1%`.* if a country has `10` articles about politicians, and `2` of them are `FA` or `GA` class articles, then the percentage of `high-quality-articles` would be `20%`.  Results formatThe results from this analysis are six `data tables`. Embed these tables in the Jupyter notebook. You do not need to graph or otherwise visualize the data for this assignment. The tables will show:1. **Top 10 countries by coverage**10 highest-ranked countries in terms of number of politician articles as a proportion of country population1. **Bottom 10 countries by coverage**10 lowest-ranked countries in terms of number of politician articles as a proportion of country population1. **Top 10 countries by relative quality**10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality1. **Bottom 10 countries by relative quality**10 lowest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality1. **Regions by coverage**Ranking of regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population1. **Regions by coverage**Ranking of regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-quality**❗Hint:** You will find what country belongs to which region (e.g. `ASIA`) also in `export_2019.csv`. You need to calculate the total poulation per region. For that you could use `groupby` and also check out `apply`.  There are some missalignments with the unit in which the population is represented.For some countries the pop. is represented in millions. These are countries with high population and some countries where the population is represented in thousands. These are smaller countires like Guyana with. Without fixing this differnt representation in the dataset comparison related to population is for most of the cases wrong. In the following example only countries like *China* or *USA* or *India* have a higher pop. then 300Mil. inhabitants but smaller countries are also listed as for them the pop. is represented in thousands.<jupyter_code>#merged_df_pop = merged_df_pop[~merged_df_pop['quality'].str.contains('RevisionNotFound:')]
pd.unique(merged_df_pop[merged_df_pop['population'] > 300]['country'])<jupyter_output><empty_output><jupyter_text>But not all low pop. countries pop. is represented in thousands. So there is no rule this dataset is following:<jupyter_code>pd.unique(merged_df_pop[merged_df_pop['population'] < 0.9]['country'])<jupyter_output><empty_output><jupyter_text>1. **Top 10 countries by coverage**10 highest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>pop_artc_propotion_df = pd.DataFrame({'country': [], 'art_prop': [] }, columns = ['country', 'art_prop'])
countries = pd.unique(merged_df_pop['country'])
#ratio: (sum(articles)/population) * 100

for country in countries:
    new_row = pd.Series(data={'country':country, 
                              'art_prop': (len(merged_df_pop[merged_df_pop['country'] == country]) / merged_df_pop[merged_df_pop['country'] == country]['population'].iloc[0]) / 100 })

    pop_artc_propotion_df = pop_artc_propotion_df.append(new_row, ignore_index=True)

pop_artc_propotion_df.iloc[pop_artc_propotion_df['art_prop'].argmin()]
pop_artc_propotion_df.iloc[pop_artc_propotion_df['art_prop'].argmax()]
alt.Chart(pop_artc_propotion_df.sort_values(by=['art_prop'], ascending=False)[0:10] ).mark_bar().encode(
    y = alt.Y('country:N', sort='-x', title='country'),
    x = alt.X('art_prop:Q', title='proportion of population / number of politician articles')
).properties(
    width=800,
    height=250,
    title={
      "text": ["Top 10 countries by coverage"], 
      "subtitle": ["10 highest-ranked countries in terms of number of politician articles as a proportion of country population"],
      "color": "black",
      "subtitleColor": "darkgray"    
    }
)<jupyter_output><empty_output><jupyter_text>2. **Bottom 10 countries by coverage**10 lowest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>alt.Chart(pop_artc_propotion_df.sort_values(by=['art_prop'], ascending=True)[:10] ).mark_bar().encode(
    y = alt.Y('country:N', sort='x', title='country'),
    x = alt.X('art_prop:Q', title='proportion of population / number of politician articles')
).properties(
    width=800,
    height=250,
    title={
      "text": ["Bottom 10 countries by coverage"], 
      "subtitle": ["10 lowest-ranked countries in terms of number of politician articles as a proportion of country population"],
      "color": "black",
      "subtitleColor": "darkgray"    
    }
)<jupyter_output><empty_output><jupyter_text>3. **Top 10 countries by relative quality**10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>artc_quality_rank_df = pd.DataFrame({'country': [], 'high_ratio': [] , 'all articles': [], 'high_quality_art': [] }, columns = ['country', 'high_ratio', 'all articles','high_quality_art'])
#countries = pd.unique(merged_df_pop['country'])
#ratio: (sum(articles)/population) * 100

for country in countries:
    
    all_articles  = len(merged_df_pop[(merged_df_pop['country'] == country)])
    good_articles = len(merged_df_pop[(merged_df_pop['country'] == country) & ((merged_df_pop['quality'] == 'FA') | (merged_df_pop['quality'] == 'GA'))])
                                     
    if(good_articles > 0):
        good_art_ratio = good_articles / all_articles 
    else:
        good_art_ratio = 0
 
    new_row = pd.Series(data={'country':country, 
                              'high_ratio': good_art_ratio,
                               'all articles': all_articles,
                               'high_quality_art': good_articles})

    artc_quality_rank_df = artc_quality_rank_df.append(new_row, ignore_index=True)
#remove all which has no high quality articles.
artc_quality_rank_df = artc_quality_rank_df[artc_quality_rank_df['high_ratio'] > 0]
country = 'Switzerland'
len(merged_df_pop[(merged_df_pop['country'] == country) & ((merged_df_pop['quality'] == 'FA') | (merged_df_pop['quality'] == 'GA'))])
len(merged_df_pop[(merged_df_pop['country'] == country )])
artc_quality_rank_df.iloc[artc_quality_rank_df['high_ratio'].argmax()]
artc_quality_rank_df.iloc[artc_quality_rank_df['high_ratio'].argmin()]
top_10 = alt.Chart(artc_quality_rank_df.sort_values(by=['high_ratio'], ascending=False)[:10] ).mark_bar().encode(
    y = alt.Y('country:N', sort='-x', title='country'),
    x = alt.X('high_ratio:Q', title='proposion: total politician article / article with high quality'),
    tooltip = ['all articles:Q', 'high_quality_art:Q']
).properties(
    width=800,
    height=250,
    title={
      "text": ["Top 10 countries by relative quality"], 
      "subtitle": ["10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality"],
      "color": "black",
      "subtitleColor": "darkgray"    
    }
)<jupyter_output><empty_output><jupyter_text>4. **Bottom 10 countries by relative quality**10 lowest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>bottom_10 = alt.Chart(artc_quality_rank_df.sort_values(by=['high_ratio'], ascending=True)[:10] ).mark_bar().encode(
    y = alt.Y('country:N', sort='x', title='country'),
    x = alt.X('high_ratio:Q', title='proposion: total politician article / article with high quality'),
    tooltip = ['all articles:Q', 'high_quality_art:Q']
).properties(
    width=800,
    height=250,
    title={
      "text": ["Bottom 10 countries by relative quality"], 
      "subtitle": ["10 lowest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality"],
      "color": "black",
      "subtitleColor": "darkgray"    
    }
)
top_10 & bottom_10<jupyter_output><empty_output><jupyter_text>5. **Regions by coverage**Ranking of regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population<jupyter_code>pop_artc_propotion_df = pd.DataFrame({'region': [], 'art_prop': [] }, columns = ['region', 'art_prop'])
regions = pd.unique(merged_df_pop['region'])

for region in regions:
    new_row = pd.Series(data={'region':region, 
                              'art_prop': (len(merged_df_pop[merged_df_pop['region'] == region]) / merged_df_pop[merged_df_pop['region'] == region]['population'].iloc[0]) / 100 })

    pop_artc_propotion_df = pop_artc_propotion_df.append(new_row, ignore_index=True)

pop_artc_propotion_df
alt.Chart(pop_artc_propotion_df.sort_values(by=['art_prop'], ascending=True) ).mark_bar().encode(
    y = alt.Y('region:N', sort='-x', title='region'),
    x = alt.X('art_prop:Q', title='proportion: population / number of politician articles')
).properties(
    width=800,
    height=250,
    title={
      "text": ["Regions by coverage"], 
      "subtitle": ["Ranking of regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population"],
      "color": "black",
      "subtitleColor": "darkgray"    
    }
)<jupyter_output><empty_output><jupyter_text>6. **Regions by coverage**Ranking of regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-qualit<jupyter_code>artc_quality_rank_df = pd.DataFrame({'region': [], 'high_ratio': [] , 'all articles': [], 'high_quality_art': [] }, columns = ['region', 'high_ratio', 'all articles','high_quality_art'])

#ratio: (sum(articles)/population) * 100

for region in regions:
    all_articles  = len(merged_df_pop[(merged_df_pop['region'] == region)])
    good_articles = len(merged_df_pop[(merged_df_pop['region'] == region) & ((merged_df_pop['quality'] == 'FA') | (merged_df_pop['quality'] == 'GA'))])
                                     
    if(good_articles > 0):
        good_art_ratio = good_articles / all_articles 
    else:
        good_art_ratio = 0
 
    new_row = pd.Series(data={'region':           region, 
                              'high_ratio':       good_art_ratio,
                              'all articles':     all_articles,
                              'high_quality_art': good_articles})

    artc_quality_rank_df = artc_quality_rank_df.append(new_row, ignore_index=True)
artc_quality_rank_df
alt.Chart(artc_quality_rank_df.sort_values(by=['high_ratio'], ascending=False) ).mark_bar().encode(
    y = alt.Y('region:N', sort='-x', title='region'),
    x = alt.X('high_ratio:Q', title='proposion: total politician article / article with high quality'),
    tooltip = ['all articles:Q', 'high_quality_art:Q']
).properties(
    width=800,
    height=250,
    title={
      "text": ["Regions by coverage"], 
      "subtitle": ["Ranking of regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-qualit"],
      "color": "black",
      "subtitleColor": "darkgray"    
    }
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1258327781479014, 'fraction_numerical': 0.015198201199200533, 'mean_word_length': 4.523689052437902, 'pattern_counts': {'":': 24, '<': 63, '<?xml version=': 0, '>': 70, 'https://': 13, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '602702', 'license': 'MIT', 'n_tokens_mistral': 7521, 'n_tokens_neox': 7051, 'n_words': 2237, 'path': 'A3_Wikipedia_ORES_Bias.ipynb', 'repo_name': 'Alioio/A3-hcds-hcc-bias', 'size': '24036'}