<jupyter_start><jupyter_text>Read the CSV Files<jupyter_code>page_data = pd.read_csv("../data/page_data.csv")
page_data.head()
population = pd.read_csv("../data/WPDS_2020_data - WPDS_2020_data.csv")
population.head()<jupyter_output><empty_output><jupyter_text>Data Processing The page_data contains some page names that start with the string "Template:". These pages are not Wikipedia articles, and should not be included in your analysis.<jupyter_code># Filter out the data with page names starting with "Template:"
page_data = page_data[~page_data['page'].str.startswith('Template:')]
# Reset the index
page_data.reset_index(drop=True, inplace=True)
page_data.head()<jupyter_output><empty_output><jupyter_text>WPDS_2020_data.csv contains some rows that provide cumulative regional population counts, rather than country-level counts. These rows are distinguished by having ALL CAPS values in the 'geography' field (e.g. AFRICA, OCEANIA). These rows won't match the country values in page_data.csv.<jupyter_code># Get the population of countries. (Excluding cumulative regional counts)
population_country = population[population['Type'] == 'Country']
population_country.reset_index(drop=True, inplace=True)
population_country.head()<jupyter_output><empty_output><jupyter_text>Obtain article quality Now you need to get the predicted quality scores for each article in the Wikipedia dataset. We're using a machine learning system called ORES. This was originally an acronym for "Objective Revision Evaluation Service" but was simply renamed “ORES”. ORES is a machine learning tool that can provide estimates of Wikipedia article quality. The article quality estimates are, from best to worst:  FA - Featured article  GA - Good article  B - B-class article  C - C-class article  Start - Start-class article  Stub - Stub-class article  These were learned based on articles in Wikipedia that were peer-reviewed using the Wikipedia content assessment procedures.These quality classes are a sub-set of quality assessment categories developed by Wikipedia editors. For this assignment, you only need to know that these categories exist, and that ORES will assign one of these 6 categories to any rev_id you send it.  In order to get article predictions for each article in the Wikipedia dataset, you will first need to read page_data.csv into Python (or R), and then read through the dataset line by line, using the value of the rev_id column to make an API query.<jupyter_code># Obtain rev_ids as a list
rev_ids = page_data['rev_id'].tolist()
import requests

headers = {
    'User-Agent': 'https://github.com/azhou5211',
    'From': '<EMAIL>'
}

def api_call(endpoint, rev_id):
    """
    Function used to call API. Will return response.
    
    :param endpoint: API URL endpoint
    :param parameters: Parameter settings in the API call
    """
    call = requests.get(endpoint.format(rev_id=rev_id), headers=headers)
    response = call.json()
    return response
request_url = 'https://ores.wikimedia.org/v3/scores/enwiki/{rev_id}/articlequality'
# Get the article quality for each wikipedia article
article_quality = []
for rev_id in rev_ids:
    response = api_call(request_url, rev_id)
    try:
        pred = response['enwiki']['scores'][str(rev_id)]['articlequality']['score']['prediction']
    except:
        pred = np.nan
    article_quality.append(pred)
# Add the article quality into the pages dataframe
page_data['quality'] = article_quality
page_data.head()<jupyter_output><empty_output><jupyter_text>List of wikipedia pages that did not have a predicted quality from the ORES<jupyter_code># List of wikipedia pages that did not have a predicted quality from ORES
page_data[page_data['quality'].isna()]
page_data[page_data['quality'].isna()].to_csv("Non-Predicted ORES pages.csv", index=False)<jupyter_output><empty_output><jupyter_text>Filter the page_data to include only the values that have a predicted quality<jupyter_code>page_data_filtered = page_data[~page_data['quality'].isna()]
page_data_filtered<jupyter_output><empty_output><jupyter_text>Combining the datasets Merge the wikipedia data and population data together.   Both have fields containing country names for just that purpose. After merging the data, you'll invariably run into entries which cannot be merged. Either the population dataset does not have an entry for the equivalent Wikipedia country, or vise versa.<jupyter_code>df = page_data_filtered.merge(population_country, left_on='country', right_on='Name', how='left')
df<jupyter_output><empty_output><jupyter_text>Filter to only the relevant columns<jupyter_code>df = df[['country','page','rev_id','quality','Population']]
df.rename(columns={'page':'article_name','rev_id':'revision_id','quality':'article_quality_est.','Population':'population'}, inplace=True)
df<jupyter_output>/usr/local/lib/python3.7/site-packages/pandas/core/frame.py:4308: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,<jupyter_text>Filter the rows that do not have data and output it to ```wp_wpds_countries-no_match.csv```<jupyter_code>df[df['population'].isna()].to_csv("../output_files/wp_wpds_countries-no_match.csv", index=False)
df[df['population'].isna()]<jupyter_output><empty_output><jupyter_text>Filter the df to contain non null values. Output the dataframe to ```wp_wpds_politicians_by_country.csv```<jupyter_code>df = df[~df['population'].isna()]
df.to_csv("../output_files/wp_wpds_politicians_by_country.csv", index=False)
df<jupyter_output><empty_output><jupyter_text>Analysis Calculate the proportion (as a percentage) of articles-per-population and high-quality articles for each country AND for each geographic region.  By "high quality" articles, in this case we mean the number of articles about politicians in a given country that ORES predicted would be in either the "FA" (featured article) or "GA" (good article) classes. Filter the data to include only "high" quality articles<jupyter_code>df_analysis = df[(df['article_quality_est.']=='FA') | (df['article_quality_est.']=='GA')]
df_analysis<jupyter_output><empty_output><jupyter_text>Groupby country to get the count of high quality papers<jupyter_code>df_analysis_country = df_analysis.groupby('country').agg({"article_quality_est.":'count', 'population':'first'})
df_analysis_country.rename(columns={"article_quality_est.":"count_of_high_quality"}, inplace=True)
df_analysis_country<jupyter_output><empty_output><jupyter_text>Calculate the percentage of articles per population<jupyter_code>df_analysis_country['percentage_of_articles-per-population'] = df_analysis_country['count_of_high_quality']/df_analysis_country['population']
df_analysis_country<jupyter_output><empty_output><jupyter_text>Get the total count of articles and add it to the analysis dataframe<jupyter_code>count_of_articles = df.groupby('country').agg({"article_quality_est.":'count'})
df_analysis_country = df_analysis_country.merge(count_of_articles, how="left", left_index=True, right_index=True)
df_analysis_country<jupyter_output><empty_output><jupyter_text>Calculate the percentage of high-quality articles<jupyter_code>df_analysis_country['percentage_of_high-quality_articles'] = df_analysis_country['count_of_high_quality']/df_analysis_country['article_quality_est.']
df_analysis_country.rename(columns={'article_quality_est.':'total_count'}, inplace=True)
df_analysis_country<jupyter_output><empty_output><jupyter_text>Results  Top 10 countries by coverage: 10 highest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>df_analysis_country.sort_values(by=['percentage_of_articles-per-population'], ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Bottom 10 countries by coverage: 10 lowest-ranked countries in terms of number of politician articles as a proportion of country population<jupyter_code>df_analysis_country.sort_values(by=['percentage_of_articles-per-population']).head(10)<jupyter_output><empty_output><jupyter_text>Top 10 countries by relative quality: 10 highest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>df_analysis_country.sort_values(by=['percentage_of_high-quality_articles'], ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Bottom 10 countries by relative quality: 10 lowest-ranked countries in terms of the relative proportion of politician articles that are of GA and FA-quality<jupyter_code>df_analysis_country.sort_values(by=['percentage_of_high-quality_articles']).head(10)<jupyter_output><empty_output><jupyter_text>Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population Obtain the regions from the population dataframe<jupyter_code>region = "NORTHERN AFRICA"

regions = ['WORLD', 'AFRICA', 'NORTHERN AFRICA']
for i in range(3, len(population)):
    if population.iloc[i]['Type']=='Sub-Region':
        region = population.iloc[i]['Name']
    regions.append(region)
population['Region'] = regions
population
df_analysis_country.reset_index(inplace=True)<jupyter_output><empty_output><jupyter_text>Merge the region with analysis<jupyter_code>df_analysis_country_ = df_analysis_country.merge(population[['Name', 'Region']], left_on='country', right_on='Name', how='left')
df_analysis_country_.drop(columns=['Name'], inplace=True)
df_analysis_country_<jupyter_output><empty_output><jupyter_text>Groupby region and sum the paper counts<jupyter_code>df_analysis_region = df_analysis_country_.groupby('Region').agg({'count_of_high_quality':'sum','total_count':'sum'})
df_analysis_region<jupyter_output><empty_output><jupyter_text>Merge the population value<jupyter_code>population_region = population[population['Type']=='Sub-Region']
df_analysis_region_ = df_analysis_region.merge(population_region[['Name', 'Population']], left_index=True, right_on='Name')
df_analysis_region_<jupyter_output><empty_output><jupyter_text>Calculate percentage of articles-per-population and percentage of high-quality articles by region<jupyter_code>df_analysis_region_['percentage_of_articles-per-population'] = df_analysis_region_['count_of_high_quality']/df_analysis_region_['Population']
df_analysis_region_['percentage_of_high-quality_articles'] = df_analysis_region_['count_of_high_quality']/df_analysis_region_['total_count']
df_analysis_region_ = df_analysis_region_[['Name', 'percentage_of_articles-per-population', 'percentage_of_high-quality_articles']]
df_analysis_region_.rename(columns={'Name':'Region'}, inplace=True)
df_analysis_region_<jupyter_output>/usr/local/lib/python3.7/site-packages/pandas/core/frame.py:4308: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,<jupyter_text>Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the total count of politician articles from countries in each region as a proportion of total regional population<jupyter_code>df_analysis_region_.sort_values(by=['percentage_of_articles-per-population'], ascending=False)<jupyter_output><empty_output><jupyter_text>Geographic regions by coverage: Ranking of geographic regions (in descending order) in terms of the relative proportion of politician articles from countries in each region that are of GA and FA-quality<jupyter_code>df_analysis_region_.sort_values(by=['percentage_of_high-quality_articles'], ascending=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09920201595968081, 'fraction_numerical': 0.004619907601847963, 'mean_word_length': 7.8652271034996275, 'pattern_counts': {'":': 3, '<': 104, '<?xml version=': 0, '>': 104, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '491167', 'license': 'MIT', 'n_tokens_mistral': 3658, 'n_tokens_neox': 3520, 'n_words': 941, 'path': 'notebooks/hcds-a2-bias.ipynb', 'repo_name': 'azhou5211/data-512-a2', 'size': '11910'}