<jupyter_start><jupyter_text>Inequality Data Processing (WIID) Data Dictionary| Variable           | Definition                                                                                                                                                                                         ||--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|| id                 | Identifier                                                                                                                                                                                         || country            | Country/area                                                                                                                                                                                       || c3                 | 3-digit country code in ISO 3166-1 alpha-3 format                                                                                                                                                  || c2                 | 2-digit country code in ISO 3166-1 alpha-2 format                                                                                                                                                  || year               | Year. Note that when a survey continues for more than a year, the year when it is finished is considered                                                                                           || gini_reported      | Gini coefficient as reported by the source (in most cases based on microdata, in some older observations estimates derive from grouped data)                                                       || q1-q5              | Quintile group shares of resource                                                                                                                                                                  || d1-d10             | Decile group shares of resource                                                                                                                                                                    || bottom5 and top5   | Bottom five and top five percent group shares of resource                                                                                                                                          || resource           | Resource concept                                                                                                                                                                                   || resource_detailed  | Detailed resource concept                                                                                                                                                                          || scale              | Equivalence scale                                                                                                                                                                                  || scale_detailed     | Detailed equivalence scale                                                                                                                                                                         || sharing_unit       | Income sharing unit/statistical unit                                                                                                                                                               || reference_unit     | Unit of analysis, indicates whether the data has been weighted with a person or a household weight                                                                                                 || areacovr           | Area coverage. The land area which was included in the original sample surveys etc.                                                                                                                || areacovr_detailed  | Detailed area coverage                                                                                                                                                                             || popcovr            | Population coverage. The population covered in the sample surveys in the land area (all, rural, urban etc.) which was included                                                                     || popcovr_detailed   | Detailed population coverage, including age coverage information in certain cases                                                                                                                  || region_un          | Regional grouping based on United Nations geoscheme                                                                                                                                                || region_un_sub      | Sub-regional grouping based on United Nations geoscheme                                                                                                                                            || region_wb          | Regional grouping based on World Bank classification                                                                                                                                               || eu                 | Current EU member state                                                                                                                                                                            || oecd               | Current OECD member state                                                                                                                                                                          || incomegroup        | World Bank classification by country income                                                                                                                                                        || mean               | Survey mean given with the same underlying definitions as the Gini coefficient and the share data                                                                                                  || median             | Survey median given with the same underlying definitions as the Gini coefficient and the share data                                                                                                || currency           | Currency for the mean and median values. If the reference is US$2011PPP it means that the currency is in 2011 US dollar per month, with purchasing power parity applied on it.                     || reference_period   | Time period for measuring mean and median values                                                                                                                                                   || exchangerate       | Conversion rate from local currency units (LCU) to United States Dollars (USD)                                                                                                                     || mean_usd           | Mean measure in United States Dollar (USD)                                                                                                                                                         || median_usd         | Median measure in United States Dollar (USD)                                                                                                                                                       || gdp_ppp_pc_usd2011 | Gross Domestic Product (GDP) is converted to United States Dollars (USD) using purchasing power parity rates and divided by total population. Data are in constant 2011 United States Dollar (USD) || population         | Population of countries from the UN population prospects                                                                                                                                           || revision           | Indicates the time of the revision when the observation was included to the database                                                                                                               || quality            | Quality assessment                                                                                                                                                                                 || quality_score      | Computed quality score                                                                                                                                                                             || source             | Source type                                                                                                                                                                                        || source_detailed    | Source from which the observation was obtained                                                                                                                                                     || source_comments    | Additional source comments                                                                                                                                                                         || survey             | Originating survey information                                                                                                                                                                     |<jupyter_code>import re

import numpy as np
import pandas as pd
import pycountry

%matplotlib inline

pd.set_option('display.float_format', lambda x: '%.3f' % x)
pd.set_option('display.max_columns', None)<jupyter_output><empty_output><jupyter_text>Load The File<jupyter_code>df = pd.read_excel('../data/external/Inequality/WIID/WIID_19Dec2018.xlsx')<jupyter_output><empty_output><jupyter_text>Standardize Country Codes<jupyter_code>""" Only Select rows with valid country codes
"""
country_locations = []
for country in df['c3']:
    try:
        pycountry.countries.lookup(country)
        country_locations.append(True)
    except LookupError:
        country_locations.append(False)
df = df[country_locations]<jupyter_output><empty_output><jupyter_text>Standardize Indexes<jupyter_code>df.rename(
    {
        "c3": "Country Code",
        "year": "Year"
    },
    axis='columns',
    inplace=True)<jupyter_output><empty_output><jupyter_text>Remove out of scope rows (consumption/gross)<jupyter_code>df = df[(df.resource != "Consumption")]<jupyter_output><empty_output><jupyter_text>Remove out of scope rows by year<jupyter_code>df = df[df["Year"] > 1994]
df = df[df["Year"] < 2018]

df = df.groupby(["Country Code","Year"]).mean()<jupyter_output><empty_output><jupyter_text>Save Data<jupyter_code>df.to_pickle("../data/processed/Inequality_WIID.pickle")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05420801176038221, 'fraction_numerical': 0.005145167217934583, 'mean_word_length': 0.5356941309255079, 'pattern_counts': {'":': 2, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 268, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '6356', 'license': 'MIT', 'n_tokens_mistral': 1311, 'n_tokens_neox': 1229, 'n_words': 489, 'path': 'notebooks/7.ipynb', 'repo_name': 'noahbjohnson/senior-project', 'size': '10884'}