<jupyter_start><jupyter_text>Purpose* Load data an filter the variables that have been selected for the study.* Remove instances that are noisy or contain missing valuesFor information about the original dataset, you can read the document "data/mixed/spanish_living_conditions/disreg_ecv18.doc". Data has been downloaded from here. Note: both the document and the web are in spanish.  Load original data<jupyter_code>import pyarrow.parquet as pq
import pandas as pd
import numpy as np

esudb18d = pd.read_csv("../data/mixed/spanish_living_conditions/esudb18d.csv")
esudb18d.name = "esudb18d"
print(esudb18d.shape)

esudb18h = pd.read_csv("../data/mixed/spanish_living_conditions/esudb18h.csv")
esudb18h.name = "esudb18h"
print(esudb18h.shape)

esudb18p = pd.read_csv("../data/mixed/spanish_living_conditions/esudb18p.csv")
esudb18p.name = "esudb18p"
print(esudb18p.shape)

esudb18r = pd.read_csv("../data/mixed/spanish_living_conditions/esudb18r.csv")
esudb18r.name = "esudb18r"
print(esudb18r.shape)<jupyter_output>(13368, 11)
(13368, 153)
(28372, 236)
(33734, 37)<jupyter_text>Select variablesWe will focus on variables related to house and family.<jupyter_code># Combine data from esdub18d and esudb18h
esudb18dh = esudb18d.join(esudb18h)
print(esudb18dh.shape)

# Remove those instances with missing or noisy values
esudb18dh = esudb18dh.drop(esudb18dh[(esudb18dh.HY020_F == -1) | 
                                     (esudb18dh.HY020_F > 19999) | # Check it is net income (it starts with 2)
                                     (esudb18dh.HY030N_F == -1) |
                                     (esudb18dh.HS021_F == -2) |
                                     (esudb18dh.HS021_F == -1) |
                                     (esudb18dh.HS040_F == -1) |
                                     (esudb18dh.HS050_F == -1) |
                                     (esudb18dh.HS060_F == -1) |
                                     (esudb18dh.HS070_F == -1) |
                                     (esudb18dh.HS080_F == -1) |
                                     (esudb18dh.HS090_F == -1) |
                                     (esudb18dh.HS100_F == -1) |
                                     (esudb18dh.HS110_F == -1) |
                                     (esudb18dh.HS120_F == -1) |
                                     (esudb18dh.HS160_F == -1) |
                                     (esudb18dh.HS170_F == -1) |
                                     (esudb18dh.HS180_F == -1) |
                                     (esudb18dh.HS190_F == -1) |
                                     (esudb18dh.HH010_F == -1) |
                                     (esudb18dh.HH021_F == -1) |
                                     (esudb18dh.HH030_F == -1) |
                                     (esudb18dh.HH040_F == -1) |
                                     (esudb18dh.HH050_F == -1) |
                                     (esudb18dh.HH081_F == -1) |
                                     (esudb18dh.HH091_F == -1) |
                                     (esudb18dh.HX060 == ' ')
                                    ].index)
print(esudb18dh.shape)

selected_columns = ["HY020", "HY030N","HS021", "HS040", "HS050", "HS060", "HS070", "HS080", "HS090", "HS100", "HS110",
                   "HS120", "HS160", "HS170", "HS180", "HS190", "HH010", "HH021", "HH030", "HH040", "HH050", "HH081", "HH091",
                   "HX040", "HX060", "DB100", "HX240"]
print(esudb18dh[selected_columns].shape)<jupyter_output>(13368, 164)
(13222, 164)
(13222, 27)<jupyter_text>Estimate the equivalent incomeFor a better comparison of the income of different types of households (number of people who are part of the household and different ages), the concept of equivalent household income is used, which standardizes households according to the number of equivalent consumption units that make them up.The concept of equivalent consumption unit takes into account the economies of scale that occur when all members of a household share the income, and the number of equivalent consumption units is determined using the modified OECD scale. A person living alone constitutes a unit of consumption, since it alone covers all the household expenses. The first adult in the household is assigned a weight of 1, the other adults a weight of 0.5 and a weight of 0.3 for children under 14 years of age.The equivalent income of a household is calculated by dividing the total disposable income of the household by the number of equivalent consumption units that comprise it. This equivalent income is assigned equally to all members who are part of the household, obtaining an indicator of the economic resources per unit of consumption of the people (or equivalent income of the person) available in a standardized household.<jupyter_code>equivalent_income = esudb18dh["HY020"] / esudb18dh["HX240"]
equivalent_income<jupyter_output><empty_output><jupyter_text>Generate the dataset of the study 1. Urban_degree (**DB100**)2. Home_rooms (**HH030**)3. Home_ownership (**HH021**)4. Family_members (**HX040**)5. Equivalent_income (**HY020** divided by **HX240**)  Income divided by the number of consumer units6. invoices_overdue (**HS021**)7. Afford_holidays (**HS040**)8. Afford_meal (**HS050**)9. Afford_unexpected (**HS060**)10. Have_phone (**HS070**)11. Have_tv (**HS080**)12. Have_computer (**HS090**)13. Have_washer (**HS100**)14. Have_car (**HS110**)15. Have_shower (**HH081**)16. Have_wc (**HH091**)17. Problem_light (**HS160**)18. Problem_noise (**HS170**)19. Problem_pollution (**HS180**)20. Problem_vandal (**HS190**)21. Problem_leaks (**HH040**)22. Problem_temp (**HH050**)<jupyter_code>df = pd.DataFrame()
df["urban_degree"] = esudb18dh["DB100"]
df["home_rooms"] = esudb18dh["HH030"]
df["home_ownership"] = esudb18dh["HH021"]
df["family_members"] = esudb18dh["HX040"]
df["equivalent_income"] = equivalent_income
df["invoices_overdue"] = esudb18dh["HS021"]
df["afford_holidays"] = esudb18dh["HS040"]
df["afford_meal"] = esudb18dh["HS050"]
df["afford_unexpected"] = esudb18dh["HS060"]
df["have_phone"] = esudb18dh["HS070"]
df["have_tv"] = esudb18dh["HS080"]
df["have_computer"] = esudb18dh["HS090"]
df["have_washer"] = esudb18dh["HS100"]
df["have_car"] = esudb18dh["HS110"]
df["have_shower"] = esudb18dh["HH081"]
df["have_wc"] = esudb18dh["HH091"]
df["problem_light"] = esudb18dh["HS160"]
df["problem_noise"] = esudb18dh["HS170"]
df["problem_pollution"] = esudb18dh["HS180"]
df["problem_vandal"] = esudb18dh["HS190"]
df["problem_leaks"] = esudb18dh["HH040"]
df["problem_temp"] = esudb18dh["HH050"]

df = df.round(decimals = 2)
df<jupyter_output><empty_output><jupyter_text>Generate categorical variablesThe reason why we generate categorical variables is twofold:* Rename the states of variables so that they are understandable without looking at the Word document in Spanish.* Combine the states "cant_afford" and "no" of the variables "have_tv", "have_shower", "have _...", etc.* Combine the "rent_low" and "free_transfer" states of the "home_ownership" variable so that we can use the priors of the ECH data. Rename variable states<jupyter_code>from pandas import Series, Categorical


df["urban_degree"] = df["urban_degree"].astype('category')
df["urban_degree"] = df["urban_degree"].cat.rename_categories(["high_pop", "medium_pop", "low_pop"])

df["home_ownership"] = df["home_ownership"].astype('category')
df["home_ownership"] = df["home_ownership"].cat.rename_categories(["property", "mortgage", "rent", "rent_low", "rent_low_or_free_transfer"])

df["invoices_overdue"] = df["invoices_overdue"].astype('category')
df["invoices_overdue"] = df["invoices_overdue"].cat.rename_categories(["yes", "yes_multiple", "no"])

df["afford_holidays"] = df["afford_holidays"].astype('int32')
df["afford_holidays"] = df["afford_holidays"].astype('category')
df["afford_holidays"] = df["afford_holidays"].cat.rename_categories(["yes", "no"])

df["afford_meal"] = df["afford_meal"].astype('int32')
df["afford_meal"] = df["afford_meal"].astype('category')
df["afford_meal"] = df["afford_meal"].cat.rename_categories(["yes", "no"])

df["afford_unexpected"] = df["afford_unexpected"].astype('int32')
df["afford_unexpected"] = df["afford_unexpected"].astype('category')
df["afford_unexpected"] = df["afford_unexpected"].cat.rename_categories(["yes", "no"])

df["have_phone"] = df["have_phone"].astype('int32')
df["have_phone"] = df["have_phone"].astype('category')
df["have_phone"] = df["have_phone"]= df["have_phone"].cat.rename_categories(["yes", "no", "cant"])

df["have_tv"] = df["have_tv"].astype('int32')
df["have_tv"] = df["have_tv"].astype('category')
df["have_tv"] = df["have_tv"].cat.rename_categories(["yes", "no", "cant"])

df["have_computer"] = df["have_computer"].astype('int32')
df["have_computer"] = df["have_computer"].astype('category')
df["have_computer"] = df["have_computer"].cat.rename_categories(["yes", "no", "cant"])

df["have_washer"] = df["have_washer"].astype('int32')
df["have_washer"] = df["have_washer"].astype('category')
df["have_washer"] = df["have_washer"].cat.rename_categories(["yes", "no", "cant"])

df["have_car"] = df["have_car"].astype('int32')
df["have_car"] = df["have_car"].astype('category')
df["have_car"] = df["have_car"].cat.rename_categories(["yes", "no", "cant"])

df["have_shower"] = df["have_shower"].astype('int32')
df["have_shower"] = df["have_shower"].astype('category')
df["have_shower"] = df["have_shower"].cat.rename_categories(["yes", "no", "cant"])

df["have_wc"] = df["have_wc"].astype('int32')
df["have_wc"] = df["have_wc"].astype('category')
df["have_wc"] = df["have_wc"].cat.rename_categories(["yes", "no", "cant"])

df["problem_light"] = df["problem_light"].astype('int32')
df["problem_light"] = df["problem_light"].astype('category')
df["problem_light"] = df["problem_light"].cat.rename_categories(["yes", "no"])

df["problem_noise"] = df["problem_noise"].astype('int32')
df["problem_noise"] = df["problem_noise"].astype('category')
df["problem_noise"] = df["problem_noise"].cat.rename_categories(["yes", "no"])

df["problem_pollution"] = df["problem_pollution"].astype('int32')
df["problem_pollution"] = df["problem_pollution"].astype('category')
df["problem_pollution"] = df["problem_pollution"].cat.rename_categories(["yes", "no"])

df["problem_vandal"] = df["problem_vandal"].astype('int32')
df["problem_vandal"] = df["problem_vandal"].astype('category')
df["problem_vandal"] = df["problem_vandal"].cat.rename_categories(["yes", "no"])

df["problem_leaks"] = df["problem_leaks"].astype('int32')
df["problem_leaks"] = df["problem_leaks"].astype('category')
df["problem_leaks"] = df["problem_leaks"].cat.rename_categories(["yes", "no"])

df["problem_temp"] = df["problem_temp"].astype('int32')
df["problem_temp"] = df["problem_temp"].astype('category')
df["problem_temp"] = df["problem_temp"].cat.rename_categories(["yes", "no"])<jupyter_output><empty_output><jupyter_text>Combine states<jupyter_code>df["have_phone"][df["have_phone"] == "cant"] = "no"
df["have_phone"] = df["have_phone"].cat.remove_unused_categories()

df["have_tv"][df["have_tv"] == "cant"] = "no"
df["have_tv"] = df["have_tv"].cat.remove_unused_categories()

df["have_computer"][df["have_computer"] == "cant"] = "no"
df["have_computer"] = df["have_computer"].cat.remove_unused_categories()

df["have_washer"][df["have_washer"] == "cant"] = "no"
df["have_washer"] = df["have_washer"].cat.remove_unused_categories()

df["have_car"][df["have_car"] == "cant"] = "no"
df["have_car"] = df["have_car"].cat.remove_unused_categories()

df["have_shower"][df["have_shower"] == "cant"] = "no"
df["have_shower"] = df["have_shower"].cat.remove_unused_categories()

df["have_wc"][df["have_wc"] == "cant"] = "no"
df["have_wc"] = df["have_wc"].cat.remove_unused_categories()

df["invoices_overdue"][df["invoices_overdue"] == "yes_multiple"] = "yes"
df["invoices_overdue"] = df["invoices_overdue"].cat.remove_unused_categories()

df["home_ownership"][df["home_ownership"] == "rent_low"] = "rent_low_or_free_transfer"
df["home_ownership"] = df["home_ownership"].cat.remove_unused_categories()
df["home_ownership"]
df<jupyter_output><empty_output><jupyter_text>Export data in CSV format<jupyter_code>df.to_csv("spanish_living_conditions.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1692797648211661, 'fraction_numerical': 0.05038379879144211, 'mean_word_length': 4.472296693476318, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 27, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '483441', 'license': 'Apache-2.0', 'n_tokens_mistral': 4667, 'n_tokens_neox': 4191, 'n_words': 788, 'path': 'python-project/prepare_slc_data.ipynb', 'repo_name': 'ferjorosa/bayesian-latent-forests', 'size': '12246'}