<jupyter_start><jupyter_text>Geographical aggregatesWe load, as an example, two data sets with age and sex aggregates per region. The data is borrowed from this book (free online, for R users): https://spatial-microsim-book.robinlovelace.net/what-is.htmlIn `SimpleWorld`, we have **33 individuals** and we know:- Distribution per age and region- Distribution per gender and regionIn particular, we also know:- Distribution per age- Distribution per gender- Distribution per regionWe are interested in calculating:- Distribution per (age, gender, region). This is done through `ipfn`  1. Read the data and turn it into "long" format.<jupyter_code>age = pd.read_csv("./data/SimpleWorld/age.csv")
gender = pd.read_csv("./data/SimpleWorld/sex.csv")
age['region'] = age.index
gender['region'] = gender.index
age
age = pd.melt(age, id_vars=['region'], value_vars=['a.50-', 'a.50+'], value_name='total', var_name='age')
gender = pd.melt(gender, id_vars=['region'], value_vars=['m', 'f'], value_name='total', var_name='gender')
age<jupyter_output><empty_output><jupyter_text>2. Calculate marginals<jupyter_code>age_marginal = age.groupby('age')['total'].sum()
gender_marginal = gender.groupby('gender')['total'].sum()
region_marginal = age.groupby('region')['total'].sum()
region_age_marginal = age.groupby(['age','region'])['total'].sum()
region_gender_marginal = gender.groupby(['region', 'gender'])['total'].sum()
age_marginal
aggregates = [age_marginal, region_marginal, gender_marginal,  region_age_marginal, region_gender_marginal]
dimensions = [['age'], ['region'], ['gender'],  ['age','region'], ['region', 'gender']]
dimensions<jupyter_output><empty_output><jupyter_text>3. Create a "placeholder" dataframe that will have the desired structure.<jupyter_code>res = pd.DataFrame(columns=['age', 'gender', 'region','total'])
for r in age['region'].unique():
    for a in age['age'].unique():
        for g in gender['gender'].unique():
            row = pd.DataFrame.from_dict({'age':[a], 'gender':[g], 'region':[r], 'total':[1]})
            res = res.append(row)
res
IPF = ipfn.ipfn(res, aggregates, dimensions)
res = IPF.iteration()
res
res['total'].sum() # This should give 33 individuals = size of the population<jupyter_output><empty_output><jupyter_text>4. Estimating income per region Based on the results from a sample data (let's say, a survey) and using the synthetic population that was generated above, we can estimate income per region.<jupyter_code>survey = pd.read_csv('./data/SimpleWorld/ind-full.csv')
survey
out = pd.merge(
    survey, res, 
    how='left', 
    left_on=['age', 'region', 'gender'], 
    right_on = ['age', 'region', 'gender'], 
)
out['total_income'] = out['total']*out['income']
out.groupby('region')['total_income'].sum()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1668460710441335, 'fraction_numerical': 0.0046645138141370645, 'mean_word_length': 6.167095115681234, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '312196', 'license': 'MIT', 'n_tokens_mistral': 914, 'n_tokens_neox': 893, 'n_words': 232, 'path': 'Example.ipynb', 'repo_name': 'jpmaldonado/spatial-microsim', 'size': '2787'}