<jupyter_start><jupyter_text>Doppelganger! (Simple)Welcome to the simplified Doppelganger example. If you have not already done so, please see the [README document](https://github.com/sidewalklabs/doppelganger/blob/master/README.md) for installation instructions and information on what Doppelganger is doing under the hood. For a more thorough walkthrough, take a look at [doppelganger_example_full](./doppelganger_example_full.ipynb).  Getting StartedDoppelganger lets you configure which census fields you use, the relationships among these fields (network structure), and the data preprocessing. We'll begin by loading the necessary packages, and then load a simple configuration file.<jupyter_code>import pandas as pd

from doppelganger import (
    allocation,
    inputs,
    Configuration,
    HouseholdAllocator,
    PumsData,
    SegmentedData,
    BayesianNetworkModel,
    Population,
    Preprocessor,
    Marginals
)

configuration = Configuration.from_file('sample_data/config.json')<jupyter_output><empty_output><jupyter_text>Loading and Cleaning DataThe following loads our data and cleans it according to the configuration. Reusing the same preprocessor ensures all data is cleaned consistently. We'll use California's PUMA `00106` for our demonstration.<jupyter_code>PUMA = '00106'

preprocessor = Preprocessor.from_config(configuration.preprocessing_config)

# Take pums fields from the config and from the default fields needed for
# the household allocation process.
household_fields = tuple(set(
    field.name for field in allocation.DEFAULT_HOUSEHOLD_FIELDS).union(
        set(configuration.household_fields)
))

households_data = PumsData.from_csv('sample_data/households_00106_dirty.csv').clean(
    household_fields, preprocessor, puma=PUMA
)

persons_fields = tuple(set(
    field.name for field in allocation.DEFAULT_PERSON_FIELDS).union(
        set(configuration.person_fields)
))
persons_data = PumsData.from_csv('sample_data/persons_00106_dirty.csv').clean(
    persons_fields, preprocessor, puma=PUMA
)<jupyter_output><empty_output><jupyter_text>Household AllocationNow we will allocate persons and households to tracts to align with census controls.  First, load our controls based on ACS marginals.<jupyter_code>controls = Marginals.from_csv('sample_data/marginals_00106.csv')<jupyter_output><empty_output><jupyter_text>Now use `HouseholdAllocator` to generate household allocations.<jupyter_code>allocator = HouseholdAllocator.from_cleaned_data(controls, households_data, persons_data)<jupyter_output><empty_output><jupyter_text>Bayesian Network GenerationLet's create models to generate characteristics for the people and households we just allocated. We'll start by loading up our pums data.  Our model learns different probability distributions for each category of person.  The category can be whatever you want and is specified by passing a segmentation function when you load training data.<jupyter_code>segmentation_function = lambda x: x[inputs.AGE.name]
person_training_data = SegmentedData.from_data(
    persons_data,
    list(configuration.person_fields),
    inputs.PERSON_WEIGHT.name,
    segmenter=segmentation_function
)
person_model = BayesianNetworkModel.train(
    person_training_data,
    configuration.person_structure,
    configuration.person_fields
)

household_segmenter = lambda x: x[inputs.NUM_PEOPLE.name]

household_training_data = SegmentedData.from_data(
    households_data,
    list(configuration.household_fields),
    inputs.HOUSEHOLD_WEIGHT.name,
    household_segmenter,
)
household_model = BayesianNetworkModel.train(
    household_training_data,
    configuration.household_structure,
    configuration.household_fields
)<jupyter_output><empty_output><jupyter_text>Population SynthesisNow for the main event! We can synthesize a population by taking the household allocationswe produced above and filling out missing categories with our Bayesian Networks.<jupyter_code>population = Population.generate(allocator, person_model, household_model)<jupyter_output><empty_output><jupyter_text>We can access the people and households as Pandas DataFrames and work with them directly. Households and people are unique by household_id. We can also join them to create a fat table of individual and household attributes.<jupyter_code>people = population.generated_people
households = population.generated_households

merge_cols = [inputs.HOUSEHOLD_ID.name]
combined = pd.merge(people, households, on=merge_cols)

print(combined)<jupyter_output><empty_output><jupyter_text>We can easily save this population to a csv.<jupyter_code>population.write('generated_people.csv', 'generated_households.csv')<jupyter_output><empty_output><jupyter_text>We can additionally save any of our intermediary stages and load them up againwhenever we want.  For example, we could save our Bayesian Network and reuse themagain later with the same or different household allocations.<jupyter_code>person_model.write('person_model.json')
person_model_reloaded = BayesianNetworkModel.from_file('person_model.json', segmenter=segmentation_function)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06636736287331642, 'fraction_numerical': 0.004879953152449736, 'mean_word_length': 6.579881656804734, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '211915', 'license': 'Apache-2.0', 'n_tokens_mistral': 1477, 'n_tokens_neox': 1425, 'n_words': 411, 'path': 'examples/doppelganger_example_simple.ipynb', 'repo_name': 'christophertull/doppelganger', 'size': '5123'}