<jupyter_start><jupyter_text>How to visualize the distribution of latent factorsWe show how to create Kernel density plots for pairs of latent factors in two or three dimensions. As illustration we use the same example as in the [introductory tutorial](../getting_started/tutorial.ipynb). For more details of how to obtain the filtered states, also see that tutorial.There are two kinds of data that can be visualized with the function described below:1. Filtered states, i.e. the estimates of the latent factors in an empirical dataset2. Simulated states, i.e. a synthetic dataset of latent factors that is generated for a parametrized model. Below, we show how to get both kinds of datasets, how to visualize the distribution of latent factors given one dataset and how to visualize the difference in distributions between two datasets. Getting filtered states<jupyter_code>with open(TEST_DIR/"model2.yaml") as y:
        model_dict = yaml.load(y, Loader=yaml.FullLoader)

params = pd.read_csv(TEST_DIR / "regression_vault" / f"one_stage_anchoring.csv")
params = params.set_index(["category", "period", "name1", "name2"])

data = pd.read_stata(TEST_DIR / "model2_simulated_data.dta")
data.set_index(["caseid", "period"], inplace=True)
filtered_states = get_filtered_states(
    model_dict=model_dict,
    data=data,
    params=params,
)
filtered_states["anchored_states"]["states"]<jupyter_output><empty_output><jupyter_text>Plotting one dataset of states<jupyter_code>fig = plot_factor_distributions(
    states=filtered_states["anchored_states"]["states"],
    model_dict=model_dict,
    add_3d_plots=True,
    period=1,
)<jupyter_output><empty_output><jupyter_text>Optional arguments of the plotting function- You can omit the 3d Plots in the upper triangle by leaving out `add_3d_plots=True`. - You can modify the trade-off between runtime and plot quality by setting `n_points`, i.e. the number of points per dimension to different values. Default is 50.- You can return the individual plots instead of a grid by setting `combine_plots_in_grid=False`. In that case the function returns a dictionary with figures that you can save for later use. - You can manually tweek the ranges over which the distributions are plotted. For that, you need to specify the argument `state_ranges`. This is a dictionary. The keys are the names of the latent factors. The values are DataFrames with the columns "period", "minimum", "maximum". The state_ranges are used to define the axis limits of the plots.- lower_kde_kws (dict): Keyword arguments for seaborn.kdeplot, used to generate the plots in the lower triangle of the grid, i.e. the two dimensional kdeplot for each factor pair.- diag_kde_kws (dict): Keyword arguments for seaborn.kdeplot, used to generate the plots on the diagonal of the grid, i.e. the one dimensional kdeplot for each factor. - surface_kws (dict): Keyword arguments for Axes.plot_surface, used to generate the plots in the upper triangle of the grid, i.e. the surface plot of the kernel density estimates for each factor pair.  Getting simulated datasets (with and without policy)One of the main application of skill formation models is to simulate the effect of counterfactual policies. To visualize the effect of a policy on factor distributions, we first need to simulate a dataset in which a policy has been active.<jupyter_code>sim_states = simulate_dataset(
    model_dict=model_dict, 
    params=params, 
    data=data,
)["anchored_states"]["states"]
policies = [
        {"period": 1, "factor": "fac1", "effect_size": 3.5, "standard_deviation": 0.0},
        {"period": 1, "factor": "fac2", "effect_size": 3.5, "standard_deviation": 0.0}]
sim_states_policy = simulate_dataset(
    model_dict=model_dict, 
    params=params, 
    data=data,
    policies=policies,
)["anchored_states"]["states"]<jupyter_output><empty_output><jupyter_text>Plotting differences in distributions<jupyter_code>fig = plot_factor_distributions(
    states= {"baseline": sim_states, "subsidy": sim_states_policy}, 
    model_dict=model_dict,
    period=1,
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08096868884540118, 'fraction_numerical': 0.00611545988258317, 'mean_word_length': 5.252293577981652, 'pattern_counts': {'":': 10, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '284993', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1200, 'n_tokens_neox': 1144, 'n_words': 468, 'path': 'docs/source/how_to_guides/how_to_visualize_pairwise_factor_distribution.ipynb', 'repo_name': 'suri5471/skillmodels', 'size': '4088'}