<jupyter_start><jupyter_text>Segregation Index Decomposition  Table of Contents* [Decomposition framework of the PySAL *segregation* module](Decomposition-framework-of-the-PySAL-*segregation*-module)	* [Map of the composition of the Metropolitan area of Los Angeles](Map-of-the-composition-of-the-Metropolitan-area-of-Los-Angeles)	* [Map of the composition of the Metropolitan area of New York](Map-of-the-composition-of-the-Metropolitan-area-of-New-York)	* [Composition Approach (default)](Composition-Approach-%28default%29)	* [Share Approach](Share-Approach)	* [Dual Composition Approach](Dual-Composition-Approach)	* [Inspecting a different index: Relative Concentration](Inspecting-a-different-index:-Relative-Concentration) This is a notebook that explains a step-by-step procedure to perform decomposition on comparative segregation measures.First, let's import all the needed libraries.<jupyter_code>import pandas as pd
import pickle
import numpy as np
import matplotlib.pyplot as plt

import segregation
from segregation.decomposition import DecomposeSegregation<jupyter_output><empty_output><jupyter_text>In this example, we are going to use census data that the user must download its own copy, following similar guidelines explained in https://github.com/spatialucr/geosnap/blob/master/examples/01_getting_started.ipynb where you should download the full type file  of 2010. The zipped file download will have a name that looks like `LTDB_Std_All_fullcount.zip`. After extracting the zipped content, the filepath of the data should looks like this:<jupyter_code>#filepath = '~/LTDB_Std_2010_fullcount.csv'<jupyter_output><empty_output><jupyter_text>Then, we read the data:<jupyter_code>df = pd.read_csv(filepath, encoding = "ISO-8859-1", sep = ",")<jupyter_output><empty_output><jupyter_text>We are going to work with the variable of the nonhispanic black people (`nhblk10`) and the total population of each unit (`pop10`). So, let's read the map of all census tracts of US and select some specific columns for the analysis:<jupyter_code># This file can be download here: https://drive.google.com/open?id=1gWF0OCn6xuR_WrEj7Ot2jY6KI2t6taIm
with open('data/tracts_US.pkl', 'rb') as input:
    map_gpd = pickle.load(input)
    
map_gpd['INTGEOID10'] = pd.to_numeric(map_gpd["GEOID10"])
gdf_pre = map_gpd.merge(df, left_on = 'INTGEOID10', right_on = 'tractid')
gdf = gdf_pre[['GEOID10', 'geometry', 'pop10', 'nhblk10']]<jupyter_output><empty_output><jupyter_text>In this notebook, we use the Metropolitan Statistical Area (MSA) of US (we're also using the word 'cities' here to refer them). So, let's read the correspondence table that relates the tract id with the corresponding Metropolitan area...<jupyter_code># You can download this file here: https://drive.google.com/open?id=10HUUJSy9dkZS6m4vCVZ-8GiwH0EXqIau
with open('data/tract_metro_corresp.pkl', 'rb') as input:
    tract_metro_corresp = pickle.load(input).drop_duplicates()<jupyter_output><empty_output><jupyter_text>..and merge them with the previous data.<jupyter_code>merged_gdf = gdf.merge(tract_metro_corresp, left_on = 'GEOID10', right_on = 'geoid10')<jupyter_output><empty_output><jupyter_text>We now build the composition variable (`compo`) which is the division of the frequency of the chosen group and total population. Let's inspect the first rows of the data.<jupyter_code>merged_gdf['compo'] = np.where(merged_gdf['pop10'] == 0, 0, merged_gdf['nhblk10'] / merged_gdf['pop10'])
merged_gdf.head()<jupyter_output><empty_output><jupyter_text>Now, we chose two different metropolitan areas to compare the degree of segregation.  Map of the composition of the Metropolitan area of Los Angeles<jupyter_code>la_2010 = merged_gdf.loc[(merged_gdf.name == "Los Angeles-Long Beach-Anaheim, CA")]
la_2010.plot(column = 'compo', figsize = (10, 10), cmap = 'OrRd', legend = True)
plt.axis('off')<jupyter_output><empty_output><jupyter_text>Map of the composition of the Metropolitan area of New York<jupyter_code>ny_2010 = merged_gdf.loc[(merged_gdf.name == 'New York-Newark-Jersey City, NY-NJ-PA')]
ny_2010.plot(column = 'compo', figsize = (20, 10), cmap = 'OrRd', legend = True)
plt.axis('off')<jupyter_output><empty_output><jupyter_text>We first compare the Gini index of both cities. Let's import the `Gini_Seg` class from `segregation`, fit both indexes and check the difference in point estimation.<jupyter_code>from segregation.aspatial import GiniSeg

G_la = GiniSeg(la_2010, 'nhblk10', 'pop10')
G_ny = GiniSeg(ny_2010, 'nhblk10', 'pop10')

G_la.statistic - G_ny.statistic<jupyter_output><empty_output><jupyter_text>Let's decompose these difference according to *Rey, S. et al "Comparative Spatial Segregation Analytics". Forthcoming*. You can check the options available in this decomposition below:<jupyter_code>help(DecomposeSegregation)<jupyter_output>Help on class DecomposeSegregation in module segregation.decomposition.decompose_segregation:

class DecomposeSegregation(builtins.object)
 |  Decompose segregation differences into spatial and attribute components.
 |  
 |  Given two segregation indices of the same type, use Shapley decomposition
 |  to measure whether the differences between index measures arise from
 |  differences in spatial structure or population structure
 |  
 |  Parameters
 |  ----------
 |  index1 : segregation.SegIndex class
 |      First SegIndex class to compare.
 |  index2 : segregation.SegIndex class
 |      Second SegIndex class to compare.
 |  counterfactual_approach : str, one of
 |                            ["composition", "share", "dual_composition"]
 |      The technique used to generate the counterfactual population
 |      distributions.
 |  
 |  Attributes
 |  ----------
 |  
 |  c_s : float
 |      Shapley's Spatial Component of the decomposition
 |              
 |  c_a : float
 |      Shaple[...]<jupyter_text>Composition Approach (default) The difference of -0.10653 fitted previously, can be decomposed into two components. The Spatial component and the attribute component. Let's estimate both, respectively.<jupyter_code>DS_composition = DecomposeSegregation(G_la, G_ny)
DS_composition.c_s
DS_composition.c_a<jupyter_output><empty_output><jupyter_text>So, the first thing to notice is that attribute component, i.e., given by a difference in the population structure (in this case, the composition) plays a more important role in the difference, since it has a higher absolute value.The difference in the composition can be inspected in the plotting method with the type `cdfs`:<jupyter_code>DS_composition.plot(plot_type = 'cdfs')<jupyter_output><empty_output><jupyter_text>If your data is a GeoDataFrame, it is also possible to visualize the counterfactual compositions with the argument `plot_type = 'maps'`The first and second contexts are Los Angeles and New York, respectively.<jupyter_code>DS_composition.plot(plot_type = 'maps')<jupyter_output><empty_output><jupyter_text>*Note that in all plotting methods, the title presents each component of the decomposition performed.*  Share Approach The share approach takes into consideration the share of each group in each city. Since this approach takes into consideration the focus group and the complementary group share to build the "counterfactual" total population of each unit, it is of interest to inspect all these four cdf's.*ps.: The share is the population frequency of each group in each unit over the total population of that respectively group.*<jupyter_code>DS_share = DecomposeSegregation(G_la, G_ny, counterfactual_approach = 'share')
DS_share.plot(plot_type = 'cdfs')<jupyter_output><empty_output><jupyter_text>We can see that curve between the contexts are closer to each other which represent a drop in the importance of the population structure (attribute component) to -0.062. However, this attribute still overcomes the spatial component (-0.045) in terms of importance due to both absolute magnitudes.<jupyter_code>DS_share.plot(plot_type = 'maps')<jupyter_output><empty_output><jupyter_text>We can see that the counterfactual maps of the composition (outside of the main diagonal), in this case, are slightly different from the previous approach.  Dual Composition Approach The `dual_composition` approach is similar to the composition approach. However, it uses also the counterfactual composition of the cdf of the complementary group.<jupyter_code>DS_dual = DecomposeSegregation(G_la, G_ny, counterfactual_approach = 'dual_composition')
DS_dual.plot(plot_type = 'cdfs')<jupyter_output><empty_output><jupyter_text>It is possible to see that the component values are very similar with slight changes from the `composition` approach.<jupyter_code>DS_dual.plot(plot_type = 'maps')<jupyter_output><empty_output><jupyter_text>The counterfactual distributions are virtually the same (but not equal) as the one from the `composition` approach.  Inspecting a different index: Relative Concentration<jupyter_code>from segregation.spatial import RelativeConcentration

RCO_la = RelativeConcentration(la_2010, 'nhblk10', 'pop10')
RCO_ny = RelativeConcentration(ny_2010, 'nhblk10', 'pop10')

RCO_la.statistic - RCO_ny.statistic
RCO_DS_composition = DecomposeSegregation(RCO_la, RCO_ny)
RCO_DS_composition.c_s
RCO_DS_composition.c_a<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08766443821436273, 'fraction_numerical': 0.014448997196463231, 'mean_word_length': 6.042520880789674, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '9244', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2968, 'n_tokens_neox': 2755, 'n_words': 900, 'path': 'notebooks/07_decomposition_example.ipynb', 'repo_name': 'noahbouchier/segregation', 'size': '9274'}