<jupyter_start><jupyter_text>Spatial LagThis python notebook introduces the concept of spatial lag for **spatial autocorrelation**, using the [`lag_spatial` function](https://pysal.org/libpysal/generated/libpysal.weights.lag_spatial.html) from PySAL. It uses an example previously presented by [Rey *et al.* (2020)](https://geographicdata.science/book/notebooks/06_spatial_autocorrelation.htmlspatial-lag) and data therein. The example presented here shows how spatial lag can be interpreted as the average (mean) value of a focal location's neighbours.  ConceptUsed with spatial weights matrices, the spatial lag operator is an important element of many spatial analysis techniques. For example it is used when calculating, interpreting and visualising local Moran's $I$ (a common quantitative measure of spatial autocorrelation, covered in detail in another notebook). The spatial lag operator can be expressed as: $$y_{sl-i} = \sum_j w_{ij} y_j$$where:- $y$ is the variable of interest, - $i$ and $j$ are locations in space, - and $w_{ij}$ is the value of a spatial weights matrix for the relationship between locations $i$ and $j$. Assuming that non-neighbours receive a weight of zero, $y_{sl-i}$ is the product (multiplication) of $y$ values and weights for $i$’s neighbours.  Thus, spatial lag operator essentially smoothes the values of a variable across space. For example, if a row-standardised spatial weights matrix is used, the effect of the operator is to produce spatially lagged local mean values of a variable. To understand this concept concretely with real world data, we'll look at the results of the UK Brexit vote (as presented in [Rey *et al.* 2020](https://geographicdata.science/book/notebooks/06_spatial_autocorrelation.htmlan-empirical-illustration-the-eu-referendum)).As [provided](https://geographicdata.science/book/data/brexit/brexit_cleaning.html) by Rey *et al.* (2020), there are two distinct datasets that will need to be loaded and combined:- Electoral Comission data on vote percentages at the local authority level. [CSV]- ONS Local Authority Districts (December 2016) Generalised Clipped Boundaries in the UK WGS84. [geojson]  SetupImport required libraries<jupyter_code># Graphics
import matplotlib.pyplot as plt

# Analysis
import geopandas as gpd
import pandas as pd
from pysal.lib import weights<jupyter_output><empty_output><jupyter_text>Load the vote percentage data as a pandas df:<jupyter_code>brexit_data_path = '../../data/brexit/brexit_vote.csv'
ref = pd.read_csv(brexit_data_path, index_col='Area_Code')
ref.head()<jupyter_output><empty_output><jupyter_text>Next load the local authority district boundaries as a geopandas geodf:<jupyter_code>lads = gpd.read_file("../../data/brexit/local_authority_districts.geojson").set_index('lad16cd')
lads.head()<jupyter_output><empty_output><jupyter_text>And then `join` the df to the geodf (the two df have the same index, using local autority district codes) and select only the columns we need for the analysis:<jupyter_code>gdf = gpd.GeoDataFrame(lads.join(ref[['Pct_Leave']]), crs=lads.crs)
gdf = gdf[['objectid', 'lad16nm', 'Pct_Leave', 'geometry']].dropna()
gdf.head()<jupyter_output><empty_output><jupyter_text>Let's quickly visualise the vote percentage data:<jupyter_code>fig, ax = plt.subplots(1, figsize=(9, 9))
gdf.plot(column='Pct_Leave', 
    cmap='viridis', scheme='quantiles', k=5, 
    linewidth=0., legend=True, legend_kwds={"title":'Leave Vote (%)',"loc": 2},
    ax=ax
)<jupyter_output><empty_output><jupyter_text>To more easily visualise the local smoothing (averaging) effect of the spatial lag operator, we'll use the following function to enable us to zoom into particular locations across the UK.<jupyter_code>def plotLeaveLocation(axes, idx, var, xlim_coords, ylim_coords):

    gdf.plot(column=var, 
        cmap='viridis', scheme='quantiles', k=5,
        linewidth=0., legend=True, legend_kwds={"title":var,"loc": 2},
        ax=axes
    )

    gdf.loc[[idx]].plot(ax=axes, color='none', edgecolor='black', linewidth=3)

    axes.set_xlim(xlim_coords)
    axes.set_ylim(ylim_coords) 
    axes.set_title(gdf.loc[idx]['lad16nm'])
    #plt.show()<jupyter_output><empty_output><jupyter_text>ApplicationIn this example we will compare the Local Authority Districts (LADs) of Liverpool and Midlothian (i.e. Edinburgh).<jupyter_code>fig, ax = plt.subplots(1, 2, figsize=(12, 9))

plotLeaveLocation(ax[0], 'E08000012', 'Pct_Leave', xlim_coords=(-4,-2), ylim_coords=(53,54))
plotLeaveLocation(ax[1], 'S12000019', 'Pct_Leave', xlim_coords=(-4,-2), ylim_coords=(55.5,56.5))

gdf.loc[['E08000012', 'S12000019'], 
       ['lad16nm','Pct_Leave']]<jupyter_output><empty_output><jupyter_text>From the maps and table above you should be able to see that these LADs had similar Vote Leave % in the Brexit vote. But the maps also show that the distribution of leave vote % of their neighbours differs noticeably; while Liverpool is surrounded by neighbours that have *higher* leave vote %, Midlothian is surrounded by by neighbours that have *lower* leave vote %. Thus, the spatial context of the LADs is different. It is this spatial context that the spatial lag operator measures, and so we should be able to see differences in the spatial lag for these two LADs. First we need to create a spatial weights matrix for our data. We'll use K Nearest Neighbours, with `k=8`:<jupyter_code># Generate W from the GeoDataFrame
w = weights.KNN.from_dataframe(gdf, k=8)
w.full()[0]<jupyter_output><empty_output><jupyter_text>Next we row-standarize the weights. This is a common transformation; for example, Moran's $I$ uses row-standarized weights. Row-standardization means *add all the weights in a row and divides by the number of neighbours*. This ensures the sum of all values of each row equals 1. Here we have 8 neighbours, so there will be 8 non-zero weights in each row. In turn, this means row-standarization will change all our all weights to 0.125 (1 divided by 8):<jupyter_code>w.transform = 'R'
w.full()[0]<jupyter_output><empty_output><jupyter_text>The effect of row-standardising the weights matrix is for the spatial lag operator to measure the spatially lagged mean values of each location (i.e. the mean of the focal location's neighbours' values). Think about why this is; the mean is calculated by adding all values and dividing by the number of values. In this case the values are those of the focal location's neighbours. Assuming the eight neighbours have values 1-8 (of some variable) we get:$$ \frac{(1 + 2 + 3 + 4 + 5 + 6 + 7 + 8)}{8} = \frac{36}{8} = 4.5$$Multiplying standardised weights by neighbours' observed values and then summing does essentially the same thing (and gets exactly the same result): $$((1\times0.125) + (2\times0.125) + (3\times0.125) + (4\times0.125) + (5\times0.125) + (6\times0.125) + (7\times0.125) + (8\times0.125))$$which is:$$0.125 + 0.25 + 0.375 + 0.5 + 0.625 + 0.75 + 0.875 + 1 = 4.5$$ With our row-standardized weights matrix, we now calculate the lagged values using the [lag_spatial](https://pysal.org/libpysal/generated/libpysal.weights.lag_spatial.html) function:<jupyter_code>gdf['Pct_Leave_lag'] = weights.spatial_lag.lag_spatial(w, gdf['Pct_Leave'])<jupyter_output><empty_output><jupyter_text>Let's compare the original vote leave % values for our LADs of interest with their spatially lagged local mean values:<jupyter_code>gdf.loc[['E08000012', 'S12000019'], 
       ['lad16nm','Pct_Leave', 'Pct_Leave_lag']]<jupyter_output><empty_output><jupyter_text>Remember how in the maps above we saw that Liverpool was surrounded by neighbours with generally *higher* leave vote %, while Midlothian was surrounded by by neighbours that have *lower* leave vote %. This is now shown in the spatially lagged value calculated for each LAD, which is much higher for Liverpool than for Midlothian. And we can now map these spatially lagged vote leave % for *all* LADs:<jupyter_code>fig, ax = plt.subplots(1, 2, figsize=(12, 9))

plotLeaveLocation(ax[0], 'E08000012', 'Pct_Leave_lag', xlim_coords=(-4,-2), ylim_coords=(53,54))
plotLeaveLocation(ax[1], 'S12000019', 'Pct_Leave_lag', xlim_coords=(-4,-2), ylim_coords=(55.5,56.5))<jupyter_output><empty_output><jupyter_text>Note how the map for Liverpool has changed much more than the map for Midlothian. The original map for Liverpool was much more spatially hetergenous (i.e. varied) then the original map for Midlothian. Hence the map of spatially lagged (local mean) values for Liverpool has become smoother compared to its original than the map for Midlothian, which was originally already quite homogenous (i.e. smooth). And we can see the effect on the entire UK:<jupyter_code>fig, axes = plt.subplots(1,2, figsize=(12, 9))
gdf.plot(column='Pct_Leave', 
    cmap='viridis', scheme='quantiles', k=5, 
    linewidth=0., legend=True, legend_kwds={"title":'Leave Vote (%)',"loc": 2},
    ax=axes[0]
)
axes[0].set_title("Original Vote Data")

gdf.plot(column='Pct_Leave_lag', 
    cmap='viridis', scheme='quantiles', k=5, 
    linewidth=0., legend=True, legend_kwds={"title":'Leave Vote Lag (%)',"loc": 2},
    ax=axes[1]
)
axes[1].set_title("Spatially Lagged Vote Data")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11001303214596003, 'fraction_numerical': 0.02812771503040834, 'mean_word_length': 5.601433691756273, 'pattern_counts': {'":': 8, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '21252', 'license': 'MIT', 'n_tokens_mistral': 3140, 'n_tokens_neox': 2877, 'n_words': 1013, 'path': 'code/python/SpatialLag.ipynb', 'repo_name': 'jamesdamillington/spar', 'size': '9208'}