<jupyter_start><jupyter_text>Breaking a variable to levels The scenario for this tutorial is that, you have a series of a variable, such as the population density of different cities. And, you need to classify them into different groups according to this variable, e.g. the very high, medium high, medium, medium low, very low population density, etc.In some cases, you already have a GeoDataFrame/DataFrame, in other cases, you just have a list that contain the numbers. So, the following cover two major functions:1. tm.leveling_vector, which take a dataframe and a column name for the classifying; and2. bk.get_levels, which take a list.The two functions takes a break_method for the breaking methods, such as quantile(default), head_tail_break, natural_break, equal_interval (and manual).They take a break_N parameter, for specifying the number of groups.And they also take a break_cuts.  First, import things that is needed.<jupyter_code>import geopandas as gpd # for reading and manupulating shapefile
import matplotlib.pyplot as plt # for making figure
import seaborn as sns # for making distplot

from colouringmap import theme_mapping as tm # a function named leveling_vector in tm will be used
from colouringmap import breaking_levels as bk # a function named get_levels in bk will be used

# magic line for matlotlib figure to be shown inline in jupyter cell
%matplotlib inline<jupyter_output><empty_output><jupyter_text>read a demo file, and take a look<jupyter_code>grid_res = gpd.read_file('data/community_results.shp')
grid_res.head()<jupyter_output><empty_output><jupyter_text>take a look at the data distribution. using seaborn distplot.<jupyter_code>sns.distplot(grid_res['usercount'], kde=False)<jupyter_output><empty_output><jupyter_text>the above plot showed that the data is potentially an exponential distribution.so lets try to make the yscale log.<jupyter_code>ax = sns.distplot(grid_res['usercount'], kde=False)
#ax.set_xscale("log", nonposx='clip')
ax.set_yscale("log", nonposy='clip')<jupyter_output><empty_output><jupyter_text>using different break method:1. quantile2. head_tail_break3. natural_break4. equal_interval the following is the most simple way of converting the column of a gdf to levels<jupyter_code>level_list, cuts = tm.leveling_vector(grid_res, 'usercount') #, break_method='quantile') #default method is quantile<jupyter_output><empty_output><jupyter_text>Normally, the level_list is used to be assign to the gdf. This is what I did in other functions of mapping.<jupyter_code>grid_res['user_level'] = level_list
grid_res.head()<jupyter_output><empty_output><jupyter_text>cuts contain the breaking values, and the min/max at the both end of the list.<jupyter_code>cuts
ax = sns.distplot(grid_res['usercount'], kde=False)
#ax.set_xscale("log", nonposx='clip')
ax.set_yscale("log", nonposy='clip')
for c in cuts:
    ax.axvline(x=c)
lev = list(set(level_list))
count = [ level_list.count(l) for l in lev ]
print lev
print count<jupyter_output>[0, 1, 2, 3, 4]
[568, 585, 531, 550, 554]<jupyter_text>quantile has a similar count for each level. Lets try some other break method.<jupyter_code>level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='head_tail_break')
print cuts
ax = sns.distplot(grid_res['usercount'], kde=False)
#ax.set_xscale("log", nonposx='clip')
ax.set_yscale("log", nonposy='clip')
for c in cuts:
    ax.axvline(x=c)
lev = list(set(level_list))
count = [ level_list.count(l) for l in lev ]
print lev
print count
level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='natural_break')
print cuts
ax = sns.distplot(grid_res['usercount'], kde=False)
#ax.set_xscale("log", nonposx='clip')
ax.set_yscale("log", nonposy='clip')
for c in cuts:
    ax.axvline(x=c)
lev = list(set(level_list))
count = [ level_list.count(l) for l in lev ]
print lev
print count
level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='equal_interval')
print cuts
ax = sns.distplot(grid_res['usercount'], kde=False)
#ax.set_xscale("log", nonposx='clip')
ax.set_yscale("log", nonposy='clip')
for c in cuts:
    ax.axvline(x=c)
lev = list(set(level_list))
count = [ level_list.count(l) for l in lev ]
print lev
print count<jupyter_output>[0, 1, 2, 3, 4]
[2713, 53, 12, 3, 7]<jupyter_text>specifying the number of level The number of level is set to the parameter break_N, which is default to 5.After setting the break_N to N, the number of cuts become N+1, because it contain both the largest and the smallest values.<jupyter_code>level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='head_tail_break', break_N=3)
print cuts
level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='head_tail_break', break_N=5)
print cuts
level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='head_tail_break', break_N=7)
print cuts
level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='head_tail_break', break_N=9)
print cuts<jupyter_output>[0.0, 111.01004304160689, 483.8207547169811, 1173.1554054054054, 2146.409090909091, 3247.6875, 3889.375, 4475.0, 4506.0, 4506.0]<jupyter_text>note that what head_tail_break do for increased number of levels.   specifying cuts manually There are two ways of using the cuts. This will return a cut list, and a level_list that is in the same length and same sequence with the input vector. 1. using quantile as method, and the cuts are some float numbers betweent 0-1. 2. using manual as method, and the cuts are some user defined cuts. NOTE that the cut list has to include the minimum and maximum values.<jupyter_code>level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='quantile', break_cuts=[0.,.25,.5,.75,1.])
print cuts
level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='quantile', break_cuts=[0.,0.1,.5,.99,1.])
print cuts
level_list, cuts = tm.leveling_vector(grid_res, 'usercount', break_method='manual', break_cuts=[0.0, 120, 490, 1200, 2200, 4506.0])
print cuts<jupyter_output>[0.0, 0.0, 120, 490, 1200, 2200, 4506.0]<jupyter_text>breaking a list instead of a column of a dataframe Let say you have a list, instead of a dataframe/geodataframe.<jupyter_code>a_list = grid_res['usercount'].tolist()<jupyter_output><empty_output><jupyter_text>And you want to get the break levels, another function is also provided (the function that is called by tm.leveling_vector).<jupyter_code>level_list, cuts = bk.get_levels(a_list, method='head_tail_break', N=5)
print cuts
len(level_list)==len(a_list)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10380517503805176, 'fraction_numerical': 0.031659056316590564, 'mean_word_length': 6.492588369441277, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '223292', 'license': 'MIT', 'n_tokens_mistral': 2353, 'n_tokens_neox': 2223, 'n_words': 701, 'path': 'example/breaking a variable to levels.ipynb', 'repo_name': 'wcchin/colouringmap', 'size': '6570'}