<jupyter_start><jupyter_text>Tutorial of MST-DBSCAN in pysda<jupyter_code>import pysda
import os
import pandas as pd
import geopandas as gpd
import matplotlib.pyplot as plt
from matplotlib import patches as mpatches<jupyter_output><empty_output><jupyter_text>Use the build-in functions to read data (highly recommend).<jupyter_code>folder = r"D:\pySDA\test_data"<jupyter_output><empty_output><jupyter_text>Method 1: Load .csv<jupyter_code>filename = "DengueKS2014.csv"
path = os.path.join(folder, filename)
crs="+init=epsg:4326"

pysda_data = pysda.data.readCSV(path, xtitle="X", ytitle="Y", ttitle="OnsetDay", crs=crs, tunit="week")<jupyter_output><empty_output><jupyter_text>Method 2: Load DataFrame<jupyter_code>filename = "DengueKS2014.csv"
path = os.path.join(folder, filename)
crs="+init=epsg:4326"
df = pd.read_csv(path, encoding="utf-8")

pysda_data = pysda.data.readDF(df, xtitle="X", ytitle="Y", ttitle="OnsetDay", crs=crs, tunit="week")<jupyter_output><empty_output><jupyter_text>Method 3: Load .shp<jupyter_code>filename = "DengueKS2014.shp"
path = os.path.join(folder, filename)

pysda_data = pysda.data.readSHP(path, ttitle="OnsetDay", tunit="week")<jupyter_output><empty_output><jupyter_text>Method 4: Load geoDataFrame<jupyter_code>filename = "DengueKS2014.shp"
path = os.path.join(folder, filename)
gdf = gpd.read_file(path, encoding="utf-8")

pysda_data = pysda.data.readGDF(gdf, ttitle="OnsetDay", tunit="week")<jupyter_output><empty_output><jupyter_text>Arguments explanation:1. **xtitle** is the name of the column which records the x coordinate of each point.2. **ytitle** is the name of the column which records the y coordinate of each point.  Note: The values of x and y must be **projected** coordinates rather than longitude and latitude.3. **crs** is the Coordinate Reference System of the x and y.4. **ttitle** is the name of the column which records the time of each point either in integer format or in date format. If its values are in integer format, pysda will directly use them as the time stamps; otherwise, pysda will firstly transform them into integer format (through *tunit* argument).5. **tunit** is the temporal resolution for analysis, and the first time stamp is the first date in the input data. There are several choices:    - int: assign that the ttitle column in the input data is integer format.    - hour: 1 hour      - day: 1 days    - week: 7 days    - month: 30 days    - year: 365 days    - tunit also can be any string accepted by pandas' *date_range* function. For more details, please refer to https://pandas.pydata.org/pandas-docs/stable/timeseries.htmltimeseries-offset-aliases ****** Initialize an instance<jupyter_code>mst = pysda.MSTDBSCAN(pysda_data)<jupyter_output><empty_output><jupyter_text>Set parameters  Basic parameters:- eps_spatial: spatial search radius - eps_temporalLow: the minimum value of the temporal search window - eps_temporalHigh: the maximum value of the temporal search window- mis_pts: The minimum number of neughbors for a point to become a *core* and further form a cluster Advanced parameters:- movingRatio (default is 0.1):     This is the threshold value to check whether a cluster's center moves.    When the distance between centers in two sequential times over the eps_spatial is greater than the value, a cluster is considered as moving its center; otherwise, it stays.    - areaRatio (default is 0.1):    This is the threshold value to check whether a cluster's area changes.     When the **absolute** difference of area in two sequential times is greater than the areaRatio, a cluster is considered as changing its area; otherwise, it keeps a similar area.     Further details: For more details about the meanging of each parameter, please refer to the following documents:- journal article: https://www.tandfonline.com/doi/full/10.1080/24694452.2017.1407630<jupyter_code>eps_spatial = 300
eps_temporalLow = 1
eps_temporalHigh = 2
min_pts = 3
movingRatio = 0.1
areaRatio = 0.1

mst.setParams(eps_spatial, eps_temporalLow, eps_temporalHigh, min_pts, movingRatio, areaRatio)<jupyter_output><empty_output><jupyter_text>Start clustering<jupyter_code>mst.run()
result = mst.result<jupyter_output>core algorithm costs --- 1.9986937046051025 seconds ---<jupyter_text>****** Get the result of dynamic clustersExplaination about the columns:- **clusterID**: the ID of a cluster.      (Note: the value starts from 0, so 0 is the first cluster.)        - **mstTime**: a specific time (integer format) when a point is still alive.- **mstDate**: a specific time (date format) when a point is still alive.     - **type**: the evolution type of a cluster at a specific time.      (Note: There are 10 possible types. Please refer to the paper for a detailed description.)        - **centerX**, **centerY**: the coordinate of the center of a cluster at a specific time.    - **shape**: the polygon of a cluster at a specific time.<jupyter_code>clusterGDF = result.clusters
clusterGDF.head(10)<jupyter_output><empty_output><jupyter_text>Load a polygon .shp for plotting<jupyter_code>filename = "KSVillage.shp"
path = os.path.join(folder, filename)
polygonGDF = gpd.read_file(path, encoding="utf-8")<jupyter_output><empty_output><jupyter_text>Plot clusters<jupyter_code>part = clusterGDF[clusterGDF["mstDate"] == '2014/08/26-00:00:00']

fig, ax = plt.subplots(figsize=(8,16))
ax.set_aspect("equal")
ax.set_title("clusters in the week starting from 2014/08/26")
ax.set_xticks([])
ax.set_yticks([])

polygonGDF.plot(ax=ax, color="gray")
part.plot(ax=ax, categorical=True, column="clusterID")<jupyter_output><empty_output><jupyter_text>****** Get the result of dynamic pointsExplaination about the columns:- **pointID**: the original ID of a point.    - **mstTime**: a specific time (integer format) when a point is still alive.- **mstDate**: a specific time (date format) when a point is still alive.    - **clusterID**: the cluster to which a point belongs at a specific time.    (Note: -1 means the point does not belong to any cluster, so its role must be noise.)        - **role**: the role (core, border or noise) of a point at a specific time.<jupyter_code>pointGDF = result.points
pointGDF.head(10)<jupyter_output><empty_output><jupyter_text>****** Add polygon data to get the result of further analysis<jupyter_code>result.setPolygons(polygonGDF)<jupyter_output><empty_output><jupyter_text>Get the result of polygonsExplaination about the columns:  - **DZ**: diffusion zones. The polygons that are assigned to the same zone undergo a similar diffusion procedure.- **The dates**: each date is a mstDate. Each column records the situations (increas, decrease, keep, or no cluster) that every polygons undergo at that time.    - increase: a polygon is covered by a cluster whose evolution type is Emerge, Growth, Directional growth, or Merge.     - decrease: a polygon is covered by a cluster whose evolution type is Reduction, Directional reduction, or Split.    - keep: a polygon is covered by a cluster whose evolution type is Steady, Move, or SplitMerge.    - no cluster: a polygon is not covered by any cluster.<jupyter_code>polygonResultGDF = result.polygons
polygonResultGDF.head(10)<jupyter_output><empty_output><jupyter_text>ploat diffusion zones<jupyter_code>fig, ax = plt.subplots(figsize=(8,16))
ax.set_aspect("equal")
ax.set_title(column[0:10])
ax.set_xticks([])
ax.set_yticks([])<jupyter_output><empty_output><jupyter_text>plot polygons' dynamic<jupyter_code>column = '2014/08/26-00:00:00'

fig, ax = plt.subplots(figsize=(8,16))
ax.set_aspect("equal")
ax.set_title(column[0:10])
ax.set_xticks([])
ax.set_yticks([])

increase_patch = mpatches.Patch(color="red", label='Increase')
keep_patch = mpatches.Patch(color="blue", label='Keep')
decrease_patch = mpatches.Patch(color="green", label='Decrease')
non_patch = mpatches.Patch(color="gray", label='No cluster')

ax.legend(handles=[increase_patch, keep_patch, decrease_patch, non_patch])


groupedGDF = polygonResultGDF.groupby(column)
for name, group in groupedGDF:
    if name == "increase":
        group.plot(ax=ax, color="red")
    elif name == "keep":
        group.plot(ax=ax, color="blue")
    elif name == "decrease":
        group.plot(ax=ax, color="green")
    else:
        group.plot(ax=ax, color="gray")<jupyter_output><empty_output><jupyter_text>****** Get all results (clusters, points and polygons)<jupyter_code>allResults = result.getAll()
clusters = allResults["clusters"]
clusters.head(10)
points = allResults["points"]
points.head(10)
polygons = allResults["polygons"]
polygons.head(10)<jupyter_output><empty_output><jupyter_text>Directly save the results to the perfered direction<jupyter_code>result.saveAll(folder, prefix="mst_")<jupyter_output><empty_output><jupyter_text>Save the animation of the evolutnio of polygons (.gif)<jupyter_code>result.saveAnimation(figsize=(8,16), dirpath=folder, prefix="mst_")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11227124733355788, 'fraction_numerical': 0.019198383294038397, 'mean_word_length': 5.712886209495101, 'pattern_counts': {'":': 3, '<': 80, '<?xml version=': 0, '>': 80, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '386204', 'license': 'MIT', 'n_tokens_mistral': 3022, 'n_tokens_neox': 2847, 'n_words': 892, 'path': 'pysda-mstdbscan-tutorial.ipynb', 'repo_name': 'wenlab501/pysda', 'size': '8907'}