<jupyter_start><jupyter_text>Density-based Spatial Clustering of Applications with Noise (DBSCAN)This algorithm is similar to mean-shift (density-based). One different thing about DBSCAN is how it processes outliers: Meanshift includes outliers into a cluster, meanwhile DBSCAN identifies them as noisesPros: - no need to specify number of clusters- good for data with similar density- can handle arbitrary distance functions, such as haversine/geodetic distance functions -> more suitable for this GPS dataset Cons:- does not work well with varying density clusters (harder to estimate distance threshold and min points)<jupyter_code># df_small = df.iloc[-100000:]
coords = df[['latitude','longitude']].values
from sklearn.cluster import DBSCAN
kms_per_radian = 6371.0088
epsilon = 2 / kms_per_radian
db = DBSCAN(eps=epsilon, min_samples=100, algorithm='ball_tree', metric='haversine',n_jobs=-1)
cluster = db.fit_predict(np.radians(coords))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08135942327497425, 'fraction_numerical': 0.01956745623069001, 'mean_word_length': 6.476923076923077, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '308629', 'license': 'MIT', 'n_tokens_mistral': 297, 'n_tokens_neox': 272, 'n_words': 102, 'path': 'dbscan_clustering.ipynb', 'repo_name': 'anhquan0412/google-location', 'size': '971'}