<jupyter_start><jupyter_text>1  Data Processing  1.1 Read Data and Crop a Patch<jupyter_code>path_cloud = 'data/Vaihingen3D_Traininig.pts'
# Load dataset as pandas dataframe
cloud      = load_cloud(path_cloud)
newcloud   = cloud_preprocess(cloud)
# Crop a patch from cloud
patch      = crop_patch(newcloud, (150, 300), (150, 300))<jupyter_output>-------------------- Load Data ---------------------
Data Loading...
=> Done.
----------------------------------------------------
--------------- Data Pre-Processing ----------------
Drop duplicates...
=> Duplicates #: 40946, Ratio: 5.43% of the dataset.
Re-center the data...
=> Done.
----------------------------------------------------
-------------------- Crop Patch --------------------
Crop the patch...
=> Crop a patch from: x:[150, 300], y:[150, 300].
=> Points left for patch: 157488.
----------------------------------------------------<jupyter_text>1.2 Data Visualization<jupyter_code>plot_graph_3D(patch, 
              figsize=(16, 8), 
              marksize=0.02, 
              markerscale=15, 
              xRange=(150, 300), 
              yRange=(150, 300))
plot_graph_2D(patch, 
              figsize=(6,6), 
              marksize=0.02,
              markerscale=15,
              blackBack=False, 
              xRange=(150, 300), 
              yRange=(150, 300))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.29472140762463345, 'fraction_numerical': 0.062316715542521994, 'mean_word_length': 1.3575129533678756, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '411502', 'license': 'MIT', 'n_tokens_mistral': 459, 'n_tokens_neox': 377, 'n_words': 89, 'path': '1_visualization.ipynb', 'repo_name': 'yinanzhangepfl/Filter-Learning-LiDAR', 'size': '1364'}