<jupyter_start><jupyter_text>Working with filter pipelines This Jupyter notebook explains the workflow of setting up and configuring a ground point filtering pipeline. This is an advanced workflow for users that want to define their own filtering workflows. For basic use, preconfigured pipelines are (or rather: will be) provided by `adaptivefiltering`. As always, we first need to import our library:<jupyter_code>import adaptivefiltering<jupyter_output><empty_output><jupyter_text>Also, we need to load at least one data set which we will use to interactively preview our filter settings. Note that for a good interactive experience with no downtimes, you should restrict your datasets to a reasonable size (see the [Working with datasets](datasets.ipynb) notebook for how to do it).<jupyter_code>dataset = adaptivefiltering.DataSet(filename="500k_NZ20_Westport.laz")<jupyter_output><empty_output><jupyter_text>Filtering backends `adaptivefiltering` does not implement its own ground point filtering algorithms. Instead, algorithms from existing packages are accessible through a common interface. Currently, the following backends are available:* [PDAL](https://pdal.io/): The Point Data Abstraction Library is an open source library for point cloud processing.* [OPALS](https://opals.geo.tuwien.ac.at/html/stable/index.html) is a proprietary library for processing Lidar data. It can be tested freely for datasets <1M points.* [LASTools](https://rapidlasso.com/) has a proprietary tool called `lasground_new` that can be used for ground point filtering.PDAL is always available when using `adaptivefiltering` and is used internally for many tasks that are not directly related to ground point filtering. In order to enable the OPALS backend, `adaptivefiltering` needs to be given the information where your OPALS installation (potentially including your license key) is located. This can either be done by setting the environment variable `OPALS_DIR` or by setting the path at runtime:<jupyter_code>adaptivefiltering.set_opals_directory("/path/to/opals")<jupyter_output><empty_output><jupyter_text>Similarly, you can set the path to your installation of LASTools either through the environment variable `LASTOOLS_DIR` or at runtime:<jupyter_code>adaptivefiltering.set_lastools_directory("/path/to/lastools")<jupyter_output><empty_output><jupyter_text>Please note that LASTools only ships Windows binaries. Therefore, you will need [Wine](https://www.winehq.org/) installed on your system to successfully use the LASTools backend.  Configuring a filter pipeline The main pipeline configuration is done by calling the `pipeline_tuning` function with your dataset as the parameter. This will open the interactive user interface which waits for your user input until you hit the *Finalize* button. The configured filter is then accessible as the Python object `pipeline`:<jupyter_code>pipeline = adaptivefiltering.pipeline_tuning(dataset)<jupyter_output><empty_output><jupyter_text>If you want to inspect multiple data sets in parallel while tuning a pipeline, you can do so by passing a list of datasets to the `pipeline_tuning` function. Note that `adaptivefiltering` does currently not parallelize the execution of filter pipeline execution which may have a negative impact on wait times while tuning with multiple parameters.<jupyter_code>pipeline2 = adaptivefiltering.pipeline_tuning(datasets=[dataset, dataset])<jupyter_output><empty_output><jupyter_text>Storing and reloading filter pipelines Pipeline objects can be stored on disk with the `save_filter` command from `adaptivefiltering`. We typically use the extension `json` for filters. It stands for *JavaScript Object Notation* and is a widely used format for storing custom data structures:<jupyter_code>adaptivefiltering.save_filter(pipeline, "myfilter.json")<jupyter_output><empty_output><jupyter_text>The appropriate counterpart is `load_filter`, which restores the pipeline object from a file:<jupyter_code>old_pipeline = adaptivefiltering.load_filter("myfilter.json")<jupyter_output><empty_output><jupyter_text>A filter pipeline loaded from a file can be edited using the `pipeline_tuning` command by passing it to the function. As always, the pipeline object returned by `pipeline_tuning` will be a new object - no implicit changes of the loaded pipeline object will occur:<jupyter_code>edited_pipeline = adaptivefiltering.pipeline_tuning(dataset, pipeline=old_pipeline)<jupyter_output><empty_output><jupyter_text>Applying filter pipelines to data Pipeline objects can also be used to transform data sets by applying the ground point filtering algorithms. This is one of the core tasks of the `adaptivefiltering` library, but this will rarely be done in this manual fashion, as we will provide additional interfaces for (locally adaptive) application of filter pipelines:<jupyter_code>filtered = pipeline.execute(dataset)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05913432229221703, 'fraction_numerical': 0.001422475106685633, 'mean_word_length': 6.837579617834395, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 41, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '13293', 'license': 'MIT', 'n_tokens_mistral': 1299, 'n_tokens_neox': 1253, 'n_words': 554, 'path': 'jupyter/filtering.ipynb', 'repo_name': 'readthedocs-assistant/adaptivefiltering', 'size': '4921'}