<jupyter_start><jupyter_text>Remove rivers from waterbody polygons * **Compatibility:** Notebook currently compatible with the `NCI` environment only. You can make this notebook `Sandbox` compatible by pointing it to the DEA Waterbodies timeseries located in AWS.* **Products used:** None.* **Special requirements**    * River line dataset for filtering out polygons comprised of river segments.        * Variable name: `MajorRiversDataset`        * Here we use the [Bureau of Meteorology's Geofabric v 3.0.5 Beta (Suface Hydrology Network)](ftp://ftp.bom.gov.au/anon/home/geofabric/), filtered to only keep features tagged as `major rivers`.         * There are some identified issues with this data layer that make the filtering using this data inconsistent (see the discussion below)        * We therefore turn this off during the production of the water bodies shapefile. * **Prerequisites:** This notebook explores the individual waterbody timeseries csvs contained within the DEA Waterbodies dataset. It has been designed with that very specific purpose in mind, and is not intended as a general analysis notebook.   DescriptionThis notebook applies the `FilterRivers` filter from the [`TurnWaterObservationsIntoWaterbodyPolygons.ipynb`](../TurnWaterObservationsIntoWaterbodyPolygons.ipynb) notebook. This allows this filtering step to be applied to the final Waterbody polygon dataset, in order to produce a further refined version in which the polygon UIDs still match up with the all encompassing version.1. Load in modules and set up some functions2. Load in the river lines dataset3. Load in the waterbodies dataset4. Find where the two intersect and remove waterbodies that intersect with river lines5. Write out the results to a new shapefile***  Getting startedTo run this analysis, run all the cells in the notebook, starting with the "Load packages" cell. Note that file paths have been hardcoded below. To run this notebook, make sure that these are still correct.  Load packages and functionsImport Python packages that are used for the analysis.<jupyter_code>import geopandas as gp
def Filter_shapefile_by_intersection(gpdData,
                                     gpdFilter,
                                     filtertype='intersects',
                                     invertMask=True,
                                     returnInverse=False):
    '''
    Filter out polygons that intersect with another polygon shapefile. 
    
    Parameters
    ----------
    
    gpdData: geopandas dataframe
        Polygon data that you wish to filter
    gpdFilter: geopandas dataframe
        Dataset you are using as a filter
    
    Optional
    --------
    filtertype: default = 'intersects'
        Options = ['intersects', 'contains', 'within']
    invertMask: boolean
        Default = 'True'. This determines whether you want areas that DO ( = 'False') or DON'T ( = 'True')
        intersect with the filter shapefile.
    returnInnverse: boolean
        Default = 'False'. If true, then return both parts of the intersection - those that intersect AND 
        those that don't as two dataframes.
    
    Returns
    -------
    gpdDataFiltered: geopandas dataframe
        Filtered polygon set, with polygons that intersect with gpdFilter removed.
    IntersectIndex: list of indices of gpdData that intersect with gpdFilter
    
    Optional
    --------
    if 'returnInverse = True'
    gpdDataFiltered, gpdDataInverse: two geopandas dataframes
        Filtered polygon set, with polygons that DON'T intersect with gpdFilter removed.
    '''

    # Check that the coordinate reference systems of both dataframes are the same

    #assert gpdData.crs == gpdFilter.crs, 'Make sure the the coordinate reference systems of the two provided dataframes are the same'

    Intersections = gp.sjoin(gpdFilter, gpdData, how="inner", op=filtertype)

    # Find the index of all the polygons that intersect with the filter
    IntersectIndex = sorted(set(Intersections['index_right']))

    # Grab only the polygons NOT in the IntersectIndex
    # i.e. that don't intersect with a river
    if invertMask:
        gpdDataFiltered = gpdData.loc[~gpdData.index.isin(IntersectIndex)]
    else:
        gpdDataFiltered = gpdData.loc[gpdData.index.isin(IntersectIndex)]

    if returnInverse:
        # We need to use the indices from IntersectIndex to find the inverse dataset, so we
        # will just swap the '~'.

        if invertMask:
            gpdDataInverse = gpdData.loc[gpdData.index.isin(IntersectIndex)]
        else:
            gpdDataInverse = gpdData.loc[~gpdData.index.isin(IntersectIndex)]

        return gpdDataFiltered, IntersectIndex, gpdDataInverse
    else:

        return gpdDataFiltered, IntersectIndex<jupyter_output><empty_output><jupyter_text>Load in the datasetsWe use the [Bureau of Meteorology's Geofabric v 3.0.5 Beta (Suface Hydrology Network)](ftp://ftp.bom.gov.au/anon/home/geofabric/) to filter out polygons that intersect with major rivers. This is done to remove river segments from the polygon dataset. We use the `SH_Network AHGFNetworkStream any` layer within the `SH_Network_GDB_V2_1_1.zip` geodatabase, and filter the dataset to only keep rivers tagged as `major`. It is this filtered dataset we use here.Note that we reproject this dataset to `epsg 3577`, Australian Albers coordinate reference system. If this is not correct for your analysis, you can change this in the cell below. Note when using the Geofabric to filter out riversThe option to filter out rivers was switched off for the production of our water bodies dataset. During testing, the Geofabric dataset was shown to lead to inconsistencies in what was removed, and what remained within the dataset. * The Geofabric continues the streamline through on-river dams, which means these polygons are filtered out. This may not be the desired result. ![Stream and Dam intersection](../DocumentationFigures/OnRiverDam.JPG "An in-river dam that would be removed by the river filter, but may not be the desired result")  Read in the river lines dataset<jupyter_code># Where is this file located?
MajorRiversDataset = '/g/data/r78/cek156/ShapeFiles/SH_Network_GDB_National_V3_0_5_Beta/SH_Network_GDB_National_V3_0_5_Beta_MajorFiltered.shp'

# Read in the major rivers dataset (if you are using it)
MajorRivers = gp.GeoDataFrame.from_file(MajorRiversDataset) 
MajorRivers = MajorRivers.to_crs({'init':'epsg:3577'})<jupyter_output><empty_output><jupyter_text>Read in the waterbodies dataset<jupyter_code>WaterPolygons = gp.read_file('/g/data/r78/cek156/dea-notebooks/Scientific_workflows/DEAWaterbodies/AusAllTime01-005HybridWaterbodies/AusWaterBodiesFINAL.shp')
WaterPolygons = WaterPolygons.to_crs({'init':'epsg:3577'})<jupyter_output><empty_output><jupyter_text>Filter out polygons that intersect with a major river<jupyter_code>WaterBodiesBigRiverFiltered, Index = Filter_shapefile_by_intersection(WaterPolygons, 
                                                                      MajorRivers)<jupyter_output><empty_output><jupyter_text>Write out the amended shapefile<jupyter_code>WaterBodiesBigRiverFiltered.crs = {'init': 'epsg:3577'}
WaterBodiesBigRiverFiltered.to_file(
    '/g/data/r78/cek156/dea-notebooks/Scientific_workflows/DEAWaterbodies/AusAllTime01-005HybridWaterbodies/AusWaterBodiesFINALRiverFiltered.shp',
    driver='ESRI Shapefile')<jupyter_output><empty_output><jupyter_text>*** Additional information**License:** The code in this notebook is licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). Digital Earth Australia data is licensed under the [Creative Commons by Attribution 4.0](https://creativecommons.org/licenses/by/4.0/) license.**Contact:** If you need assistance, please post a question on the [Open Data Cube Slack channel](http://slack.opendatacube.org/) or on the [GIS Stack Exchange](https://gis.stackexchange.com/questions/ask?tags=open-data-cube) using the `open-data-cube` tag (you can view previously asked questions [here](https://gis.stackexchange.com/questions/tagged/open-data-cube)).If you would like to report an issue with this notebook, you can file one on [Github](https://github.com/GeoscienceAustralia/dea-notebooks).**Last modified:** January 2020**Compatible datacube version:** N/A  TagsBrowse all available tags on the DEA User Guide's [Tags Index](https://docs.dea.ga.gov.au/genindex.html)<jupyter_code>**Tags**: :index:`DEA Waterbodies`, :index:`no_testing`<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07577537593984962, 'fraction_numerical': 0.008576127819548873, 'mean_word_length': 3.9725467289719627, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '244213', 'license': 'Apache-2.0', 'n_tokens_mistral': 2517, 'n_tokens_neox': 2334, 'n_words': 856, 'path': 'Scientific_workflows/DEAWaterbodies/DEAWaterbodiesToolkit/RemoveRiversfromWaterBodyPolygons.ipynb', 'repo_name': 'RichardScottOZ/dea-notebooks', 'size': '8512'}