<jupyter_start><jupyter_text>Pre-process input data for coastal variable extractionAuthor: <NAME>; <EMAIL>*** Basic setupWe recommend that you install this package in the ArcGIS Pro conda environment through pip: ```bat\ArcGIS\Pro\bin\Python\Scripts\proenvpip install git+https://github.com/esturdivant-usgs/BI-geomorph-extraction.git```You must run this notebook within the ArcGIS Pro conda environment. To do so, type the following in your command prompt (assuming it has the default set-up and substituting path\to\dir with the location of the repository):```batcd path\to\dir\BI-geomorph-extraction\ArcGIS\Pro\bin\Python\Scripts\proenvjupyter notebook```This process requires some manipulation of the spatial layers by the user. Those times are described in the notebook. Sometimes the user needs to run the code through the ArcPy console (when it involves selecting layers). Pre-processing steps1. Pre-created geomorphic features: dunes, shoreline points, armoring.2. Inlets3. Shoreline4. Transects - extend and sort5. Transects - tidy  Import modules<jupyter_code>import os
import sys
import arcpy
import pandas as pd
try:
    import core.functions_warcpy as fwa
except ImportError as err:
    print(err)
    print("Looks like you need to install the CoastalVarExtractor module to your ArcGIS environment.")
    print("To do so: pip install git+https://github.com/esturdivant-usgs/BI-geomorph-extraction.git")
    
from core.setvars import *<jupyter_output>site: FireIsland
year: 2010
setvars.py initialized variables.<jupyter_text>If you don't want to formally install the module, you'll need to add the path to the package to your system path: ```pythonmod_path = r"C:\Users\esturdivant\Code\BI-geomorph-extraction"  replace with path to modulesys.path.insert(0, mod_path)import CoastalVarExtractor.functions_warcpy as fwa```  Initialize variablesBased on the project directory, and the site and year you have input, setvars.py will set a bunch of variables as the names of folders, files, and fields. 1) set-up the project folder and paths:<jupyter_code># from CoastalVarExtractor.configmap import *
# proj_dir = r"\\Mac\stor\Projects\TransectExtraction\FireIsland2012"
# site = "Fisherman"
# year = "2014"

orig_trans = os.path.join(arcpy.env.workspace, 'DelmarvaS_SVA_LT')

SA_bounds = 'SA_bounds'

extendedTrans = os.path.join(home, 'fiis_trans')
inletLines = os.path.join(home, 'inletLines')
ShorelinePts = os.path.join(home, 'SLpts')
dlPts = os.path.join(home, 'DLpts')
dhPts = os.path.join(home, 'DHpts')
armorLines = os.path.join(home, 'armorLines')
elevGrid = os.path.join(home, 'DEM')
elevGrid_5m = os.path.join(home, 'DEM_5m')

barrierBoundary = os.path.join(home, 'bndpoly_2sl_edited')  
shoreline = os.path.join(home, 'ShoreBetweenInlets')

SubType = os.path.join(home, 'FI11_SubType')
VegType = os.path.join(home, 'FI11_VegType')
VegDens = os.path.join(home, 'FI11_VegDens')
GeoSet = os.path.join(home, 'FI11_GeoSet')
DisMOSH = os.path.join(home, 'FI11_DisMOSH')<jupyter_output><empty_output><jupyter_text>Dunes and armoring Display the points and the DEM in a GIS to check for irregularities. For example, if shoreline points representing a distance less than X m are visually offset from the general shoreline, they should likely be removed. Another red flag is when the positions of dlows and dhighs in relation to the shore are illogical, i.e. dune crests are seaward of dune toes.  Replace fill values with Null.<jupyter_code>fwa.ReplaceValueInFC(dhPts, oldvalue=fill, newvalue=None, fields=["dhigh_z"])      # Dhighs
fwa.ReplaceValueInFC(dlPts, oldvalue=fill, newvalue=None, fields=["dlow_z"])       # Dlows
fwa.ReplaceValueInFC(ShorelinePts, oldvalue=fill, newvalue=None, fields=["slope"]) # Shoreline<jupyter_output><empty_output><jupyter_text>Project to UTM if not already. If this happens, we need to change the file name for future processing.<jupyter_code>dhPts = fwa.ReProject(dhPts, dhPts+'_utm', proj_code=proj_code)      # Dhighs
dlPts = fwa.ReProject(dlPts, dlPts+'_utm', proj_code=proj_code)       # Dlows
ShorelinePts = fwa.ReProject(ShorelinePts, ShorelinePts+'_utm', proj_code=proj_code) # Shoreline<jupyter_output>The projection of \\Mac\stor\Projects\TransectExtraction\FireIsland2010\FireIsland2010.gdb\DHpts was changed. The new file is \\Mac\stor\Projects\TransectExtraction\FireIsland2010\FireIsland2010.gdb\DHpts_utm.
The projection of \\Mac\stor\Projects\TransectExtraction\FireIsland2010\FireIsland2010.gdb\DLpts was changed. The new file is \\Mac\stor\Projects\TransectExtraction\FireIsland2010\FireIsland2010.gdb\DLpts_utm.
The projection of \\Mac\stor\Projects\TransectExtraction\FireIsland2010\FireIsland2010.gdb\SLpts was changed. The new file is \\Mac\stor\Projects\TransectExtraction\FireIsland2010\FireIsland2010.gdb\SLpts_utm.<jupyter_text>ArmoringIf the dlows do not capture the entire top-of-beach due to atypical formations caused by anthropogenic modification, you may need to digitize the beachfront armoring. The next code block will generate an empty feature class. Refer to the DEM and orthoimagery. If there is no armoring in the study area, continue. If there is armoring, use the Editing toolbar to add lines to the feature class that trace instances of armoring. Common manifestations of what we call armoring are sandfencing and sandbagging and concrete seawalls. If there is no armoring file in the project geodatabase, the extractor script will notify you that it is proceeding without armoring.*__Requires manipulation in GIS__*<jupyter_code>arcpy.CreateFeatureclass_management(home, armorLines, 'POLYLINE', spatial_reference=utmSR)
print("{} created. Now manually digitize the shorefront armoring.".format(armorLines))<jupyter_output><empty_output><jupyter_text>InletsWe also need to manually digitize inlets if an inlet delineation does not already exist. To do, the code below will produce the feature class. After which, use the Editing toolbar to create a line where the oceanside shore meets a tidal inlet. If the study area includes both sides of an inlet, that inlet will be represented by two lines. The inlet lines are use to define the bounds of the oceanside shore, which is also considered the point where the oceanside shore meets the bayside. Inlet lines must intersect the MHW contour. What do we do when the study area and not an inlet is the end?*__Requires manipulation in GIS__*<jupyter_code># manually create lines that correspond to end of land and cross the MHW line (use bndpoly/DEM)
arcpy.CreateFeatureclass_management(home, inletLines, 'POLYLINE', spatial_reference=utmSR)
print("{} created. Now we'll stop for you to manually create lines at each inlet.".format(inletLines))<jupyter_output><empty_output><jupyter_text>ShorelineThe shoreline is produced through a combination of the DEM and the shoreline points. The first step converts the DEM to both MTL and MHW contour polygons. Those polygons are combined to produce the full shoreline, which is considered to fall at MHW on the oceanside and MTL on the bayside (to include partially submerged wetland).If the study area does not end cleanly at an inlet, create a separate polyline feature class (default name is 'SA_bounds') and add lines that bisect the shoreline; they should look and function like inlet lines. Specify this in the arguments for DEMtoFullShorelinePoly() and CreateShoreBetweenInlets().At some small inlets, channel depth may be above MTL. In this case, the script left to its own devices will leave the MTL contour between the two inlet lines. This can be rectified after processing by deleting the mid-inlet features from the temp file 'shore_2split.'<jupyter_code>SA_bounds = 'SA_bounds'

bndpoly = fwa.DEMtoFullShorelinePoly(elevGrid_5m, MTL, MHW, inletLines, ShorelinePts)
print('Select features from {} that should not be included in the final shoreline polygon. '.format(bndpoly))<jupyter_output>Creating the MTL contour polgon from the DEM...
Creating the MHW contour polgon from the DEM...
Combining the two polygons...
Isolating the above-MTL portion of the polygon to the bayside...

User input required! Select extra features in bndpoly for deletion.

        Recommended technique: select the polygon/s to keep and then Switch Selection.<jupyter_text>*__Requires display in GIS__*User input is required to identify only the areas within the study area and eliminate isolated landmasses that are not. Once the features to delete are selected, either delete in the GIS or run the code below. Make sure the bndpoly variable matches the layer name in the GIS.__Do not...__ select the features in ArcGIS and then run DeleteFeatures in this Notebook Python kernel. That will delete the entire feature class. ```arcpy.DeleteFeatures_management(bndpoly)```The next step snaps the boundary polygon to the shoreline points anywhere they don't already match and as long as as they are within 25 m of each other.<jupyter_code>bndpoly = 'bndpoly'
barrierBoundary = fwa.NewBNDpoly(bndpoly, ShorelinePts, barrierBoundary, '25 METERS', '50 METERS')
shoreline = fwa.CreateShoreBetweenInlets(barrierBoundary, inletLines, shoreline, 
                                         ShorelinePts, proj_code)<jupyter_output>Splitting \\Mac\stor\Projects\TransectExtraction\FireIsland2010\FireIsland2010.gdb\bndpoly_2sl_edited at inlets...
Preserving only those line segments that intersect shoreline points...
Dissolving the line to create \\Mac\stor\Projects\TransectExtraction\FireIsland2010\FireIsland2010.gdb\ShoreBetweenInlets...<jupyter_text>After this step, you'll want to make sure the shoreline looks okay. There should be only one line segment for each stretch of shore between two inlets. Segments may be incorrectly deleted if the shoreline points are missing in the area. Segments may be incorrectly preserved if they are intersect a shoreline point. To rectify, either perform manual editing or rerun this code with modifications.   Transects - extend, sort, and tidyCreate extendedTrans, NASC transects for the study area extended to cover the island, with gaps filled, and sorted in the field sort_ID. 1. Extend the transects and use a copy of the lines to fill alongshore gaps<jupyter_code># Delete transects over 200 m outside of the study area.
if input("Need to remove extra transects? 'y' if barrierBoundary should be used to select. ") == 'y':
    fwa.RemoveTransectsOutsideBounds(orig_trans, barrierBoundary)
fwa.ExtendLine(orig_trans, trans_extended, extendlength, proj_code)
fwa.CopyAndWipeFC(trans_extended, trans_presort, ['sort_ID'])
print("MANUALLY: use groups of existing transects in new FC '{}' to fill gaps.".format(trans_presort))<jupyter_output>Need to remove extra transects? 'y' if barrierBoundary exists and should be used to select. y
\\Mac\stor\Projects\TransectExtraction\Fisherman2014\Fisherman2014.gdb\DelmarvaS_SVA_LT is already projected in UTM.
MANUALLY: use groups of existing transects in new FC '\\Mac\stor\Projects\TransectExtraction\Fisherman2014\scratch.gdb\trans_presort_temp' to fill gaps.<jupyter_text>*__Requires manipulation in GIS__*1. Edit the trans_presort_temp feature class. __Move and rotate__ groups of transects to fill in gaps that are greater than 50 m alongshore. There is no need to preserve the original transects, but avoid overlapping the transects with each other and with the originals. Do not move any transects slightly. If they are moved, they will not be deleted in the next stage. If you slightly move any, you can either undo or delete that line entirely.<jupyter_code>fwa.RemoveDuplicates(trans_presort, trans_extended, barrierBoundary)<jupyter_output><empty_output><jupyter_text>2. Sort the transects along the shoreUsually if the shoreline curves, we need to identify different groups of transects for sorting. This is because the GIS will not correctly establish the alongshore order by simple ordering from the identified sort_corner. If this is the case, answer __yes__ to the next prompt.<jupyter_code>sort_lines = fwa.SortTransectPrep(spatialref=utmSR)<jupyter_output>Do we need to sort the transects in batches to preserve the order? (y/n) y
MANUALLY: Add features to sort_lines. Indicate the order of use in 'sort' and the sort corner in 'sort_corn'.<jupyter_text>*__Requires manipulation in GIS__*The last step generated an empty sort lines feature class if you indicated that transects need to be sorted in batches to preserve the order. Now, the user creates lines that will be used to spatially sort transects in groups. For each group of transects:1. __Create a new line__ in 'sort_lines' that intersects all transects in the group. The transects intersected by the line will be sorted independently before being appended to the preceding groups.  (*__add example figure__*)2. __Assign values__ for the fields 'sort,' 'sort_corner,' and 'reverse.' 'sort' indicates the order in which the line should be used and 'sort_corn' indicates the corner from which to perform the spatial sort ('LL', 'UL', etc.). 'reverse' indicates whether the order should be reversed (roughly equivalent to 'DESCENDING').3. Run the following code to create a new sorted transect file.<jupyter_code>fwa.SortTransectsFromSortLines(trans_presort, extendedTrans, sort_lines, tID_fld)<jupyter_output>Creating new feature class \\Mac\stor\Projects\TransectExtraction\Fisherman2014\Fisherman2014.gdb\extTrans to hold sorted transects...
Sorting sort lines by field sort...
For each line, creating subset of transects and adding them in order to the new FC...
Copying the generated OID values to the transect ID field (sort_ID)...<jupyter_text>3. Tidy the extended (and sorted) transects to remove overlap*__Requires manipulation in GIS__*Overlapping transects cause problems during conversion to 5-m points and to rasters. We create a separate feature class with the 'tidied' transects, in which the lines don't overlap. This is largely a manually process with the following steps: 1. __Select__ transects to be used to split other transects. Prioritize transects that a) were originally from NASC, b) have dune points within 25 m, and c) are oriented perpendicular to shore. (*__add example figure__*)2. Use the __Copy Features__ geoprocessing tool to copy only the selected transects into a new feature class. If desired, here is the code that could be used to copy the selected features and clear the selection:    ```python    arcpy.CopyFeatures_management(extendedTrans, overlapTrans_lines)    arcpy.SelectLayerByAttribute_management(extendedTrans, "CLEAR_SELECTION")    ```3. Run the code below to split the transects at the selected lines of overlap.<jupyter_code>overlapTrans_lines = os.path.join(arcpy.env.scratchGDB, overlapTrans_lines)
if not arcpy.Exists(overlapTrans_lines):
    overlapTrans_lines = input("Filename of the feature class of only 'boundary' transects: ")
arcpy.Intersect_analysis([extendedTrans, overlapTrans_lines], trans_x,
                         'ALL', output_type="POINT")
arcpy.SplitLineAtPoint_management(extendedTrans, trans_x, extTrans_tidy)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06916733173716413, 'fraction_numerical': 0.00951050811386007, 'mean_word_length': 5.431565440547477, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '213325', 'license': 'CC0-1.0', 'n_tokens_mistral': 4640, 'n_tokens_neox': 4297, 'n_words': 1751, 'path': 'temp/prepper_FireIsland2010.ipynb', 'repo_name': 'esturdivant-usgs/geomorph-working-files', 'size': '15059'}