<jupyter_start><jupyter_text>Title: Spatial Database of Planted Trees (SDPT) DescriptionThe Spatial Database of Planted Trees (SDPT) was compiled by Global Forest Watch using data obtained from national governments, non-governmental organizations and independent researchers. Data were compiled for 82 countries around the world, with most country maps originating from supervised classification or manual polygon delineation of Landsat, SPOT or RapidEye satellite imagery.The category of “planted trees” in the SDPT includes forest plantations of native or introduced species, established through deliberate human planting or seeding. Sometimes called “tree farms,” these forests infuse the global economy with a constant stream of lumber for construction, pulp for paper and fuelwood for energy. The data set also includes agricultural tree crops like oil palm plantations, avocado farms, apple orchards and even Christmas tree farms. The SDPT makes it possible to identify planted forests and tree crops as being separate from natural forests and enables changes in these planted areas to be monitored independently from changes in global natural forest cover.The SDPT contains 173 million hectares of planted forest and 50 million hectares of agricultural trees, or approximately 82% of the world’s total planted forest area in 2015 (FAO 2015). The SDPT was compiled through a procedure that included cleaning and processing each individual data set before creating a harmonized attribute table.Available as single country datasets, as well as a compiled global version (excluding countries with no data notably Canada, Russia and some African countries). FLINTThis dataset has been pre-processed/checked and is suitable for use in FLINT. Please adhere to individual dataset licence conditions and citations. Processed data can be accessed here: https://datasets.mojaglobal.workers.dev/ FormatExtent: Compiled country - global but excluding some countriesFormat: polygon geoJSON .jsonCordinate system: EPSG:4326 (WGS84)Temporal Resolution: multipleSize: 10GB+ (individual country file sizes smaller)   Original sourceOriginal Source: https://www.arcgis.com/home/item.html?id=224e00192f6d408fa5147bbfc13b62ddZip: http://gfw-files.s3.amazonaws.com/plantations/final/global/plantations_v1_3_dl.gdb.zip Accessed 13/12/2020Note this is a ESRI Geodatabase feature class (polygon) and is 8GB.  Multiple countries but not all, vector, shapefile by country. Cordinate system EPSG:4326 (WGS84) Licence[CC BY 4.0] (https://creativecommons.org/licenses/by/4.0/) CitationHarris, N., <NAME> and <NAME>. “Spatial Database of Planted Trees (SDPT) Version 1.0.” Accessed through Global Forest Watch on 20/12/2020. www.globalforestwatch.org. Metadatahttps://www.arcgis.com/home/item.html?id=224e00192f6d408fa5147bbfc13b62ddMetadata states there is data available for 82 countries, however only 43 from the downloaded source are available. NotesThis dataset is a compilation of planted trees data from a variety of countries and sources. As a result, there are definitional and temporal inconsistencies within the database, as well as an absence of a uniform accuracy assessment and incomplete spatial coverage, notably in Canada, Russia and countries in Africa.File sizes are large due to detail and resoltion of the vectors. Processing time is lengthy and final vector json sizes large. Significant self intersection, significant gaps but overlaps are generally ok. ProcessingRepair geometry, fix topologial error (remove overlaps and small gaps), convert to geojson, EPSG:4326 (WGS84), remove/disable Z values. The Sth Korea, USA, South Africa and India datasets are very large, detailed with many attributes and vertices, and are 1GB+ in json format. South Korea was so large (6GB) - it was split into three, simplified using the Douglas-Peucker algorithm with a 5m tolerance and stictched back together. It is not recommeded the large files are combined into a global dataset without simplification in GIS first (someone please let me know if you want this).<jupyter_code># Import arcpy and os
import arcpy
import os

# Input local variables
in_folder = r"C:\SpatialDatabasePlantedTrees\plantations_v1_3_dl.gdb"
scr_folder = r"C:\Data\scratch.gdb"
out_folder = r"C:\Data\json\plantedtrees"

# Environments
workspace = in_folder
arcpy.env.workspace = workspace
arcpy.env.outputCoordinateSystem = arcpy.SpatialReference(4326)
arcpy.env.outputZFlag = "Disabled"
arcpy.env.overwriteOutput = True
scr = arcpy.CreateFileGDB_management(r"C:\Data", "scratch")

# List features to process
featureclasses = arcpy.ListFeatureClasses()
print(featureclasses)

# Repair/check topology and make FLINT ready
for fc in featureclasses:
    fcname = os.path.join(os.path.splitext(fc)[0])
    outjson = os.path.join(out_folder, fcname)
    
    print(fcname + ' processing...')
    geomRepair = arcpy.management.RepairGeometry(fixedlyr, "DELETE_NULL", "OGC")[0]
    
    fLayer = "project_Layer"
    arcpy.management.MakeFeatureLayer(fc, fLayer)
    
    projectIntersect = os.path.join(scr_folder, "projectIntersect")
    arcpy.analysis.Intersect(fLayer, projectIntersect, "ONLY_FID")
    
    projectSingle = os.path.join(scr_folder, "projectSingle")
    arcpy.management.MultipartToSinglepart(projectIntersect, projectSingle)

    dissolveSlither = os.path.join(scr_folder, "dissolveSlither")
    arcpy.management.Dissolve(projectSingle, dissolveSlither, None, None,"SINGLE_PART")
    
    whereclause = "FID_" + fcname + "= -1 OR AREA_GEO <= 5000 Or AREA_GEO IS NULL"
    
    # Take action if overlaps
    if arcpy.management.GetCount(dissolveSlither)[0] == "0":
        print('no overlaps detected')
        
        projectUnion = os.path.join(scr_folder, "projectUnion")
        arcpy.analysis.Union(fLayer,projectUnion, "ALL", None, "NO_GAPS")
        arcpy.management.AddGeometryAttributes("projectUnion", "AREA_GEODESIC", None, "SQUARE_METERS")

        uniSelect = os.path.join(scr_folder, "uniSelect")
        arcpy.analysis.Select(projectUnion, uniSelect, whereclause)
        
        if arcpy.management.GetCount(uniSelect)[0] == "0":
                     
            # Progress report no error
            print(fcname, 'No gaps and overlaps. Repairing geometry and conversion to json...')
    
            # Process: Repair Geometry (non-simple geometry)
            geomRepair = arcpy.management.RepairGeometry(fLayer, "DELETE_NULL", "OGC")[0]

            # Process: Features To JSON
            arcpy.conversion.FeaturesToJSON(fLayer, outjson, "NOT_FORMATTED", "NO_Z_VALUES", "NO_M_VALUES", "GEOJSON", "WGS84", "USE_FIELD_NAME")

            print(outjson, '.geojson complete')
            
        else:
            # Take action if gaps
            print('gaps detected')

            appendGap = arcpy.management.Append(uniSelect, fLayer, "NO_TEST")     
            selectGap = arcpy.management.SelectLayerByAttribute(fLayer, "NEW_SELECTION", "final_id = ''")

            fixedlyr = os.path.join(scr_folder, "fixedlyr")
            arcpy.management.Eliminate(selectGap, fixedlyr, "LENGTH")

            # Progress report 
            print(fcname, 'No overlaps, gaps detected and repaired. Repairing geometry and conversion to json...')

            # Process: Repair Geometry (non-simple geometry)
            geomRepair = arcpy.management.RepairGeometry(fixedlyr, "DELETE_NULL", "OGC")[0]

            # Process: Features To JSON
            arcpy.conversion.FeaturesToJSON(fixedlyr, outjson, "NOT_FORMATTED", "NO_Z_VALUES", "NO_M_VALUES", "GEOJSON", "WGS84", "USE_FIELD_NAME")


    else:
        # Fix overlaps
        projectErase = os.path.join(scr_folder, "projectErase")
        arcpy.analysis.Erase(fLayer, dissolveSlither, projectErase)
        arcpy.management.Append(dissolveSlither, projectErase, "NO_TEST")
    
        selectSlither = arcpy.management.SelectLayerByAttribute(projectErase, "NEW_SELECTION", "final_id = ''")
        
        eliminateSlither = os.path.join(scr_folder, "eliminateSlither")
        arcpy.management.Eliminate(selectSlither, eliminateSlither, "LENGTH")
        
        print('overlaps detected and fixed')
        
        projectUnion = os.path.join(scr_folder, "projectUnion")
        arcpy.analysis.Union(eliminateSlither, projectUnion, "ALL", None, "NO_GAPS")
        arcpy.management.AddGeometryAttributes("projectUnion", "AREA_GEODESIC", None, "SQUARE_METERS")
        
        uniSelect = os.path.join(scr_folder, "uniSelect")
        
        arcpy.analysis.Select(projectUnion, uniSelect, "FID_eliminateSlither = -1 OR AREA_GEO <= 5000 Or AREA_GEO IS NULL")
        
        if arcpy.management.GetCount(uniSelect)[0] == "0":
            
            # Progress report no error
            print(fcname, 'Overlaps detected and repaired. No gaps detected. Repairing geometry and conversion to json...')
    
            # Process: Repair Geometry (non-simple geometry)
            geomRepair = arcpy.management.RepairGeometry(eliminateSlither, "DELETE_NULL", "OGC")[0]

            # Process: Features To JSON
            arcpy.conversion.FeaturesToJSON(eliminateSlither, outjson, "NOT_FORMATTED", "NO_Z_VALUES", "NO_M_VALUES", "GEOJSON", "WGS84", "USE_FIELD_NAME")

            print(outjson, '.geojson complete')
            
        else:
            # Take action if gaps
            appendGap = arcpy.management.Append(uniSelect, eliminateSlither, "NO_TEST")
            
            selectGap = arcpy.management.SelectLayerByAttribute(eliminateSlither, "NEW_SELECTION", "final_id = ''")
            
            fixedlyr = os.path.join(scr_folder, "fixedlyr")
            
            arcpy.management.Eliminate(selectGap, fixedlyr, "LENGTH")
            
            print('gaps detected and repaired')
            
            # Progress report 
            print(fcname, 'Gaps and overlaps repaired. Repairing geometry and conversion to json...')
            
            # Process: Repair Geometry (non-simple geometry)
            geomRepair = arcpy.management.RepairGeometry(fixedlyr, "DELETE_NULL", "OGC")[0]

            # Process: Features To JSON
            arcpy.conversion.FeaturesToJSON(fixedlyr, outjson, "NOT_FORMATTED", "NO_Z_VALUES", "NO_M_VALUES", "GEOJSON", "WGS84", "USE_FIELD_NAME")
            
arcpy.AddMessage("All done!")
print('done')
# To reduce the file size of the Korea planted trees data - split into three files (manually)then follow below for each
# Split will need to be done manually making sure all conecting polygons are in the same file at the split zone.

# Local variables
in_file = "insert file here"
out_simple = "interim simple file and location" #inspect this to see if algorithm satisfactory
outfinalkor_plant = "insert outfile and location here"
outjson = "insert output json file"

# Processes 3 split files (otherwise process will fail - too many parts)
arcpy.cartography.SimplifyPolygon(in_file, out_simple, "POINT_REMOVE", "5 Meters", "200 SquareMeters", "RESOLVE_ERRORS", "NO_KEEP", None)

# merge back together
arcpy.management.Merge("kor1Simple_plant;kor2Simple_plant;kor3Simple_plant", outfinalkor_plant)
geomRepair = arcpy.management.RepairGeometry(outfinalkor_plant, "DELETE_NULL", "OGC")[0]
arcpy.conversion.FeaturesToJSON(outfinalkor_plant, outjson, "NOT_FORMATTED", "NO_Z_VALUES", "NO_M_VALUES", "GEOJSON", "WGS84", "USE_FIELD_NAME")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08515686791457949, 'fraction_numerical': 0.013797345988223922, 'mean_word_length': 3.8487430762675756, 'pattern_counts': {'":': 3, '<': 9, '<?xml version=': 0, '>': 7, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '17208', 'license': 'BSD-Source-Code', 'n_tokens_mistral': 3391, 'n_tokens_neox': 3195, 'n_words': 1019, 'path': 'Data/LandCover/SpatialDatabasePlantedTrees_SDPT.ipynb', 'repo_name': 'moja-global/Land-Sector-Datasets', 'size': '11386'}