<jupyter_start><jupyter_text>__Description__This notebook consults the Wyoming Natural Diversity Database (WYNDD) via web services (documentation can be found at http://www.uwyo.edu/wyndd/data-dissemination/web-mapping-services/ ).  WYNDD has a comprehensive list of species that are found in Wyoming.  We use this as a starting point.  We then use ranges provided by WYNDD to understand which species are modeled to occur within the WLCI study area (n=6556).  We cache information provided by WYNDD and also cache information about range overlap within the WLCI boundaries (e.g. Percent of WLCI HUC10s and list of WLCI HUC10s with the species modeled occurrence).We also explore ways to retrieve geospatial information from WYNDD for mapping purposes.__Source(s)__'../sources/WLCI_Boundary/WLCI_Boundary.shp' : Shapefile of the WLCI boundary stored in sources folder for convenience.__Output(s)__"../cache/WYNDD_all_species_data.json" : species data from WYNDD"../cache/WYNDD_WLCI_species_data.json" : subset of species data from WYNDD for species that have ranges overlapping with WLCI boundary.  Also includes some calculated fields to document where species occur within the WLCI region.<jupyter_code>#Import needed packages
import json
import bispy
from IPython.display import display
from owslib.wfs import WebFeatureService
from geopandas.tools import sjoin
import geopandas as gpd
from requests import Request
import requests

bis_utils = bispy.bis.Utils()<jupyter_output><empty_output><jupyter_text>First we get information about the WYNDD web service so that we can better construct queries.  Again some documentation is provided at the following: http://www.uwyo.edu/wyndd/data-dissemination/web-mapping-services/ .<jupyter_code># URL for WFS backend
url = "http://prod.wyndd.org/geoserver/WYNDD/ows"

# See details about this particular WFS
# -------------------------------------
# Initialize
wfs = WebFeatureService(url=url)

# Service provider 
print(f'title: {wfs.identification.title}')

# Get WFS version
print(f'\nwfs version: {wfs.version}')

# Available methods
print(f'\nwfs methods: {[operation.name for operation in wfs.operations]}')

# Available data layers
print(f'\ndata layers: {list(wfs.contents)}')

# If needed you can print all metadata of all layers
#for layer, meta in wfs.items():
#    print(meta.__dict__)<jupyter_output>title: GeoServer Web Feature Service

wfs version: 1.0.0

wfs methods: ['GetCapabilities', 'DescribeFeatureType', 'GetFeature', 'Transaction', 'LockFeature', 'GetFeatureWithLock']

data layers: ['WYNDD:d_current_hunt_areas', 'WYNDD:d_data_products', 'WYNDD:d_data_request_origins', 'WYNDD:df_data_request_locations', 'WYNDD:ecological_systems', 'WYNDD:generalized_distribution_models', 'WYNDD:generalized_observations', 'WYNDD:generalized_ranges', 'WYNDD:generalized_species_list', 'WYNDD:hexes', 'WYNDD:hucs', 'WYNDD:location2', 'WYNDD:taxon_ecological_system_links', 'WYNDD:townships']<jupyter_text>--------------------------------------------------------------------------**************************************************************************Using WYNDD web servicesn retrieve the complete list of species from WYNDD database (species in Wyoming)<jupyter_code>result = bis_utils.processing_metadata()
result["processing_metadata"]["status"] = "failure"
result["processing_metadata"]["status_message"] = "Search failed"

# URL for WFS backend
url = "http://prod.wyndd.org/geoserver/WYNDD/ows"

# Defines layer of interest
layer = 'WYNDD:generalized_species_list'

# Specify the parameters for fetching the data
params = dict(service='WFS', version="1.0.0", request='GetFeature',typeName=layer, outputFormat='json')
result["parameters"] = params

# Parse the URL with parameters
query_url = Request('GET', url, params=params).prepare().url
result["processing_metadata"]["api"] = query_url

try:
    species_list = requests.get(url=query_url).json()
    result["processing_metadata"]["status"] = "success"
    result["processing_metadata"]["status_message"] = "species data returned"
    result["data"] = species_list
except:
    print ('FAILED')
#Create a list of species data
wyndd_spp_list = [i['properties'] for i in species_list['features']]

# of species in WYNDD species list
print (f'Number of species in WYNDD database: {len(wyndd_spp_list)}')
#Cache the array of WYNDD species data 
with open ('../cache/WYNDD_all_species_data.json', "w") as f:
    f.write(json.dumps(result))<jupyter_output><empty_output><jupyter_text>--------------------------------------------------------------------------**************************************************************************Range data in WYNDD are linked to HUC10s.  Here we find HUC10s that intersect the WLCI boundary.  This will help determine which species in WYNDD have an overlap with the WLCI boundary.<jupyter_code>#First we import the HUC10 data from WYNDD web services

# URL for WFS backend
url = "http://prod.wyndd.org/geoserver/WYNDD/ows"

# Initialize
wfs = WebFeatureService(url=url)

# Get data from WFS
# -----------------

# Fetch the last available layer (as an example) --> 'vaestoruutu:vaki2017_5km'
layer = 'WYNDD:hucs'

# Specify the parameters for fetching the data
params = dict(service='WFS', version="1.0.0", request='GetFeature',
      typeName=layer,  outputFormat='json')

# Parse the URL with parameters
q = Request('GET', url, params=params).prepare().url

# Read data from URL
huc10s = gpd.read_file(q)
huc_crs=huc10s.crs

#NOTE field names: Index(['id', 'huc10_id', 'geometry'], dtype='object')

print (f'Total Number of HUC10s in Wyoming: {huc10s.shape[0]}')

#Show first 3 records
huc10s.head(3)
#Now import the WLCI boundary from source folder
wlci_boundary = gpd.read_file('../sources/WLCI_Boundary/wlci_boundary.shp')

#reproject the WLCI boundary into a common coordinate system as HUC10 data from WYNDD
wlci_boundary = wlci_boundary.to_crs(huc_crs)
#Intersect HUC10s with WLCI boundary
spatial_join_gdf = sjoin(huc10s,wlci_boundary,how='right',op='intersects')

#Create list of HUC10s that Intersect WLCI boundary
wlci_huc10s=spatial_join_gdf['huc10_id'].tolist()

print (f'Total Number of HUC10s in WLCI: {len(wlci_huc10s)}')<jupyter_output>Total Number of HUC10s in WLCI: 156<jupyter_text>--------------------------------------------------------------------------**************************************************************************Query WYNDD range data to determine which species can be found in WLCI using HUC10 data from above.<jupyter_code>from bispy import bis
bis_utils = bis.Utils()

def wyndd_get_wlci(wyndd_spp_properties):
    '''
    description: pass in properties on a species from the wyndd species list, summarize range data (so we don't have to store it all) and return summary
    '''
    # build query url
    result = bis_utils.processing_metadata()
    result["processing_metadata"]["status"] = "failure"
    result["processing_metadata"]["status_message"] = "Search failed"
    
    query_str = f"taxon_id='{str(wyndd_spp_properties['taxon_id'])}'"
    params = dict(service='WFS', version="1.0.0", request='GetFeature',typeName=layer, CQL_FILTER=query_str,  outputFormat='json')
    q = Request('GET', url, params=params).prepare().url
    result["processing_metadata"]["api"] = q
    result["parameters"] = params
    result['Scientific Name'] = wyndd_spp_properties['sciname']
    
    spp_info = dict()
    
    try:
    
        # Read data from URL into geopandas
        spp_data = gpd.read_file(q)

        #Select only hucs in spp range
        range_huc10s = spp_data.loc[spp_data['in_range']==True]

        #create list of huc10s in WlCI boundary and spp range
        wlci_spp_huc10s = [i for i in range_huc10s['huc10_id'].tolist() if i in wlci_huc10s]

        #Note there are 156 huc10s that intersect WLCI boundary
        
        #I need to verify this but it looks like if one record and huc10_id = 000000000 then it occurs state wide
        #In that case assign full list of hucs to spp
        if len(wlci_spp_huc10s)>0 and '0000000000' in wlci_spp_huc10s:
            spp_info['wyndd_taxon_data']=spp
            #All HUC10s intersect the spp range so we assign count of 156 and percent of HUCs as 100%
            spp_info['wyndd_wlci_huc10_ct']= 156
            spp_info['wyndd_wlci_huc10_pct']= 100  
            #Assign a full list of WLCI HUC10 ids 
            spp_info['wyndd_wlci_huc10_ids']= wlci_huc10s
            result['data']=spp_info

        #If specific huc data are present then 
        elif len(wlci_spp_huc10s)>0:
            spp_info['wyndd_taxon_data']=spp
            #Record count of HUC10s that intersect the spp range
            spp_info['wyndd_wlci_huc10_ct']= len(wlci_spp_huc10s)
            spp_info['wyndd_wlci_huc10_pct']= len(wlci_spp_huc10s)/156*100  
            #Assign a list of WLCI HUC10 ids for HUC10s that intersect the WLCI boundary
            spp_info['wyndd_wlci_huc10_ids']= wlci_spp_huc10s
            result['data']=spp_info
            
        result["processing_metadata"]["status"] = "success"
        result["processing_metadata"]["status_message"] = "success"
        return result
    except:
        return result
            

#NOTE:  This takes roughly an hour to run, if you want to test run a subset of species 
#replace "for spp in wyndd_spp_list:" with something similar to "for spp in wyndd_spp_list[0:10]:"

# URL for WFS backend
url = "http://prod.wyndd.org/geoserver/WYNDD/ows"

# specify range layer
layer = 'WYNDD:generalized_ranges'
wlci_wyndd_spieces_data = []

for spp in wyndd_spp_list:
    
    result = wyndd_get_wlci(spp)
    if result and "data" in result.keys():
        wlci_wyndd_spieces_data.append(result)
len(wlci_wyndd_spieces_data)
# Cache the array of WYNDD species data and return/display a random sample for verification
display(bis_utils.doc_cache("../cache/WYNDD_WLCI_species_data.json", wlci_wyndd_spieces_data))
wyndd_wlci_sciname_list = [i['Scientific Name'] for i in wlci_wyndd_spieces_data]
with open('../sources/wyndd_wlci_species_list.json', 'w') as f:
    json.dump(wyndd_wlci_sciname_list, f)
#Below is an example of how to visualize range data from the WYNDD servies
import geopandas as gpd
from requests import Request
from owslib.wfs import WebFeatureService
import matplotlib.pyplot as plt
%matplotlib inline 

# URL for WFS backend
url = "http://prod.wyndd.org/geoserver/WYNDD/ows"

# Initialize
wfs = WebFeatureService(url=url)

# Fetch the last available layer (as an example) --> 'vaestoruutu:vaki2017_5km'
layer = 'WYNDD:generalized_ranges'

# Specify the parameters for fetching the data
params = dict(service='WFS', version="1.0.0", request='GetFeature',
      typeName=layer, CQL_FILTER="""sciname='Passerculus sandwichensis'""",  outputFormat='json')

# Parse the URL with parameters
q = Request('GET', url, params=params).prepare().url

# Read data from URL
data = gpd.read_file(q)

data.plot(column='occurrence', legend=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14090536478732124, 'fraction_numerical': 0.015939520903543128, 'mean_word_length': 4.273775216138328, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '354311', 'license': 'Unlicense', 'n_tokens_mistral': 3741, 'n_tokens_neox': 3526, 'n_words': 986, 'path': 'workflow/Consult-and-Explore-WYNDD-Species-Data.ipynb', 'repo_name': 'usgs-bcb/wlci-species', 'size': '10979'}