<jupyter_start><jupyter_text>[Link to the Reveal.js Presentation](https://knu2xs.github.io/ba-lab/reveal/ba-introspection.slides.html)  Business Analyst IntrospectionThe first step in using ArcGIS Business Analyst for data enrichment is figuring out what is possible, what you *can* do. This begins with selecting a soruce, figuring out which countries are available from the given source, and discovering what variables are available in the country you are interested in working in.  The `BusinessAnalyst` objectBusiness Analyst functionality can be accessed throughout the ArcGIS Platform from a variety of sources. These sources are either locally or through a Web GIS. The `BusinessAnalyst` object, when instantiated, selects the source automatically or honors the `source` input parameter to set it explicitly.<jupyter_code>from arcgis.geoenrichment import BusinessAnalyst as BA<jupyter_output><empty_output><jupyter_text>Local Business Analyst SourceThe `BusinessAnalyst` object can explicitly be set to reference local resources using the `local` keyword for the first input parameter, `source`. Utilizing local resources requires the Python environment to have `arcpy` installed referencing ArcGIS Pro with Business Analyst and local data for at least one Country installed.<jupyter_code>ba_lcl = BA('local')

ba_lcl<jupyter_output><empty_output><jupyter_text>Web GIS `BusinessAnalyst` SourceA Web GIS can also be used with the `BusinessAnalyst` object in the form of a `GIS` object instance enabling access to either an Enterprise GIS or ArcGIS Online. If using an Enterprise GIS, Enrichment must be configured in the settings. This can either reference ArcGIS Online or a configured Business Analyst server.<jupyter_code>from arcgis.gis import GIS
gis = GIS()  # connecting to ArcGIS Online as an anonymous user

ba_gis = BA(gis)

ba_gis<jupyter_output><empty_output><jupyter_text>Implicit `BusinessAnalyst` SourceWhen instantiating the `BusinessAnalyst` object, it is not imperative to explicitly set the `source`. If left blank, it will attempt to set the source to `local` if Pro with Business Analyst is available, or utlize the `active_gis` if this has been set in the Python session.<jupyter_code>ba_impl = BA()  # since I have Pro with BA, Enrichment defaults to local

ba_impl<jupyter_output><empty_output><jupyter_text>Discovering CountriesBoth configurations enable discovering what countries are available using idential syntax.  Local Country Introspection<jupyter_code>ba_lcl.countries<jupyter_output><empty_output><jupyter_text>Web GIS Country Introspection<jupyter_code>ba_gis.countries<jupyter_output><empty_output><jupyter_text>Enrich Variable IntrospectionCurrently the `Country` object is how most data is organized in Business Analyst. Consequently, before discovering what variables are available for enrichment, a country must be retrieved from the instantiated `BusinessAnalyst` object. The syntax is idential whether using `local` or a`GIS` object instance as the source.  Local Country Instantiation<jupyter_code>usa_lcl = ba_lcl.get_country('USA')

usa_lcl<jupyter_output><empty_output><jupyter_text>Web GIS Country Introspection<jupyter_code>usa_gis = ba_gis.get_country('USA')

usa_gis<jupyter_output><empty_output><jupyter_text>Local Coutntry with Year InstantiationWith a local source, clients also periodically a need to explicitly access older data to use with models built against a specific year's data. Explicitly setting the year is supported using the `year` parameter.<jupyter_code>usa_lcl_2019 = ba_lcl.get_country('USA', year=2019)

usa_lcl_2019<jupyter_output><empty_output><jupyter_text>Enrich Variable IntrospectionOnce a country is instantiated, data available for enrichment can be discovered through the `Country` object instance. Whether the source is local or a GIS instance, the method is identical.  Local Variable Introspection<jupyter_code>usa_lcl.variables<jupyter_output><empty_output><jupyter_text>Web GIS Variable Introspection<jupyter_code>usa_gis.variables<jupyter_output><empty_output><jupyter_text>Selecting Variables for EnrichmentSince a Pandas DataFrame, this facilitates quick discovery of different variable combinations based on analysis needs.  Unique VariablesSince variables can be repeated due to being used in multiple Data Collections, we can easily remove duplicates using the functionality of the DataFrame.<jupyter_code>uniq_df = usa_lcl.variables.drop_duplicates('name').reset_index(drop=True)

uniq_df<jupyter_output><empty_output><jupyter_text>Unique Current Year VariablesOne of the things I like to do is grab all the current year demographics. Due to the naming convention, all of these variables' names end with `CY`. This enables us to quickly find them.<jupyter_code>cy_df = uniq_df[uniq_df['name'].str.endswith('CY')].drop_duplicates('name').reset_index(drop=True)

cy_df<jupyter_output><empty_output><jupyter_text>Current Year Sample VariablesFrequently, when quickly demonstrating analysis ad-hoc, I take advantage of Data Collections to grab a few useful current year variables.<jupyter_code>var_df = usa_lcl.variables

smpl_df = var_df[
    (var_df['name'].str.endswith('CY'))                  # current year
    & (var_df['data_collection'].str.startswith('Key'))  # key variables
].reset_index(drop=True)

smpl_df<jupyter_output><empty_output><jupyter_text>EnrichmentThere are two ways to access enrichment from Python. The ArcGIS Pro Geoprocessing Enrich Layer tool enables enrichment data using local resources. The Python API already includes a wrapper around the enrich REST endpoint to use a Web GIS. Each method requires the variable input in a slightly different format.  Local EnrichmentEnrichment varialbes must be concantenated into a single semicolon separated string for input into the Enrich Layer Geoprocessing tool. This can be accommplished using a little string concantenation in Python.<jupyter_code>enrich_str = ';'.join(smpl_df.enrich_name)

print(enrich_str)<jupyter_output>KeyUSFacts.TOTPOP_CY;KeyUSFacts.GQPOP_CY;KeyUSFacts.DIVINDX_CY;KeyUSFacts.TOTHH_CY;KeyUSFacts.AVGHHSZ_CY;KeyUSFacts.MEDHINC_CY;KeyUSFacts.AVGHINC_CY;KeyUSFacts.PCI_CY;KeyUSFacts.TOTHU_CY;KeyUSFacts.OWNER_CY;KeyUSFacts.RENTER_CY;KeyUSFacts.VACANT_CY;KeyUSFacts.MEDVAL_CY;KeyUSFacts.AVGVAL_CY;KeyUSFacts.POPGRW10CY;KeyUSFacts.HHGRW10CY;KeyUSFacts.FAMGRW10CY;KeyUSFacts.DPOP_CY;KeyUSFacts.DPOPWRK_CY;KeyUSFacts.DPOPRES_CY<jupyter_text>Web GIS EnrichmentEnriching using the Python API calling the rest endpoint requires slightly different syntax, a semicolon separated list of variable names.<jupyter_code>enrich_str = ';'.join(smpl_df.name)

print(enrich_str)<jupyter_output>TOTPOP_CY;GQPOP_CY;DIVINDX_CY;TOTHH_CY;AVGHHSZ_CY;MEDHINC_CY;AVGHINC_CY;PCI_CY;TOTHU_CY;OWNER_CY;RENTER_CY;VACANT_CY;MEDVAL_CY;AVGVAL_CY;POPGRW10CY;HHGRW10CY;FAMGRW10CY;DPOP_CY;DPOPWRK_CY;DPOPRES_CY<jupyter_text>Next Step - Enrich MethodThe next step is adding `enrich` method to support both local and remote resources, and handle the variable introspection dataframes as input. This enables a single workflow for discovering what data is available and retrieving the data whether the data is being retrieved from a local or Web GIS source.<jupyter_code># this does not run, but will soon...
in_pth = 'C:/path/to/data.gdb/features'
in_df = pd.DataFrame.spatial.from_featureclass(in_pth)

# local enrichment
enrich_lcl_df = usa_lcl.enrich(in_df, key_vars)

# web gis enrichment
enrich_gis_df = usa_gis.enrich(in_df, key_vars)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0640701847667154, 'fraction_numerical': 0.0033231423634188487, 'mean_word_length': 7.169381107491857, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '43045', 'license': 'Apache-2.0', 'n_tokens_mistral': 2413, 'n_tokens_neox': 2287, 'n_words': 738, 'path': 'notebooks/ba-introspection.ipynb', 'repo_name': 'knu2xs/ba-lab', 'size': '7523'}