<jupyter_start><jupyter_text>One of the general use cases for our "USGS graph" is pulling together all of the various data and information assets associated with a given topical are of some kind. With the graph, we've essentially stitched together all of the functionally disconnected catalogs and inventories we have built up separately for our people and their skills, publications and datasets and the subject matters they address, and all the other pockets of information we have describing and containing our work. The process of building the graph exercises all of that, evaluating and making decisions about the relationships between things, harmonizing different content models, and putting everything together in a way that it can be queried as a whole.One specific exercise of this concept is toward a catalog of all USGS climate science capabilities. On the one hand, this can be just about everything we do across earth system science, but we need to narrow in on those assets that are most closely connected to the topic area. This notebook explores the start to that process via a full text query across nodes containing likely content where we can initiate a search. From this, we can pull together a sub-graph based on the connections developed here.<jupyter_code>import isaid_helpers
import pandas as pd<jupyter_output><empty_output><jupyter_text>Figuring out what to query on is one part of the problem in any approach like this. From exercising things like our metadata keywords, we know that we don't have very strict alignment with controlled vocabularies or reference sources that can be exploited just yet. We've teased some of this out, making decisions about what to include and how to include it in our graph in the area of DefinedSubjectMatter and UndefinedSubjectMatter nodes.For climate science and this exercise, I've pulled in a glossary of terms from EPA that seem like they are reasonable as a starting point.<jupyter_code>epa_climate_change_glossary = pd.read_excel("https://sor.epa.gov/sor_internet/registry/termreg/searchandretrieve/glossariesandkeywordlists/search.do;jsessionid=87TsZz9rOm0k-CQjrtbfA43w1rPWKUuBf9N25Wg0XwzV9z4-yqOh!1399359231?details=&d-1342820-e=11&6578706f7274=1&glossaryName=Glossary+Climate+Change+Terms&includeInReport=")
epa_climate_change_glossary.head()<jupyter_output><empty_output><jupyter_text>To support this type of vector into the graph, we set up a full text index on names and descriptions across any of the nodes likely to contain reasonably useful content. We can run our search against this by sending all of our terms from the EPA glossary with OR conjunctions.<jupyter_code>with isaid_helpers.graph_driver.session(database=isaid_helpers.graphdb) as session:
    results = session.run("""
    CALL db.index.fulltext.queryNodes("allTexts", "%(search_list)s") YIELD node, score
    RETURN node.name AS name, labels(node)[0] AS item_type, node.url AS url, node.email AS email, node.orcid as orcid, node.doi AS doi, node.id_basis_project AS basis_project_number, node.source as source, score
    """ % {
        "search_list": "'"+"'""' OR '".join(list(epa_climate_change_glossary.Term))+"'"
    })
    query_results = results.data()
len(query_results)<jupyter_output><empty_output><jupyter_text>We do get a reasonable result set that seems logical for what we know we have in the graph at the moment. This represents about a third of our total nodes to date retrieved simply by looking for one take on key concepts associated with climate change science.<jupyter_code>df_query_results = pd.DataFrame(query_results)
display(df_query_results[['name','item_type']].groupby(['item_type']).agg(['count']))
display(df_query_results[['name','source']].groupby(['source']).agg(['count']))<jupyter_output><empty_output><jupyter_text>Some of these items are direct assets we would think about for our virtual catalog while others are starting points to network into the catalog via their relationships (e.g., we only got the people directly described in some way with these terms and we'd want to follow the graph to get everyone who has contributed).<jupyter_code>df_query_results
df_query_results.to_csv("data/usgs_climate_change_science_catalog.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06365985435752877, 'fraction_numerical': 0.010570824524312896, 'mean_word_length': 6.003289473684211, 'pattern_counts': {'":': 1, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '381676', 'license': 'Unlicense', 'n_tokens_mistral': 1183, 'n_tokens_neox': 1136, 'n_words': 506, 'path': 'isaid/Climate Change Science Catalog.ipynb', 'repo_name': 'skybristol/pylinkedcmd', 'size': '4257'}