<jupyter_start><jupyter_text>2021-04 NSF Training part 1Outline 1. The grants data model: key concepts2. Getting publications by grants id, directorate, funder groups3. Getting researcher awards, publications, patents, etc.4. Finding similar awards/publications based on a given awards/publications        PrerequisitesThis notebook assumes you have installed the [Dimcli](https://pypi.org/project/dimcli/) library and are familiar with the *Getting Started* tutorial.<jupyter_code>!pip install dimcli --quiet 

import dimcli
from dimcli.utils import *

import json
import sys
import pandas as pd
from IPython.display import Image
from IPython.core.display import HTML

print("==\nLogging in..")

ENDPOINT = "https://app.dimensions.ai"

if 'google.colab' in sys.modules:
  import getpass
  KEY = getpass.getpass(prompt='API Key: ')  
  dimcli.login(key=KEY, endpoint=ENDPOINT)
else:
  KEY = ""
  dimcli.login(key=KEY, endpoint=ENDPOINT)

dsl = dimcli.Dsl()<jupyter_output>==
Logging in..
[2mDimcli - Dimensions API Client (v0.9)[0m
[2mConnected to: https://app.dimensions.ai - DSL v1.30[0m
[2mMethod: dsl.ini file[0m<jupyter_text>1. The grants data model - quick walkthrough Reference: https://docs.dimensions.ai/dsl/datasource-grants.html  1.1 Dimensions grant ID VS 'original' grant ID (= `grant_number`)<jupyter_code>%%dsldf

search grants 
    for "covid-19 OR coronavirus" 
return grants[id+grant_number+title] limit 5<jupyter_output>Returned Grants: 5 (total = 8090)
[2mTime: 0.55s[0m<jupyter_text>If you have the IDs, you can search using them:<jupyter_code>%%dsldf

search grants 
    where id in ["grant.9606016", "grant.9525993"]
    or grant_number in ["198908", "894449", "SAB20-0064"]
return grants[id+grant_number+title]<jupyter_output>Returned Grants: 5 (total = 5)
[2mTime: 0.60s[0m<jupyter_text>1.2 Field types: 'Entity' (eg `funders`) VS lists (`funder_countries`) VS atomic data type (`funding_org_name`)<jupyter_code>%%dsldf

search grants 
    for "covid-19 OR coronavirus" 
return grants[id+funding_org_name+unnest(funder_countries)] limit 5<jupyter_output>Returned objects: 5 (total grants= 8090)
[2mTime: 0.53s[0m<jupyter_text>Use `unnest(funders)` to unpack the contents of that field (docs:https://docs.dimensions.ai/dsl/language.htmlunnesting-multi-value-entity-fields)<jupyter_code>%%dsldf

search grants 
    for "covid-19 OR coronavirus" 
return grants[id+unnest(funders)] limit 5<jupyter_output>Returned objects: 5 (total grants= 8090)
[2mTime: 2.43s[0m<jupyter_text>1.3 Funder GRID identifiers (`funders.id`)Caveat: entity filters are resolved via a 'subquery' and are better used only with identifiers or properties matching a small number of results. See also https://docs.dimensions.ai/dsl/language.htmlliteral-fields-vs-entity-fields<jupyter_code>%%dsldf

search grants for "covid-19 OR coronavirus" 
    where funders.id = "grid.425888.b"
return grants limit 5<jupyter_output>Returned Grants: 5 (total = 91)
[2mTime: 0.60s[0m<jupyter_text>1.4 Funder groups aggregations (= agencies, as defined in Dimensions)* Funder 'group' ie NSF: usually consists of more than one GRID identifier. These can be obtained from the webapp.     * https://app.dimensions.ai/browse/facet-filter-groups/publication* Division or Program: not available as such in Dimensions    * should be tackled via specific grants IDs<jupyter_code># load a CSV containing funder group infos, exported from Dimensions

pd.read_csv("Dimensions-My-groups-2021-04-07_08-05-49.csv")<jupyter_output><empty_output><jupyter_text>Let's get the GRID IDs for the NSF and put them into a Python list.Then we can generate queries programmatically using this list. > For more background on this topic, see the [Working with lists in the Dimensions API](https://api-lab.dimensions.ai/cookbooks/1-getting-started/6-Working-with-lists.html) tutorial.<jupyter_code>nsfgrids = pd.read_csv("Dimensions-My-groups-2021-04-07_08-05-49.csv")['ID'].to_list()<jupyter_output><empty_output><jupyter_text>How many grants from the NSF? The total number of results should match [what you see in Dimensions](https://app.dimensions.ai/discover/publication?and_facet_funder_shared_group_facet=574603a4-0c27-4844-9f74-7e6810e25cfb).<jupyter_code>import json

query = f"""
search grants 
    where funders.id in {json.dumps(nsfgrids)}
return grants[id+title]
"""

print(query)

grants = dsl.query(query)
grants.as_dataframe()<jupyter_output>search grants 
    where funders.id in ["grid.457768.f", "grid.457785.c", "grid.457799.1", "grid.457810.f", "grid.457836.b", "grid.457875.c", "grid.457916.8", "grid.457789.0", "grid.457813.c", "grid.457814.b", "grid.457842.8", "grid.457821.d", "grid.457772.4", "grid.457801.f", "grid.457891.6", "grid.457892.5", "grid.457845.f", "grid.457922.f", "grid.457896.1", "grid.431093.c", "grid.457758.c", "grid.457907.8", "grid.473792.c", "grid.457846.c", "grid.457898.f"]
return grants[id+title]

Returned Grants: 20 (total = 532277)
[2mTime: 0.55s[0m<jupyter_text>1.5 From grants to other entities: the links available<jupyter_code>Image(url= "https://docs.dimensions.ai/dsl/_images/data-model-overview-1.png", width=1000)
Image(url= "https://docs.dimensions.ai/dsl/_images/data-model-grants.png", width=1000)<jupyter_output><empty_output><jupyter_text>2. Getting publications from Grants  2.1 Starting from Dimensions grant IDsFirst let's retrieve a sample set of NSF grants from Dimensions<jupyter_code>query = f"""
search grants 
    for "quantum leap" 
    where funders.id in {json.dumps(nsfgrids)}
    and active_year <= 2018 
return grants[id+title]
"""

grants = dsl.query(query)
grants.as_dataframe()<jupyter_output>Returned Grants: 20 (total = 143)
[2mTime: 0.75s[0m<jupyter_text>Now we can use the IDs to retrieve all resulting publications.NOTE: the direction of the search is from publications to grants, using the `supporting_grant_ids` field.<jupyter_code>grantids = grants.as_dataframe()['id'].to_list()

query = f"""search publications
          where supporting_grant_ids in {json.dumps(grantids)}
       return publications[title+doi+year+supporting_grant_ids]"""

pubs = dsl.query(query).as_dataframe()
pubs<jupyter_output>Returned Publications: 20 (total = 113)
[2mTime: 0.60s[0m<jupyter_text>2.2 Starting from funders IDsWe can use the field `funders`,  since it is a direct link from Publications to Funder Organizations.<jupyter_code>query = f"""
search publications 
    for "quantum leap" 
    where funders.id in {json.dumps(nsfgrids)}
return publications[id+title+year+unnest(supporting_grant_ids)]
"""

pubs = dsl.query(query).as_dataframe()
pubs<jupyter_output>Returned objects: 86 (total publications= 960)
[2mTime: 0.64s[0m<jupyter_text>Now we can retrieve the associated grants.<jupyter_code>grantids = pubs['supporting_grant_ids'].to_list()

query = f"""
search grants 
    where id in {json.dumps(grantids)}
return grants[id+title+funders]
"""

linkedgrants = dsl.query(query).as_dataframe()
linkedgrants<jupyter_output>Returned Grants: 20 (total = 83)
[2mTime: 0.92s[0m<jupyter_text>More learning resources* [Enriching Grants part 2: Adding Publications Information from Dimensions](https://api-lab.dimensions.ai/cookbooks/3-grants/2-grants-enrichment-adding-publications-information.html)* [Working with longs lists of IDs](https://api-lab.dimensions.ai/cookbooks/1-getting-started/6-Working-with-lists.html5.-How-Long-can-lists-get?)  3. Getting researcher awards, publications, patents Let's start by retrieving a list of recently NSF-awarded researchers on Covid-19 related topics. We can use a facet query to quickly get this data.Note: by default, the `researchers` facet returns records sorted by number of associated grants.<jupyter_code>COVID_Q = '"2019-nCoV" OR "COVID-19" OR "SARS-CoV-2" OR "HCoV-2019" OR "hcov" OR "NCOVID-19" OR  "severe acute respiratory syndrome coronavirus 2" OR "severe acute respiratory syndrome corona virus 2" OR "coronavirus disease 2019" OR (("coronavirus"  OR "corona virus") AND (Wuhan OR China OR novel))'


query = f"""
search grants 
    for "{dsl_escape(COVID_Q)}"
    where funders.id in {json.dumps(nsfgrids)}
    and active_year >= 2020
return researchers[id+first_name+last_name+dimensions_url]
"""

researchers = dsl.query(query).as_dataframe()
researchers<jupyter_output>Returned Researchers: 20
[2mTime: 0.59s[0m<jupyter_text>Dimcli includes a function to quickly export a dataframe to google sheets ([documentation](https://digital-science.github.io/dimcli/modules.htmldimcli.utils.misc_utils.export_as_gsheets)). This is often handy to further analysing or for sharing the data.<jupyter_code>export_as_gsheets(researchers)<jupyter_output>..authorizing with google..[0m
..creating a google sheet..[0m
..uploading..[0m<jupyter_text>3.1 AwardsYou can select any number of researcher IDs and retrieve related data. Tips * If we `return grants` we get the full grant details  * If we `return funders` we can get a quick overview of who's funding these researchers.<jupyter_code>focus = researchers['id'][:1].to_list()
query = f"""
search grants 
    where researchers.id in {json.dumps(focus)}
return grants[title+active_year+id+language+grant_number+funding_org_name+researchers]
"""

awards = dsl.query(query).as_dataframe()
awards<jupyter_output>Returned Grants: 12 (total = 12)
[2mTime: 0.59s[0m<jupyter_text>3.2 Publications<jupyter_code>focus = researchers['id'][:1].to_list()<jupyter_output><empty_output><jupyter_text>Let's get recent publications:<jupyter_code>query = f"""
search publications 
    where researchers.id in {json.dumps(focus)}
    and year > 2015
return publications[id+year+authors_count+title+type+journal+field_citation_ratio+times_cited+supporting_grant_ids+funders]
sort by times_cited limit 10
"""

pubs = dsl.query(query).as_dataframe()
pubs<jupyter_output>Returned Publications: 10 (total = 52)
[2mTime: 0.60s[0m<jupyter_text>3.3 Patents<jupyter_code>focus = researchers['id'][:10].to_list()
query = f"""
search patents 
    where researchers.id in {json.dumps(focus)}
return patents[basics+researchers+dimensions_url] 
limit 10

"""

patents = dsl.query(query).as_dataframe()
patents
export_as_gsheets(patents)<jupyter_output>..authorizing with google..[0m
..creating a google sheet..[0m
..uploading..[0m<jupyter_text>More learning resources* [Enriching Grants part 3: adding related Patents and Clinical Trials data](https://api-lab.dimensions.ai/cookbooks/3-grants/3-grants-enrichment-adding-patents-cltrials-information.html)* [Retrieve all grants funding publications ](https://api-lab.dimensions.ai/cookbooks/1-getting-started/6-Working-with-lists.html9.-Patents-example-(patents--%3E-publications))  4. Finding similar awards / publicationsThe procedure is similar for both grants and publications. It involves extracting concepts and then using those concepts to find related records.     Let's generate a sample dataset of NSF grants on 'quantum leap'. NOTE this time the DSL query includes also the field `concepts_scores`.  For background, see also the [Working with concepts in the Dimensions API](https://api-lab.dimensions.ai/cookbooks/1-getting-started/7-Working-with-concepts.html) tutorial.<jupyter_code>query = f"""
search grants 
    for "quantum leap" 
    where funders.id in {json.dumps(nsfgrids)}
    and active_year >= 2015 
return grants[id+title+concepts_scores] limit 1000
"""

grants = dsl.query(query)
grants.as_dataframe()<jupyter_output>Returned Grants: 162 (total = 162)
[2mTime: 1.19s[0m<jupyter_text>We then 'explode' the concepts data so to remove nested values<jupyter_code>concepts = grants.as_dataframe_concepts()
concepts<jupyter_output><empty_output><jupyter_text>4.1 Analysing the concepts we found using frequency and score<jupyter_code>concepts_unique = concepts.drop_duplicates("concept")[['concept', 'frequency', 'score_avg']]
concepts_unique<jupyter_output><empty_output><jupyter_text>We can use the plotly visualization library to explore the concepts:<jupyter_code>import plotly.express as px

FREQ_MIN = 2 
FREQ_MAX = 70 
SCORE_MIN = 0.6 
MAX_CONCEPTS = 200 

filtered_concepts = concepts_unique.query(f"""frequency >= {FREQ_MIN} & frequency <= {FREQ_MAX} & score_avg >= {SCORE_MIN} """)\
                    .sort_values(["score_avg", "frequency"], ascending=False)[:MAX_CONCEPTS]

px.scatter(filtered_concepts,
           x="concept",
           y="frequency",
           height=700,
           color="score_avg",
           size="score_avg")<jupyter_output><empty_output><jupyter_text>4.2 Using 'shared' concepts to find similar documents Let's get a subset of those concepts to find 'similar' grants.<jupyter_code>focusconcepts = concepts_unique.query(f"""frequency >= 3 & frequency <= 20 & score_avg >= 0.7 """)\
                    .sort_values(["score_avg", "frequency"], ascending=False)
focusconcepts
names = focusconcepts['concept'].to_list()

q_focusconcepts = " OR ".join(['"%s"' % x for x in names])


query = f"""
search grants 
    in concepts 
    for "{dsl_escape(q_focusconcepts)}" 
    where active_year >= 2015
    not funders.id in {json.dumps(nsfgrids)}
return grants[id+title+funding_org_name] limit 1000
"""

print(query)

grants = dsl.query(query)
grants.as_dataframe()<jupyter_output>search grants 
    in concepts 
    for "\"second quantum revolution\" OR \"single-photon emitters\" OR \"topological insulators\" OR \"quantum information processing\" OR \"neutron stars\" OR \"exciton states\" OR \"information processing devices\" OR \"quantum emitters\" OR \"quantum photonic technologies\" OR \"quantum information technology\" OR \"ultra-high vacuum technology\" OR \"atomic species\" OR \"long coherence times\" OR \"cavity quantum electrodynamics\" OR \"quantum particle\" OR \"quantum mechanics\" OR \"two-dimensional materials\" OR \"quantum coherence\" OR \"quantum systems\" OR \"semiconductor heterostructures\" OR \"mathematical physics\" OR \"quantum optics\" OR \"quantum information community\" OR \"coherence time\" OR \"quantum algorithms\"" 
    where active_year >= 2015
    not funders.id in ["grid.457768.f", "grid.457785.c", "grid.457799.1", "grid.457810.f", "grid.457836.b", "grid.457875.c", "grid.457916.8", "grid.457789.0", "grid.457813.c", "grid.457814.b"[...]<jupyter_text>4.3 Most similar grants, by funder<jupyter_code>query = f"""
search grants 
    in concepts 
    for "{dsl_escape(q_focusconcepts)}" 
    where active_year >= 2015
    not funders.id in {json.dumps(nsfgrids)}
return funders
"""

grants = dsl.query(query).as_dataframe()

px.pie(grants,
       names='name', values="count",
      title = "Top funders (non-NSF)")<jupyter_output>Returned Funders: 20
[2mTime: 0.63s[0m
{'fraction_non_alphanumeric': 0.1261857640073705, 'fraction_numerical': 0.04292636320207466, 'mean_word_length': 5.438488576449912, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 97, 'https://': 17, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '392897', 'license': 'MIT', 'n_tokens_mistral': 5513, 'n_tokens_neox': 4903, 'n_words': 1245, 'path': 'archive/2021-04-NSF-Training/2021-04-NSF-Part-1.ipynb', 'repo_name': 'digital-science/dimensions-api-examples', 'size': '14653'}