<jupyter_start><jupyter_text>Building a concepts co-occurence networkThis Python notebook shows how to use the [Dimensions Analytics API](https://www.dimensions.ai/dimensions-apis/) in order to extract [concepts](https://docs.dimensions.ai/dsl/language.htmlsearching-using-concepts) from publications and use them to generate a 'topics map' using co-occurence information. For more background on concepts, see also the [Working with concepts in the Dimensions API](https://api-lab.dimensions.ai/cookbooks/1-getting-started/7-Working-with-concepts.html) tutorial.**Note** this tutorial is best experienced using Google Colab.   PrerequisitesThis notebook assumes you have installed the [Dimcli](https://pypi.org/project/dimcli/) library and are familiar with the *Getting Started* tutorial. The [networkx](https://networkx.github.io/documentation/stable/reference/introduction.html) and [pyvis](https://pyvis.readthedocs.io/en/latest/tutorial.html) libraries are used for generating and visualizing the network, respectively.<jupyter_code>!pip install dimcli plotly networkx pyvis jsonpickle  -U --quiet

import dimcli
from dimcli.utils import *
from dimcli.utils.networkviz import NetworkViz # custom version of pyvis - colab-compatible

import json
import sys
import pandas as pd
import networkx as nx
import plotly.express as px
import itertools

print("==\nLogging in..")
# https://digital-science.github.io/dimcli/getting-started.html#authentication
ENDPOINT = "https://app.dimensions.ai"
if 'google.colab' in sys.modules:
  import getpass
  KEY = getpass.getpass(prompt='API Key: ')  
  dimcli.login(key=KEY, endpoint=ENDPOINT)
else:
  KEY = ""
  dimcli.login(key=KEY, endpoint=ENDPOINT)
dsl = dimcli.Dsl()<jupyter_output>==
Logging in..
[2mDimcli - Dimensions API Client (v0.8.2)[0m
[2mConnected to: https://app.dimensions.ai - DSL v1.28[0m
[2mMethod: dsl.ini file[0m<jupyter_text>Step 1: Creating a dataset We start by creating a dataset consisting of the most cited 1000 publications matching a chosen keyword. The API query below will return a list of documents including all the related concepts. Tip: try changing the query keyword in order to experiment with different results.<jupyter_code>#@markdown Enter the keyword used to seed the search
KEYWORD = "Semantic Web" #@param {type:"string"}

q = f"""search publications 
            for "\\"{KEYWORD}\\"" 
        return publications[id+title+concepts_scores] 
        sort by times_cited limit 1000"""



data = dsl.query(q)
concepts = data.as_dataframe_concepts()
print("Total concepts:", len(concepts))
print("Concepts score average", concepts['score_avg'].mean())
concepts.head()<jupyter_output>Returned Publications: 1000 (total = 147455)
[2mTime: 9.02s[0m
Total concepts: 45773
Concepts score average 0.3791607853975051<jupyter_text>Step 2: Building a concepts co-occurrence networkEach publication in our dataset includes a list of related concepts. In order to build a concepts co-occurrence network, we simply add an edge between concepts that appear in the same document. Edges have a default weight of 1. If any two concepts appear together in more documents, we increase the weight each time. Note: the resulting network can be very large, so in order to make our network smaller (and more relevant), we can filter out less interesting concepts in two ways:* by setting a `frequency` and `score_avg` threshold (as shown in the [Working with concepts in the Dimensions API](https://api-lab.dimensions.ai/cookbooks/1-getting-started/7-Working-with-concepts.html) tutorial)* by keeping only nodes that, in our network, have an edge > MIN WEIGHT (that is, nodes that have more connections)<jupyter_code>G = nx.Graph() # networkX instance

#
# TIP play with these parameters in ordeto generate different types of networks
#
MIN_CONCEPT_SCORE = 0.6
MIN_CONCEPT_FREQUENCY = 4
MIN_EDGE_WEIGHT = 2

CONCEPTS_SET = concepts.query(f"score_avg >= {MIN_CONCEPT_SCORE} & frequency >=  {MIN_CONCEPT_FREQUENCY}")


#
# build nodes from concepts, including score_avg and frequency
# -- NOTE: score_bucket indicates if the concepts is above or below the mean_score
# -- this value is used in the visualization below to color-code nodes
#
mean_score = CONCEPTS_SET['score_avg'].mean()
for index, row in CONCEPTS_SET.drop_duplicates("concept").iterrows():
    score_bucket = 1 if row['score_avg'] > mean_score else 2
    G.add_node(row['concept'],frequency=row['frequency'], score_avg=row['score_avg'], score_bucket=score_bucket)
print("Nodes:", len(G.nodes()), "Edges:", len(G.edges()))

#
# build edges, based on concepts co-occurrence within pubs
# -- calculate a 'weight' based on how often two concepts co-occur
#
print(f".. adding edges from pubs cooccurrence...")
pubs_list = CONCEPTS_SET.drop_duplicates("id")['id'].to_list()

for p in pubs_list:
    concepts_for_this_pub = CONCEPTS_SET[CONCEPTS_SET['id'] == p]['concept'].to_list()
    for group in itertools.combinations(concepts_for_this_pub, 2):  # gen all permutations
        a, b = group[0], group[1]
        try:
            G.edges[a, b]['weight'] = G.edges[a, b]['weight'] + 1 
        except:
            G.add_edge(a, b, weight=1)
            
print("Nodes:", len(G.nodes()), "Edges:", len(G.edges()))

#
# this extra step is useful to remove low-weight connections
#

print(f".. cleaning up edges with weight < {MIN_EDGE_WEIGHT}...")

for a, b, w in list(G.edges(data='weight')):
    if w < MIN_EDGE_WEIGHT:
        G.remove_edge(a, b)
print("Nodes:", len(G.nodes()), "Edges:", len(G.edges()))

print(f".. removing isolated nodes...")

G.remove_nodes_from(list(nx.isolates(G)))
print("Nodes:", len(G.nodes()), "Edges:", len(G.edges()))<jupyter_output>Nodes: 178 Edges: 0
.. adding edges from pubs cooccurrence...
Nodes: 178 Edges: 1202
.. cleaning up edges with weight < 2...
Nodes: 178 Edges: 255
.. removing isolated nodes...
Nodes: 138 Edges: 255<jupyter_text>Step 3: Visualizing the network Note * We're using a custom version of [pyvis](https://pyvis.readthedocs.io/en/latest/tutorial.html) which is included in [dimcli.core.extras](https://github.com/digital-science/dimcli/blob/master/dimcli/core/extras.py) and is called `NetworkViz`. This custom version fixes a bug that prevents pyvis graphs to be displayed in Google Colab.* The pyvis `from_nx` method doesn't carry through WEIGHT or any other value from our network data; so we need to set it manually using via another pass * using `score_bucket` (see above), we can mark the higher-score concepts using a brighter color<jupyter_code>viznet = NetworkViz(notebook=True, width="100%", height="800px")
viznet.toggle_hide_edges_on_drag(True)
viznet.barnes_hut()
viznet.repulsion(300)
viznet.heading = f"Concepts co-occurrence for '{KEYWORD}' publications"


# reuse plotly color palette
palette = px.colors.diverging.Temps  # 7 colors

viznet.from_nx(G)


# update visual features 

for node in viznet.nodes:
    freq = G.nodes[node['label']]['frequency']
    score_avg = G.nodes[node['label']]['score_avg']
    score_bucket = G.nodes[node['label']]['score_bucket'] # get from original network

    node['size'] = freq * 2
    node['color'] = palette[3*score_bucket]  # get color based on score_bucket (1 or 2)
    node['borderWidthSelected'] = 5
    node['title'] = f"<h4>Concept: '{node['label']}'</h4><hr>Frequency: {freq}<br>Score avg: {score_avg}",
    # print(node)
for edge in viznet.edges:
    # get value from main Network weight
    edge['value'] = G.edges[edge['from'], edge['to']]['weight']
    # print(edge)
    
viznet.show("concepts_network.html")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12046922367207065, 'fraction_numerical': 0.01542111506524318, 'mean_word_length': 4.8685228151585465, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 12, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '459122', 'license': 'MIT', 'n_tokens_mistral': 2510, 'n_tokens_neox': 2345, 'n_words': 750, 'path': 'cookbooks/2-publications/Concepts-network-graph.ipynb', 'repo_name': 'digital-science/dimensions-api-lab', 'size': '7587'}