<jupyter_start><jupyter_text>Google ScholarThis notebook shows how to use the function `pymde.preserve_distances` to produce embeddings of networks, in which the goal is to preserve the shortest-path distances in the network.It uses an academic co-authorship network collected from Google Scholar as a case study.<jupyter_code>import pymde

import matplotlib.pyplot as plt
import numpy as np
import torch
gscholar = pymde.datasets.google_scholar()
scholars_df = gscholar.other_data['dataframe']
scholars_df
coauthorship_graph = gscholar.data
f'{coauthorship_graph.n_items:,} authors'
f'{coauthorship_graph.n_edges:,} edges'
print(f'edge density: {100*(coauthorship_graph.n_edges / (coauthorship_graph.n_all_edges)):.2f} percent')
device = 'cuda' if torch.cuda.is_available() else 'cpu'

mde = pymde.preserve_distances(
    data=coauthorship_graph,
    loss=pymde.losses.Absolute,
    max_distances=1e8,
    device=device,
    verbose=True)
plt.figure(figsize=(12, 3))
original_distances = np.sort(mde.distortion_function.deviations.cpu().numpy())
ax = plt.gca()
plt.hist(original_distances, histtype='step', bins=np.arange(1, 11), density=True, cumulative=True)
plt.xlim(1, 10)
plt.xticks(np.arange(1, 11))
plt.xlabel('graph distances')
plt.show()
mde.embed(verbose=True)
mde.distortions_cdf()
mde.plot(color_by=gscholar.attributes['coauthors'], color_map='viridis',
         figsize_inches=(12., 12.), background_color='k')
edges = coauthorship_graph.edges
indices = torch.randperm(edges.shape[0])[:1000]
edges = edges[indices].cpu().numpy()

mde.plot(edges=edges, color_by=gscholar.attributes['coauthors'], color_map='viridis', figsize_inches=(12, 12))
from matplotlib import colors


legend = {
    'bio': colors.to_rgba('tab:purple'),
    'ai': colors.to_rgba('tab:red'),
    'cs': colors.to_rgba('tab:cyan'),
    'ee': colors.to_rgba('tab:green'),
    'physics': colors.to_rgba('tab:orange')
}
scholar_disciplines_df = gscholar.other_data['disciplines']
topic_colors = [legend[code] for code in scholar_disciplines_df['topic']]
pymde.plot(mde.X[scholar_disciplines_df['node_id'].values], colors=topic_colors,
           figsize_inches=(12, 12), background_color='black')<jupyter_output><empty_output><jupyter_text>A neighbor-preserving embedding<jupyter_code>device = 'cuda' if torch.cuda.is_available() else 'cpu'

mde = pymde.preserve_neighbors(
    data=coauthorship_graph,
    device=device,
    verbose=True
)
mde.embed(verbose=True)
pymde.plot(mde.X[scholar_disciplines_df['node_id'].values], colors=topic_colors, figsize_inches=(12, 12), background_color='black')<jupyter_output>Mar 25 09:27:58 AM: Computing 15-nearest neighbors, with max_distance=3.0
Mar 25 09:27:58 AM: Computing shortest path distances (retaining 100.00 percent with max_distance=3.0) ...
Mar 25 09:27:59 AM: processed node 1/44682
Mar 25 09:28:03 AM: processed node 4469/44682
Mar 25 09:28:07 AM: processed node 8937/44682
Mar 25 09:28:10 AM: processed node 13405/44682
Mar 25 09:28:14 AM: processed node 17873/44682
Mar 25 09:28:17 AM: processed node 22341/44682
Mar 25 09:28:21 AM: processed node 26809/44682
Mar 25 09:28:24 AM: processed node 31277/44682
Mar 25 09:28:28 AM: processed node 35745/44682
Mar 25 09:28:31 AM: processed node 40213/44682
Mar 25 09:28:35 AM: processed node 44681/44682
Mar 25 09:28:37 AM: Computing quadratic initialization.
Mar 25 09:28:40 AM: Fitting a centered embedding into R^2, for a graph with 44682 items and 997885 edges.
Mar 25 09:28:40 AM: `embed` method parameters: eps=1.0e-05, max_iter=300, memory_size=10
Mar 25 09:28:40 AM: iteration 000 | distortion 0.739964 |[...]
{'fraction_non_alphanumeric': 0.12625139043381536, 'fraction_numerical': 0.089265850945495, 'mean_word_length': 6.237424547283702, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '603380', 'license': 'Apache-2.0', 'n_tokens_mistral': 1538, 'n_tokens_neox': 1276, 'n_words': 306, 'path': 'examples/google_scholar.ipynb', 'repo_name': 'vishalbelsare/pymde', 'size': '3596'}