<jupyter_start><jupyter_text>Using the following packages:* pip install python-igraph* pip install partition-igraph* pip install hypernetx<jupyter_code>import pandas as pd
import igraph as ig
# import partition_igraph
import hypernetx as hnx
import pickle

## import the hypergraph functions -- HNX version
get_ipython().magic('run ./H_functions.py');<jupyter_output><empty_output><jupyter_text>Summary of functions for HNX hypergraphs Build hypergraph and pre-compute key quantitiesWe build the hypergraph HG using:```pythonHG = hnx.Hypergraph(dict(enumerate(Edges)))```where 'Edges' is a list of sets; edges are then indexed as 0-based integers,so to preserve unique ids, we represent nodes as strings.For example Edges[0] = {'0','2'}Once the HNX hypergraph is built, the following function is called to compute node strengths, d-degrees and binomial coefficients:```pythonHNX_precompute(HG)``` PartitionsWe use two representations for partitions: list of sets (the parts) or dictionary.Those functions are used to map from one to the other:```pythondict2part(D)part2dict(A)``` H-modularityThe function to compute H-modularity for HG w.r.t. partition A (list of sets covering the vertices):```pythonHNX_modularity(HG, A, wcd=linear)```where 'wcd' is the weight function (default = 'linear'). Other choices are 'strict'and 'majority', or any user-supplied function with the following format:```pythondef linear(d,c):    return c/d if c>d/2 else 0```where d is the edge size, and d>=c>d/2 the number of nodes in the majority class. Two-section graphBuild the random-walk based 2-section graph given some hypergraph HG:```pythonG = HNX_2section(HG)```where G is an igraph Graph. Clustering: Kumar algorithmGiven hypergraph HG, compute a partition of the vertices as per Kumar's algorithm described in [1].```pythonK = HNX_Kumar(HG, delta=.01)```where delta is the convergence stopping criterion. Partition is returned as a dictionary.[1] <NAME>., <NAME>., <NAME>., <NAME>., <NAME>. (2020) *A New Measure of Modularity in Hypergraphs: Theoretical Insights and Implications for Effective Clustering*. In: <NAME>., <NAME>., <NAME>., <NAME>., <NAME>. (eds) Complex Networks and Their Applications VIII. COMPLEX NETWORKS 2019. Studies in Computational Intelligence, vol 881. Springer, Cham. https://doi.org/10.1007/978-3-030-36687-2_24 Clustering: Simple qH-based algorithmGiven hypergraph HG and initial partition L, compute a partition of the vertices as per Last-Step algorithm described in [2].```pythonA = HNX_LastStep(HG, L, wdc=linear, delta = .01)```where 'wcd' is the the weight function (default = 'linear') and delta is the convergence stopping criterion.Returned partition is a list of sets.[2] <NAME>, <NAME> and <NAME>, *Community Detection Algorithm Using Hypergraph Modularity*, to appear in the proceedings of Complex Networks 2020, Springer.  Toy example<jupyter_code>## build an hypergraph from a list of sets (the hyperedges)
## using 'enumerate', edges will have integer IDs
E = [{'A','B'},{'A','C'},{'A','B','C'},{'A','D','E','F'},{'D','F'},{'E','F'}]
HG = hnx.Hypergraph(dict(enumerate(E)))
hnx.draw(HG)

## compute node strength (add unit weight is none), d-degrees, binomial coefficients
HNX_precompute(HG)

## the edges (unit weights added by default)
HG.edges.elements

## the nodes (here strength = degree since all weights are 1)
HG.nodes.elements

HG.d_weights

## compute modularity qH for the following partitions:
A1 = [{'A','B','C'},{'D','E','F'}]
A2 = [{'B','C'},{'A','D','E','F'}]
A3 = [{'A','B','C','D','E','F'}]
A4 = [{'A'},{'B'},{'C'},{'D'},{'E'},{'F'}]

print('linear:',HNX_modularity(HG,A1),HNX_modularity(HG,A2),HNX_modularity(HG,A3),HNX_modularity(HG,A4))
print('strict:',HNX_modularity(HG,A1,strict),HNX_modularity(HG,A2,strict),HNX_modularity(HG,A3,strict),HNX_modularity(HG,A4,strict))
print('majority:',HNX_modularity(HG,A1,majority),HNX_modularity(HG,A2,majority),HNX_modularity(HG,A3,majority),HNX_modularity(HG,A4,majority))

## 2-section graph
G = HNX_2section(HG)
G.vs['label'] = G.vs['name']
ig.plot(G,bbox=(0,0,250,250))

## 2-section clustering with ECG
G.vs['community'] = G.community_ecg().membership
dict2part({v['name']:v['community'] for v in G.vs})

## Clustering with Kumar's algorithm
dict2part(HNX_Kumar(HG))
## hypergraph clustering -- start from partition A4 defined above
print('start from:',A4)
A = HNX_LastStep(HG,A4)
print('final partition:',A)<jupyter_output><empty_output><jupyter_text>Game of Thrones scenes hypergraphREF: https://github.com/jeffreylancaster/game-of-thronesWe built an hypergraph from the game of thrones scenes with he following elements:* **Nodes** are characters in the series* **Hyperedges** are groups of character appearing in the same scene(s)* **Hyperedge weights** are total scene(s) duration in seconds involving those charactersWe kept hyperedges with at least 2 characters.Moreover, we discarded characters with degree below 5.We saved the following:* *Edges*: list of sets where the nodes are 0-based integers represents as strings* *Names*: dictionary; mapping of nodes to character names* *Weights*: list; hyperedge weights (in same order as Edges)<jupyter_code>## load the GoT dataset
Edges, Names, Weights = pickle.load(open( "../Data/GoT.pkl", "rb" ))<jupyter_output><empty_output><jupyter_text>Build weighted hypergraph<jupyter_code>## Nodes are represented as strings from '0' to 'n-1'
HG = hnx.Hypergraph(dict(enumerate(Edges)))
## add edge weights
for e in HG.edges:
    HG.edges[e].weight = Weights[e]
## add full names
for v in HG.nodes:
    HG.nodes[v].name = Names[v]
## pre-compute required quantities for modularity and clustering
HNX_precompute(HG)<jupyter_output><empty_output><jupyter_text>Modularity (qH) on a random partitionShould be close to 0 and can be negative.<jupyter_code>## generate a random partition into K parts to compare results
K = 5
V = list(HG.nodes)
p = np.random.choice(K, size=len(V))
RandPart = dict2part({V[i]:p[i] for i in range(len(V))})
## compute qH
HNX_modularity(HG, RandPart)<jupyter_output><empty_output><jupyter_text>Get the 2-section graph (with igraph) and cluster with Louvain<jupyter_code>## build 2-section
G = HNX_2section(HG)
## Louvain algorithm
ML = G.community_multilevel(weights='weight')
G.vs['louvain'] = ML.membership
part = dict2part({v['name']:v['louvain'] for v in G.vs})
## Compute qH
print(HNX_modularity(HG, part))<jupyter_output><empty_output><jupyter_text>Cluster with Kumar's algorithm<jupyter_code>## run Kumar's algorithm, get partition
KU = HNX_Kumar(HG)
G.vs['kumar'] = [KU[v['name']] for v in G.vs]
## Compute qH
print(HNX_modularity(HG, dict2part(KU)))<jupyter_output><empty_output><jupyter_text>Cluster with simple H-based (Last Step) AlgorithmWe use Louvain or Kumar algorithm on the 2-section as the required initial partition<jupyter_code>## Louvain parition already computed
part = dict2part({v['name']:v['louvain'] for v in G.vs})
## H-based last step
LS = HNX_LastStep(HG, part)
## Compute qH
HNX_modularity(HG, LS)<jupyter_output><empty_output><jupyter_text>Example: top nodes in cluster with <NAME><jupyter_code>## Index for 
inv_map = {v: k for k, v in Names.items()}
JS = inv_map['<NAME>']
## JS's cluster
JS_part = part2dict(LS)[JS]
## Build dataframe: all nodes in JS_part
L = []
for n in LS[JS_part]:
    L.append([Names[n],HG.nodes[n].strength])
D = pd.DataFrame(L, columns=['character','strength'])
D.sort_values(by='strength',ascending=False).head(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14035322451164034, 'fraction_numerical': 0.01404870216751405, 'mean_word_length': 6.588832487309645, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 55, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '610642', 'license': 'MIT', 'n_tokens_mistral': 2682, 'n_tokens_neox': 2466, 'n_words': 761, 'path': 'Notebooks/Hypergraph_Clustering.ipynb', 'repo_name': 'sssleverlily/hyperNetworkZ', 'size': '7555'}