<jupyter_start><jupyter_text>What next?Now that we've loaded the data into a graph, it's time to do some actual analysis. But to do so, we need to define the problem more clearly. Otherwise, there isan intractable amount of data for many graph algorithms. Pruning to relevant subgraphs would be desirable as an initial post-processing step.I am primarily interested in English language entries. However, ~~many~~ all of the English words are derived from non-English words. It would be good to pruneentries that are not etymological roots of English words.My initial instinct is to trim any descendant nodes of English words that are in other languages. Then, we can run BFS from each English node to gather itsancestors, knowing the descendant nodes have already been trimmed. This could accidentally exclude relevant data if there are derivation paths that jump fromEnglish to another language and then back, but that's an interesting question in and of itself and might be worth investigating as preliminary matter beforepursuing this approach.However, I suspect the BFS approach may be extremely inefficient and that it will be necessary to reduce the size of the search space to something moretractable.So. What exactly are we looking for?- Nodes with the "eng:" prefix- Nodes that are direct and indirect ancestors of the English nodes<jupyter_code>"""
DAG analysis

Check if graph is directed acyclic
"""

is_dag = graph.is_dag
print(is_dag)

"""
DAG analysis

I expected that the graph would be acyclic, because logically it doesn't make sense for a word to be an ancestor of a word that is also the first word's ancestor.
However, is_directed_acyclic_graph returned false, so I am going to check for cycles and see if there's something in the source data that can be cleaned up to
enable DAG analysis
"""
cycles = graph.get_cycles()
cycle_count = len(cycles)
print(cycle_count)

"""
DAG analysis

So... looks like all the cycles are self-connected single nodes. I'm guessing if I prune those it will be acyclic.

Tried this, and still there were some loops left. they were in somewhat obscure languages so I decided to just drop them all.
graph.remove_edges_from(nx.selfloop_edges(graph))

"""
graph.remove_cycles()
is_dag = graph.is_dag
print(is_dag)

"""
DAG analysis

Ok we have a DAG, time to see what kinds of fun patterns we can find...

"""
longest_path = graph.get_longest_path()
print(longest_path)

"""
DAG analysis

Create English-related subgraph

Let's see if we can exploit the DAG properties to filter to chains with English words only
"""
roots = graph.roots()
print(len(roots))
english_nodes = graph.language_nodes("eng")
print(len(english_nodes))
english_graph = graph.language_subgraph("eng", english_nodes)
DirectedGraph.subgraph_info(english_graph)

"""
English graph analysis

Inspect highest degree nodes
"""
nodes_by_degree = DirectedGraph.nodes_by_degree(english_graph)
print(nodes_by_degree[0:99])<jupyter_output>[('eng: non-', 5748), ('eng: -ly', 5439), ('eng: un-', 4206), ('eng: -like', 2438), ('eng: -er', 2346), ('eng: -less', 1974), ('ita: -mente', 1666), ('eng: -able', 1428), ('eng: time', 1322), ('eng: anti-', 1293), ('eng: -y', 1227), ('eng: poly-', 1004), ('ita: -abile', 880), ('eng: -ic', 812), ('eng: -ed', 754), ('eng: multi-', 715), ('eng: over-', 710), ('spa: -mente', 681), ('fra: -ure', 665), ('eng: -ian', 625), ('eng: -ish', 588), ('eng: -ally', 520), ('eng: -ize', 500), ('eng: dog', 496), ('eng: disease', 493), ('deu: Stein', 485), ('eng: inter-', 461), ('fra: -er', 451), ('eng: hyper-', 445), ('eng: bird', 444), ('fra: -ique', 442), ('ita: -oso', 441), ('eng: be-', 428), ('eng: in-', 426), ('fra: -age', 424), ('ita: -tura', 419), ('eng: micro-', 401), ('eng: de-', 397), ('eng: post-', 395), ('eng: -an', 392), ('eng: -ing', 383), ('eng: -ity', 383), ('eng: -ship', 380), ('gle: -acht', 376), ('ita: -mento', 356), ('eng: Chinese', 346), ('eng: semi-', 327), ('eng: back', 326), ('it[...]
{'fraction_non_alphanumeric': 0.14090564128509991, 'fraction_numerical': 0.0402226157348849, 'mean_word_length': 4.936936936936937, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '114219', 'license': 'MIT', 'n_tokens_mistral': 1359, 'n_tokens_neox': 1186, 'n_words': 545, 'path': 'notebooks/directed_acyclic_graph_analysis.ipynb', 'repo_name': 'jmcummings77/polyphasia', 'size': '3953'}