<jupyter_start><jupyter_text>Make a graph- the value of edges should be weighted by frequency of the overlapping tag<jupyter_code>G, num_conns = post_df_to_graph(post_df, tag_df)
print(len(G.edges()), len(G.nodes()))
# show top connected posts
display(num_conns.sort_values(by="conns", ascending=False)[:5])
# take a look at the weight probabilities
weights = np.array([i['weight'] for i in list(G.edges().values())])
fig, ax = plt.subplots(ncols=1, figsize=(12,4))
ax.hist(weights, bins=50);
from wordpress_graph.paths import DATA_DIR
nx.write_gpickle(G, (DATA_DIR/"network_path.gpickle").as_posix())<jupyter_output><empty_output><jupyter_text>Cluster communiities<jupyter_code>import community
import seaborn as sns
partition = community.best_partition(G)
set(partition.values())
pal = sns.color_palette("tab10", n_colors=len(set(partition.values())))
sns.palplot(pal)

for count, com in enumerate(set(partition.values())):
    list_nodes = [nodes for nodes in partition.keys() if partition[nodes] == com]
    
    print(count)
    for node in list_nodes[:10]:
        print('\t' + node.replace('_', ' '))<jupyter_output>0
	critical behavior from deep dynamics: a hidden dimension in natural language
	rule learning by seven-month-old infants
	a compact statistical model of the song syntax in bengalese finch
	information entropy of humpback whale songs
	simple rules can explain discrimination of putative recursive syntactic structures by a songbird species
	revisiting the syntactic abilities of non-human animals: natural vocalizations and artificial grammar learning
	songs to syntax: the linguistics of birdsong
	language evolution: semantic combinations in primate calls
	are there species-universal categories in bird song phonology and syntax? a comparative study of chaffinches (fringilla coelebs), zebra finches (taenopygia guttata), and swamp sparrows (melospiza georgiana)
	behavioral factors governing song complexity in bengalese finches
1
	adversarially learned inference
	infogan: interpretable representation learning by information maximizing generative adversarial nets
	improved techniques for train[...]<jupyter_text>Get positions<jupyter_code>spring_pos = nx.spring_layout(G, k=1/len(G.nodes())**.25)
kamada_kawai_pos = nx.drawing.layout.kamada_kawai_layout(G)
int(spring_pos['adversarially_learned_inference'][0]*256)
weights = []
for ui, (u, item) in enumerate(G.edges.items()):
    weights.append(item['weight'])
plt.hist(weights, bins = 50);
nodes = [
    {
        "id": str(i),
        "label": str(i).replace('_', " "),
        "group": partition[i],
        "color": "rgba("
        + ",".join([str(int(i * 255)) for i in pal[partition[i]]])
        + ")",
        "x": int(kamada_kawai_pos[i][0] * 256),
        "y": int(kamada_kawai_pos[i][1] * 256),
        "size": 1,
        "type": "goo"
    }
    for ni, i in enumerate(G.nodes())
]
links = [
    {
        "id": str(ui),
        "source": u[0],
        "target": u[1],
        "color": "rgba(0,0,0," + str(round(np.min([item["weight"], 1])/5, 2)) + ")",
    }
    for ui, (u, item) in enumerate(G.edges.items())
]

graph_data = {"nodes": nodes, "edges": links}
with open(DATA_DIR/'post_data.json', 'w') as f:
    json.dump(graph_data,
              f, indent=4,)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13428745432399514, 'fraction_numerical': 0.013093788063337393, 'mean_word_length': 4.6833910034602075, 'pattern_counts': {'":': 14, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '220876', 'license': 'MIT', 'n_tokens_mistral': 1136, 'n_tokens_neox': 1083, 'n_words': 291, 'path': 'notebooks/1.0-make-networkx-graph.ipynb', 'repo_name': 'timsainb/graph_research_notes', 'size': '3284'}