<jupyter_start><jupyter_text>Community Detection Example  Implement from this [site](https://python-louvain.readthedocs.io/en/latest/)Beware of install using `pip install python-louvain`  Key Concepts:Maximum the modularity1. ModularityFor the disjoint sets, modularity propotional to the difference between ( of edegs in the group and the expected  edges inside the group)2. Use Null model to get the expected  edges inside the group [Source from Stanford CS224W 2019 Fall](http://snap.stanford.edu/class/cs224w-2019/slides/04-communities.pdf) Forom this example, the `karate_club_graph` from networkx is used as an graph example to demostarte the Louvain community detection method.1. First load the `karate_club_graph`2. Use `best_partition()` to find the best partition of the graph3. Draw the graph using networkx's `draw_network_nodes` and `draw_network_edges`<jupyter_code>import community as community_louvain
import matplotlib.cm as cm 
import matplotlib.pyplot as plt 
import networkx as nx  
# load the karate club graph
G = nx.karate_club_graph()

# first compute the best partition
partition = community_louvain.best_partition(G)
# draw the graph
pos = nx.spring_layout(G)

# color the nodes based on the partitions
cmap = cm.get_cmap('viridis', max(partition.values())+1)
nx.draw_networkx_nodes(G, pos, partition.keys(), node_size=80,cmap=cmap, node_color=list(partition.values()))

nx.draw_networkx_edges(G, pos, alpha=0.5)
plt.show()


partition.keys()
partition.values()
G.nodes
G.edges<jupyter_output><empty_output><jupyter_text>Implement American Colledge Football Community<jupyter_code>import urllib.request as urllib
import io
import zipfile

import matplotlib.pyplot as plt
import networkx as nx

url = "http://www-personal.umich.edu/~mejn/netdata/football.zip"

sock = urllib.urlopen(url)  # open URL
s = io.BytesIO(sock.read())  # read into BytesIO "file"
sock.close()

zf = zipfile.ZipFile(s)  # zipfile object
txt = zf.read("football.txt").decode()  # read info file
gml = zf.read("football.gml").decode()  # read gml data
# throw away bogus first line with # from mejn files
gml = gml.split("\n")[1:]
G_ball = nx.parse_gml(gml)  # parse gml data
print(txt)
for n, d in G_ball.degree():
    print(f"{n:20} {d:2}")

options = {
    "node_color": "black",
    "node_size": 50,
    "linewidths": 0,
    "width": 0.1,
}
nx.draw(G_ball, **options)
plt.show()<jupyter_output>BrighamYoung         12
FloridaState         12
Iowa                 12
KansasState          12
NewMexico            11
TexasTech            12
PennState            12
SouthernCalifornia   12
ArizonaState         11
SanDiegoState        11
Baylor               10
NorthTexas           10
NorthernIllinois     10
Northwestern         11
WesternMichigan      10
Wisconsin            12
Wyoming              11
Auburn               11
Akron                11
VirginiaTech         11
Alabama              11
UCLA                 11
Arizona              11
Utah                 11
ArkansasState        10
NorthCarolinaState   11
BallState            10
Florida              11
BoiseState            9
BostonCollege        11
WestVirginia         11
BowlingGreenState    11
Michigan             11
Virginia             10
Buffalo              11
Syracuse             11
CentralFlorida        8
GeorgiaTech          11
CentralMichigan      11
Purdue               11
Colorado             11
ColoradoState   [...]<jupyter_text>Implement commnity detection<jupyter_code>partition = community_louvain.best_partition(G_ball)
# draw the graph
pos = nx.spring_layout(G_ball)

# color the nodes based on the partitions
cmap = cm.get_cmap('viridis', max(partition.values())+1)
nx.draw_networkx_nodes(G_ball, pos, partition.keys(), node_size=40,cmap=cmap, node_color=list(partition.values()))

nx.draw_networkx_edges(G_ball, pos, alpha=0.5)
plt.show()
set(partition.values())
from collections import defaultdict
group_dict = defaultdict(list)
for k,v in partition.items():
    group_dict[v].append(k)
group_dict<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08679432976871425, 'fraction_numerical': 0.029843322556577966, 'mean_word_length': 3.2651113467656416, 'pattern_counts': {'":': 4, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '659122', 'license': 'MIT', 'n_tokens_mistral': 1409, 'n_tokens_neox': 1279, 'n_words': 360, 'path': 'ML_theory_practices/community_dectect_example.ipynb', 'repo_name': 'tahe0203/JobPrep', 'size': '4021'}