<jupyter_start><jupyter_text>`filenamevar` determines what the filename extension is, for simple version control (consider learning how to do this in git more effectively in the future). | Value  | Meaning                                                                                                          ||--------|------------------------------------------------------------------------------------------------------------------|| [None] | Networks are constructed from subreddit data where nodes are users and edges indicate >1 threads commented in common between the two users within that month.  || excl   | Networks are constructed excluding the posts from subreddit itself.                                              || excl3  | Networks are constructed with the conditions of `excl` with a threads-in-common threshold of 3 to create edges.  |<jupyter_code>filenamevar = 'excl'
# linklist_t = [{} for thing in linklist] #Transpose of the linklist to make it threads by user instead of users by thread
# for i,month in tqdm(enumerate(linklist)):
#     for thread in month:
#         for user in month[thread]:
#             if user not in linklist_t[i]:
#                 linklist_t[i][user] = {}
#             linklist_t[i][user][thread] = month[thread][user]
def networkByMonth(month):
    threadchat = {}
    smallthreadcount = 0
    #Get all the instances where two people chatted in a thread
    for thread in month:
        if len(month[thread].keys())<=1:
            smallthreadcount+=1
            continue
        for x,y in combinations(month[thread].keys(),2):
            if str("%s,%s"%(sorted([x,y])[0],sorted([x,y])[1])) not in threadchat:
                threadchat[str("%s,%s"%(sorted([x,y])[0],sorted([x,y])[1]))] = 0
            threadchat[str("%s,%s"%(sorted([x,y])[0],sorted([x,y])[1]))] += 1
    print("%i of the %i threads had only one response."%(smallthreadcount,len(month)))
    
    g = nx.Graph()
    for key in threadchat:
        if threadchat[key]<=1:
            continue
        x,y = key.split(",")
        g.add_edge(x,y,weight=threadchat[key])
    return g
for i,month in enumerate(linklist):
    g_month = networkByMonth(month)
    with open("EoBoR_%i_%s.gexf"%(i,filenamevar),'wb') as f:
        nx.write_gexf(g_month,f)<jupyter_output>205664 of the 213106 threads had only one response.
255220 of the 263470 threads had only one response.
206807 of the 212469 threads had only one response.
202233 of the 207316 threads had only one response.<jupyter_text>Observations* It looks like the population of original users drop by half, and surprisingly the number of edges drops ten-fold.  * 1280->570 nodes, 11k->1.8k edges. * Before the ban, it looks like there was two major communities, one homogeneous and one mixed, based on modularity class. Not sure why OpenOrd splits into those two groups, given that other methods don't do that. * After the ban, this structure looks similar but slightly broken? Maybe looking at assortativity would help. * Concerns about comparing networks with a completely different population. * A lot of edges occur in the subredddit.   Questions 1* Is there a correspondence between the communities between months? * How much overlap between populations?* Can you track which subreddits people went to?  How much overlap between populations across months?<jupyter_code>networklist = [nx.read_gexf('EoBoR_%i_%s.gexf'%(i,filenamevar)) for i in range(4)]
usersets = [set(x.nodes()) for x in networklist]
print("Of the %i users in Dec16, %i were found in Jan17."%(len(usersets[0]),len(usersets[0]&usersets[1])))
print("Of the %i users in Jan17, %i were found in Feb17."%(len(usersets[1]),len(usersets[1]&usersets[2])))
print("Of the %i users in Feb17, %i were found in Mar17."%(len(usersets[2]),len(usersets[2]&usersets[3])))<jupyter_output>Of the 729 users in Dec16, 526 were found in Jan17.
Of the 836 users in Jan17, 418 were found in Feb17.
Of the 570 users in Feb17, 355 were found in Mar17.<jupyter_text>Is there a correspondence between communities across months?<jupyter_code>partitions = [community.best_partition(x) for x in networklist]
#Top 5 largest communities
top5comms = [sorted(list(enumerate([partitions[i].values().count(comm) for comm in range(max(partitions[i].values()))])), key=lambda x: x[1], reverse = True)[0:5] for i in range(len(partitions))]
top5comms<jupyter_output><empty_output><jupyter_text>So what I want this to do is to give me a the community with the best overlap for each community abovefor each of the largest communities in a given month network:    set of individuals in that community    take each of the other given month networks    for each top community in that network:        create sets of the users in those communities        Take the intersection for each of these with the original community        Report the one that accounts for the most original community users<jupyter_code>def usersInComm(d,comm):
    return set(k for k,v in d.items() if v==comm)

def truncate(f, n):
    '''Truncates/pads a float f to n decimal places without rounding'''
    s = '{}'.format(f)
    if 'e' in s or 'E' in s:
        return '{0:.{1}f}'.format(f, n)
    i, p, d = s.partition('.')
    return '.'.join([i, (d+'0'*n)[:n]])
#Figure out which community has the most overlap
the_dumbest_meta = nx.DiGraph()
monthkey = {0: 'Dec', 1: 'Jan', 2: 'Feb', 3:'Mar'}

for i,month in enumerate(top5comms):
    for comm_num,comm_val in month:
        #print "Month %i, community number %i"%(i,comm_num)
        origin = monthkey[i]+str(comm_num)
        original_community_set = usersInComm(partitions[i],comm_num)
        for x in range(len(top5comms)):
            #print x
            results = []
            for y in range(len(top5comms[x])):
                test_community_set = usersInComm(partitions[x],top5comms[x][y][0])
                diff = float(len(original_community_set & test_community_set))/len(original_community_set)
                if 1>diff>=0.2:
                    #results.append( (top5comms[x][y][0], truncate(diff,2)) )
                    destination = monthkey[x]+str(top5comms[x][y][0])
                    the_dumbest_meta.add_edge(origin,destination,weight=diff)
nx.write_gexf(the_dumbest_meta,'the_dumbest_meta.gexf')<jupyter_output><empty_output><jupyter_text>Questions 2It seems like there's some correspondence between months. List of to-do: * Export modularity class from gephi and look at the cases of communities you see visually. * See the overlap between the community in question (comm-oq). List of questions:1. Is the dense weighted cluster consistent over time?2. Right now, you're comparing the out (posts not in the sub) before and after. This only answers the question "Does removing this subreddit affect other communities?". So you also need to compare the in and see whether the in+out(after_ban) = total(before_ban). 3. ? To defend the idea that the community has changed, you can construct null configuration models before and after the network changes and then excise the same users that are lost across the ban date. Then you can see if the real networks are statistically different from the nulls.<jupyter_code>def gephiCommunityMapper(csv_filename, ntwk):
    df = pd.read_csv(csv_filename)
    mod_partitions = {x:df.get_value(df.loc[df['Label']==x].index[0],'modularity_class') for x in df['Label'].tolist()}
        #df.loc[df['Label']==x].index[0]  <-- This part gets the index of the label (username) that you want
        #df.get_value gets the thing at the row and column, where row has to be indexed row
    nx.set_node_attributes(ntwk,'community',mod_partitions)
    return ntwk
gephimodfiles = ['dec2016_gephimod.csv','jan2017_gephimod.csv','feb2017_gephimod.csv','mar2017_gephimod.csv']
gephinets = []
for x,y in zip(gephimodfiles,networklist):
    gephinets.append( gephiCommunityMapper(x,y) )
comms_in_q = [8,14,14,12]
gephinet_sets = []
for i,thing in enumerate(gephinets):
    gephinet_sets.append( set([x for x,y in nx.get_node_attributes(thing,'community').items() if y==comms_in_q[i]]) )
for x in range(len(gephinet_sets)-1):
    print "Total number within the community for month %i"%x
    print len(gephinet_sets[x])
    print "Overlap with next month"
    print len(gephinet_sets[x] & gephinet_sets[x+1])<jupyter_output>Total number within the community for month 0
125
Overlap with next month
67
Total number within the community for month 1
120
Overlap with next month
41
Total number within the community for month 2
93
Overlap with next month
48<jupyter_text>Findings from previous sectionWhen looking at the really connected part of the networks, 68/122 (55%) of the users are repeats from Dec to Jan, and 41/91 (45%) of the users are carry-overs from Jan to Feb. This is much greater than the overall carryover, so these are probably the same communities. If the community survived the ban, then they must be talking somewhere, so let's see where they are talking.<jupyter_code>max(nx.get_node_attributes(gephinets[0],'community').values())<jupyter_output><empty_output><jupyter_text>Get the threads that communities participated in<jupyter_code>users_in_comms_all = []

for g in gephinets:
    comms_in_net = []
    commvals = nx.get_node_attributes(g,'community')
    for i in range(max(commvals.values())):
        comms_in_net.append(set([x for x,y in commvals.items() if y==i]))
    users_in_comms_all.append(comms_in_net)
#What threads correspond with each community?

threadcomm = []

for i, monthdata in enumerate(users_in_comms_all):
    threadcommmonth = {}
    for ct in tqdm(linklist[i]):
        if len(linklist[i][ct].keys())>1:
            for commnum in range(len(monthdata)):
                if len(set(linklist[i][ct].keys()) & monthdata[commnum])>=2: #If more than two users from that community appeared in the thread
                    if commnum not in threadcommmonth:
                        threadcommmonth[commnum] = set({})
                    threadcommmonth[commnum].add(ct)
    threadcomm.append(threadcommmonth)
threadcomm
#For every file
    #Load each document
    #If the thread matches the one in the record
    #Put it into the aggregator with the label

comm_subreddits = []

#Pull user data based on subreddit
for i,filename in enumerate(os.listdir("reddit_data/reddit_bzs/")):
    print filename
    monthvar = {}
    month = threadcomm[i]
    with bz2.BZ2File("reddit_data/reddit_bzs/"+filename,'rb') as f:
        for line in tqdm(f):
            doc = json.loads(line)
            #Checking every community
            #for month in threadcomm:
            for comm in month:
                if comm not in monthvar:
                    monthvar[comm] = {}
                for threadname in month[comm]:
                    if doc['parent_id']==threadname:
                        if doc['subreddit'] not in monthvar[comm]:
                            monthvar[comm][doc['subreddit']] = 0
                        monthvar[comm][doc['subreddit']] += 1
    comm_subreddits.append(monthvar)
with open("threadcomm.pickle",'wb') as f:
    pickle.dump(threadcomm,f)<jupyter_output><empty_output><jupyter_text>Doing statisticsThe basic idea is that if the structure of the network changed drastically, then it should have very different network properties than a random subgraph of the previous month.<jupyter_code>networklist = [nx.read_gexf('EoBoR_%i_%s.gexf'%(i,filenamevar)) for i in range(4)]
networklist_lens = [len(thing) for thing in networklist]
networklist_lens
def subsample(g,trim_num):
    if len(g)<=trim_num:
        raise ValueError('Subgraph parameters are larger than the original graph.')
    a = g.copy()
    random_nodes = sample(a.nodes(),len(a)-trim_num)
    a.remove_nodes_from(random_nodes)
    return a

def subsetTrials(orig,final,n_trials=100):
    betweenness = []
    average_degree = []
    diameter = []
    connected_components = []
    avg_shortest_path_len = []
    assortativity = []
    for x in tqdm(range(n_trials)):
        norm_g = subsample(orig,len(final))
        #Betweenness
        #betweenness.append(nx.betweenness_centrality(norm_g,weight='weight'))
        #Average degree
        average_degree.append(np.mean(norm_g.degree().values()))
        #Diameter
        giant = max(nx.connected_component_subgraphs(norm_g), key=len)
        diameter.append(nx.diameter(giant))
        #Modularity
        #Connected components
        connected_components.append(nx.number_connected_components(norm_g))
        #Average path length
        avg_shortest_path_len.append(nx.average_shortest_path_length(giant,weight='weight'))
        #Degree assortativity
        edge_weights = nx.get_edge_attributes(norm_g,'weight')
        nx.set_edge_attributes( norm_g, 'weight', {key:int(val) for key,val in edge_weights.items()} )
        assortativity.append(nx.degree_assortativity_coefficient(norm_g,weight='weight'))
    return betweenness, average_degree, diameter, connected_components, avg_shortest_path_len, assortativity
#Distribution test
betweenness, average_degree, diameter, connected_components, avg_shortest_path_len, assortativity = subsetTrials(networklist[1],networklist[2],n_trials=500)
from matplotlib import pyplot
for thing in [average_degree, diameter, connected_components, avg_shortest_path_len, assortativity]:
    plt.hist(thing,bins=30)
    plt.show()
#t test or whatever
import scipy.stats as stats
#Average degree
real_average_degree = np.mean(networklist[2].degree().values())
#Diameter
giant = max(nx.connected_component_subgraphs(networklist[2]), key=len)
real_diameter = nx.diameter(giant)
#Connected components
real_connected_components = nx.number_connected_components(networklist[2])
#Average shortest path length
real_avg_shortest_path_len = nx.average_shortest_path_length(giant,weight='weight')
#Degree assortativity
real_edge_weights = nx.get_edge_attributes(networklist[2],'weight')
nx.set_edge_attributes( networklist[2],'weight', {key:int(val) for key,val in real_edge_weights.items()} )
real_assortativity = nx.degree_assortativity_coefficient(networklist[2],weight='weight')
zscores_average_degree = stats.zscore([real_average_degree] + average_degree)
zscores_diameter = stats.zscore([real_diameter] + diameter)
zscores_connected_components = stats.zscore([real_connected_components] + connected_components)
zscores_avg_path = stats.zscore([real_avg_shortest_path_len] + avg_shortest_path_len)
zscores_assort = stats.zscore([real_assortativity] + assortativity)
print(zscores_average_degree[0])
print(zscores_diameter[0])
print(zscores_connected_components[0])
print(zscores_avg_path[0])
print(zscores_assort[0])
print stats.ks_2samp(average_degree, [real_average_degree])
#print stats.ks_2samp(diameter, [real_diameter]) <--Not valid because discrete
#print stats.ks_2samp(connected_components, [real_connected_components]) <--Not valid because discrete
print stats.ks_2samp(avg_shortest_path_len, [real_avg_shortest_path_len])
print stats.ks_2samp(assortativity, [real_assortativity])
print average_degree, diameter, connected_components, avg_shortest_path_len, assortativity<jupyter_output>[7.0877192982456139, 7.8842105263157896, 6.7473684210526317, 7.3473684210526313, 6.6385964912280704, 7.4666666666666668, 7.9228070175438594, 6.1614035087719294, 6.9298245614035086, 6.9368421052631577, 6.5999999999999996, 5.6491228070175437, 7.2842105263157899, 6.4912280701754383, 7.5157894736842108, 6.5228070175438599, 6.9894736842105267, 6.1824561403508769, 6.4982456140350875, 6.7719298245614032, 6.807017543859649, 7.5368421052631582, 6.5508771929824565, 6.2842105263157899, 6.5052631578947366, 6.7087719298245618, 7.5508771929824565, 6.4982456140350875, 6.9684210526315793, 6.4736842105263159, 6.5649122807017548, 7.9368421052631577, 7.2035087719298243, 6.6491228070175437, 6.9403508771929827, 6.1052631578947372, 6.3017543859649123, 6.5719298245614031, 7.098245614035088, 7.4000000000000004, 6.1087719298245613, 7.5228070175438599, 7.4947368421052634, 7.5649122807017548, 7.3157894736842106, 6.5578947368421057, 6.9649122807017543, 6.7789473684210524, 7.0105263157894733, 7.5333333333333332, 7[...]
{'fraction_non_alphanumeric': 0.11027631170443962, 'fraction_numerical': 0.06817758460105558, 'mean_word_length': 4.100063331222293, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '225994', 'license': 'MIT', 'n_tokens_mistral': 5794, 'n_tokens_neox': 5039, 'n_words': 1414, 'path': 'Creating networks (copy).ipynb', 'repo_name': 'vincentmwong/EoBoR', 'size': '16105'}