<jupyter_start><jupyter_text>Dynamic Network Graph Exploration  Explore interactive design and parameters combo to better detect adn define interactions between members. Code credits - <NAME> based on the original code provided by <NAME>  Notebook Requirements<jupyter_code>import os, sys
import logging
import gzip

import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import networkx as nx

# Import the data analysis tools
import openbadge_analysis as ob
import openbadge_analysis.preprocessing
import openbadge_analysis.core
# import pyvis

SELECTED_BEACON = 12
ob.__version__<jupyter_output><empty_output><jupyter_text>SettingsThe time_zone variable will be used when converting the timestamp form UTC time to your local time.<jupyter_code>time_zone = 'US/Eastern'
log_version = '2.0'
time_bins_size = '1min'

proximity_data_filenames = []

for i in range(1, 18):
    if i < 10:
        filename = 'CTSIserver{:02d}_proximity_2019-06-01.txt'.format(i)
    else:
        filename = 'CTSIserver{}_proximity_2019-06-01.txt'.format(i)
        
    proximity_data_filenames.append(filename)
    
attendees_metadata_filename = "Badge assignments_Attendees_2019.xlsx"
members_metadata_filename = "Member-2019-05-28.csv"
beacons_metadata_filename = "location table.xlsx"
data_dir = "./proximity_2019-06-01/"<jupyter_output><empty_output><jupyter_text>Pre-processing First, we load two lists that will help us with some of the analysis: list of membmers and list of location beacons<jupyter_code>members_metadata = pd.read_csv(data_dir+members_metadata_filename)
beacons_metadata = pd.read_excel(data_dir+beacons_metadata_filename, sheet_name='Sheet1')

# Calculating the id of the beacon based on it's MAC address
# beacons_metadata['id'] = beacons_metadata.apply(
#     lambda row: ob.core.mac_address_to_id(row['badge_address']),
#     axis=1
# )
# read member background data, merge to the original member metadata to keep consistent
background = pd.read_excel(data_dir+'Badge assignments_Attendees_2019.xlsx')
cleaned_members = members_metadata.merge(background, how='inner')

# separate out the member key strings of people who actually attended the meeting and those who did not
attendees_key = set(cleaned_members['member'])
all_people_key = set(members_metadata['member'])
attendees_id = set(cleaned_members['id'])
all_people_id = set(members_metadata['id'])

non_attendees_key = all_people_key-attendees_key
non_attendees_id = all_people_id-attendees_id<jupyter_output><empty_output><jupyter_text>beacons_metadata We create a translation table between the badge ID and member key. This is done based on the data itself, since it should contain data from all the badges that take part in the study. Note that we create a  pair for ever time bin. While this is not necessary at this point, it allows this mapping to change (for example, if a badge is re-assigned to a different member).<jupyter_code>idmaps = []

for proximity_data_filename in proximity_data_filenames:
    with open(os.path.join(data_dir, proximity_data_filename), 'r') as f:
        idmaps.append(ob.preprocessing.id_to_member_mapping(f, time_bins_size, tz=time_zone))
        
tmp_idmaps = idmaps[0]
for i in range(1, len(idmaps)):
    tmp_idmaps = pd.concat([tmp_idmaps, idmaps[i]])
tmp_idmaps.shape<jupyter_output><empty_output><jupyter_text>Using this translation table and the proximity data, we can create a list of "pings" - every time two badges were in close proximity<jupyter_code>m2badges = []

for proximity_data_filename in proximity_data_filenames:
    with open(os.path.join(data_dir, proximity_data_filename), 'r') as f:
        m2badges.append(ob.preprocessing.member_to_badge_proximity(f, time_bins_size, tz=time_zone))
        
tmp_m2badges = m2badges[0]

for i in range(1, len(m2badges)):
    tmp_m2badges = pd.concat([tmp_m2badges, m2badges[i]])
tmp_m2badges.shape
cleaned_m2badges = []

for m2badge in m2badges:
    drop_list = []
    tmp = m2badge.reset_index()

    for index, row in tmp.iterrows():
        if row['member'] in non_attendees_key:
            drop_list.append(index)
        else:
            if row['observed_id'] in non_attendees_id:
                drop_list.append(index)
    tmp = tmp.drop(drop_list)
    cleaned_m2badges.append(tmp)<jupyter_output><empty_output><jupyter_text>Since a badge can either be a badge worn by a participant, or a location beacon, we split the dataset into member-to-member (for network graphs) and member-to-beacon (for localization)<jupyter_code># Member to member
m2ms = []
for (m2badge, idmap) in zip(cleaned_m2badges, idmaps):
    m2ms.append(ob.preprocessing.member_to_member_proximity(m2badge, idmap))

tmp_m2ms = m2ms[0]
for i in range(1, len(m2ms)):
    tmp_m2ms = pd.concat([tmp_m2ms, m2ms[i]])
tmp_m2ms.shape
# Member to location beacon
m2bs = []
for m2badge in cleaned_m2badges:
    m2bs.append(ob.preprocessing.member_to_beacon_proximity(m2badge, beacons_metadata.set_index('id')['beacon']))
    
tmp_m2bs = m2bs[0]
for i in range(1, len(m2bs)):
    tmp_m2bs = pd.concat([tmp_m2bs, m2bs[i]])<jupyter_output><empty_output><jupyter_text>Network Graph Preparation Run every block in under this title to prepare for all Dynamic Network Member Graph analysis<jupyter_code># create time slices
def generate_time_slices(start_h, start_m, end_h, end_m, interval=2):
    time_slices = []
    tmp_time = '2019-06-01 {:02}:{:02}'.format(start_h, start_m)
    duration = (end_h - start_h) * 60 + (end_m - start_m)
    for i in range(duration/interval+1):
        if start_m < 60-interval+1:
            start_m += interval-1
        else:
            start_h += 1
            start_m = start_m - 60 + interval
                
        if start_h>=end_h and start_m>=end_m:
            time_slices.append(slice(tmp_time,'2019-06-01 {:02}:{:02}'.format(end_h, end_m)))
            break
        
        tmp_time = '2019-06-01 {:02}:{:02}'.format(start_h, start_m)
        time_slices.append(slice(start, tmp_time))
        
        if start_m >= 59:
            start = '2019-06-01 {:02}:{:02}'.format(start_h+1, 0)
        else:
            start_m += 1
            start = '2019-06-01 {:02}:{:02}'.format(start_h, start_m)
        
    return time_slices
# generate_time_slices(9,50,11,30, 1)
# create time slices
def generate_time_slices(start_h, start_m, end_h, end_m, interval=2):
    
    time_slices = []
    
    start = '2019-06-01 {:02}:{:02}'.format(start_h, start_m)
    duration = (end_h - start_h) * 60 + (end_m - start_m)
    
    for i in range(duration/interval+1):
        
        if start_h > end_h:
            break
        elif start_h == end_h:
            if start_m > end_m:
                break
            
        tmp_h = start_h
        tmp_m = start_m
        
        if tmp_m < 60-interval+1:
            tmp_m += interval-1
        else:
            tmp_h += 1
            if interval > 1:
                tmp_m = tmp_m - 60 + interval-1
            else:
                tmp_m = tmp_m - 60 + interval
            
        tmp_time = '2019-06-01 {:02}:{:02}'.format(tmp_h, tmp_m)
        
        time_slices.append(slice(start, tmp_time))
        
        start_h = tmp_h
        start_m = tmp_m
        
        if start_m == 59:
            start_h += 1
            start_m = 0
        else:
            start_m += 1
        
        start = '2019-06-01 {:02}:{:02}'.format(start_h, start_m)
    return time_slices

# create time slices
def generate_time_points(start_h, start_m, end_h, end_m, interval=2):
    time_points = []
    tmp_time = '2019-06-01 {:02}:{:02}'.format(start_h, start_m)
    duration = (end_h - start_h) * 60 + (end_m - start_m)
    start_m=start_m-1
    for i in range(duration/interval+1):
        if start_m < 60-interval:
            start_m += interval
        else:
            start_h += 1
            start_m = start_m - 60 + interval
        tmp_time = '2019-06-01 {:02}:{:02}'.format(start_h, start_m)
        if start_h>=end_h and start_m>=end_m:
            tmp_time = '2019-06-01 {:02}:{:02}'.format(end_h, end_m)
            time_points.append(tmp_time)
            break
        time_points.append(tmp_time)
        
    return time_points

# generate_time_points(9,50,11,30, 1)
tmp_m2ms_sorted = tmp_m2ms.sort_index(0,0)
def draw_graph(G, graph_layout='shell',
               node_size=200, node_color='blue', node_alpha=0.3,
               node_text_size=8,
               edge_color='blue', edge_alpha=0.3, edge_tickness=1,
               edge_text_pos=0.3,
               text_font='sans-serif'):

    # these are different layouts for the network you may try
    # shell seems to work best
    if graph_layout == 'spring':
        graph_pos=nx.spring_layout(G)
    elif graph_layout == 'spectral':
        graph_pos=nx.spectral_layout(G)
    elif graph_layout == 'random':
        graph_pos=nx.random_layout(G)
    else:
        graph_pos=nx.shell_layout(G)

    # draw graph
    nx.draw_networkx_nodes(G,graph_pos,node_size=node_size, 
                           alpha=node_alpha, node_color=node_color, cmap=plt.get_cmap('jet'))
    nx.draw_networkx_edges(G,graph_pos,width=edge_tickness,
                           alpha=edge_alpha,edge_color=edge_color)
    nx.draw_networkx_labels(G, graph_pos,font_size=node_text_size,
                            font_family=text_font)

    plt.show()<jupyter_output><empty_output><jupyter_text>Network Graph Basic ExampleThis trunk is only for demonstration, not for analysis.<jupyter_code># Filter data from specific time period

time_slice = slice('2019-06-01 10:00', '2019-06-01 11:20')
m2m_breakout = tmp_m2ms_sorted.loc[time_slice]

# keep only instances with strong signal
m2m_filter_rssi = m2m_breakout[m2m_breakout.rssi >= -70].copy()
print(len(m2m_filter_rssi))
# Count number of time members were in close proximity
# We name the count column "weight" so that networkx will use it as weight for the spring layout
m2m_edges = m2m_filter_rssi.groupby(['member1', 'member2'])[['rssi_weighted_mean']].count().rename(columns={'rssi_weighted_mean':'weight'})
m2m_edges = m2m_edges[["weight"]].reset_index()

# Keep strongest edges (threshold set manually)
m2m_edges = m2m_edges[m2m_edges.weight > 15]
print(len(m2m_edges))

# Create a graph
graph=nx.from_pandas_edgelist(m2m_edges, "member1", "member2", "weight")
fig = plt.figure(figsize=(12, 10), dpi=150)
ax = plt.subplot(1,1,1)

draw_graph(graph, graph_layout="spring")
#Create an dragable interface of nodes
from pyvis.network import Network
#try time slice iteratively
# Filter data from specific time period
time_slices=[slice('2019-06-01 9:50', '2019-06-01 10:00'),slice('2019-06-01 10:00', '2019-06-01 10:10'),
             slice('2019-06-01 10:10', '2019-06-01 10:20'),slice('2019-06-01 10:20', '2019-06-01 10:30'),
             slice('2019-06-01 10:30', '2019-06-01 10:40'),slice('2019-06-01 10:40', '2019-06-01 10:50'),
             slice('2019-06-01 10:50', '2019-06-01 11:00'),slice('2019-06-01 11:00', '2019-06-01 11:10'),
             slice('2019-06-01 11:10', '2019-06-01 11:20')]
for i in range(1,10):
    time_slice = time_slices[i-1]
    m2m_breakout = tmp_m2ms_sorted.loc[time_slice]
    # keep only instances with strong signal
    m2m_filter_rssi = m2m_breakout[m2m_breakout.rssi >= -70].copy()
    print(len(m2m_filter_rssi))
    # Count number of time members were in close proximity
    # We name the count column "weight" so that networkx will use it as weight for the spring layout
    m2m_edges = m2m_filter_rssi.groupby(['member1', 'member2'])[['rssi_weighted_mean']
                                                               ].count().rename(columns={'rssi_weighted_mean':'weight'})
    m2m_edges = m2m_edges[["weight"]].reset_index()
    # Keep strongest edges (threshold set manually)
    m2m_edges = m2m_edges[m2m_edges.weight > 5]
    print(len(m2m_edges))
    # Create a graph
    graph=nx.from_pandas_edgelist(m2m_edges, "member1", "member2", "weight")
#     fig = plt.figure(figsize=(12,90), dpi=150)
#     ax = plt.subplot(10,1,i)
#     draw_graph(graph, graph_layout="spring",node_size=40)
    net = Network(height="750px", width="100%", bgcolor="#222222", font_color="white")
    net.from_nx(graph) 
#     net.show_buttons()
    net.set_options('''
    var options = {
  "nodes": {
    "size": 12
  },
  "edges": {
    "color": {
      "inherit": true
    },
    "smooth": false
  },
  "interaction": {
    "hover": true,
    "keyboard": {
      "enabled": true
    },
    "navigationButtons": true,
    "tooltipDelay": 1000
  },
  "manipulation": {
    "enabled": true,
    "initiallyActive": true
  },
  "physics": {
    "minVelocity": 0.75
  }
}
    ''')
    net.show("Interactivenetworkx{}.html".format(i))
   
#plt.show()<jupyter_output><empty_output><jupyter_text>Lunch Time AnalysisUse data from lunch time to find the definition of close interaction This part will create a html interactive interface to identify the interactions threshold<jupyter_code># try time slice iteratively
# Filter data from specific time period
time_slices=[slice('2019-06-01 11:30', '2019-06-01 11:35'),slice('2019-06-01 11:35', '2019-06-01 11:40'),
             slice('2019-06-01 11:40', '2019-06-01 11:45'),slice('2019-06-01 11:45', '2019-06-01 11:50'),
             slice('2019-06-01 11:50', '2019-06-01 11:55'),slice('2019-06-01 11:55', '2019-06-01 12:00'),
            slice('2019-06-01 12:00', '2019-06-01 12:05'),slice('2019-06-01 12:05', '2019-06-01 12:10'),
            slice('2019-06-01 12:10', '2019-06-01 12:15'),slice('2019-06-01 12:15', '2019-06-01 12:20'),]
for i in range(1,11):
    time_slice = time_slices[i-1]
    m2m_breakout = tmp_m2ms_sorted.loc[time_slice]
    # keep only instances with strong signal
    m2m_filter_rssi = m2m_breakout[m2m_breakout.rssi >= -75].copy()
    # Count number of time members were in close proximity
    # We name the count column "weight" so that networkx will use it as weight for the spring layout
    m2m_edges = m2m_filter_rssi.groupby(['member1', 'member2'])[['rssi_weighted_mean']
                                                               ].count().rename(columns={'rssi_weighted_mean':'weight'})
    m2m_edges = m2m_edges[["weight"]].reset_index()
    # Keep strongest edges (threshold set manually)
    m2m_edges = m2m_edges[m2m_edges.weight > 8]
    # Create a graph
    graph=nx.from_pandas_edgelist(m2m_edges, "member1", "member2", "weight")
    fig = plt.figure(figsize=(12,90), dpi=150)
    ax = plt.subplot(10,1,i)
    draw_graph(graph, graph_layout="spring",node_size=200)

plt.show()<jupyter_output><empty_output><jupyter_text>Break-out Session AnalysisUse data from lunch time to find the definition of close interaction<jupyter_code>#try time slice iteratively
# Filter data from specific time period
time_slices=generate_time_slices(9,50,11,20,interval=2)

# time slice creation


for i in range(1,46):
    time_slice = time_slices[i-1]
    m2m_breakout = tmp_m2ms_sorted.loc[time_slice]
    # keep only instances with strong signal
    m2m_filter_rssi = m2m_breakout[m2m_breakout.rssi >= -73].copy()
    print(len(m2m_filter_rssi))
    # Count number of time members were in close proximity
    # We name the count column "weight" so that networkx will use it as weight for the spring layout
    m2m_edges = m2m_filter_rssi.groupby(['member1', 'member2'])[['rssi_weighted_mean']
                                                               ].count().rename(columns={'rssi_weighted_mean':'weight'})
    m2m_edges = m2m_edges[["weight"]].reset_index()
    # Keep strongest edges (threshold set manually)
    m2m_edges = m2m_edges[m2m_edges.weight > 1]
    print(len(m2m_edges))
    # Create a graph
    graph=nx.from_pandas_edgelist(m2m_edges, "member1", "member2", "weight")
    fig = plt.figure(figsize=(12,400), dpi=150)
    ax = plt.subplot(45,1,i)
    draw_graph(graph, graph_layout="spring",node_size=200)<jupyter_output><empty_output><jupyter_text>Interaction Network GraphThis tool helps find interaction between members in a certain amount of time with designated parameters, such as signal strength using the distribution of signal strength. This graph draws multiple pictures so that less detailed are lost in generalization. To choose a threshold for signal strength, the program analyzes the distribution of frequencies of signal strength and it will take the frequency of the peak -2. -2 for leave some room for fluctuation.<jupyter_code>time_interval_start_h=9
time_interval_start_m=50
time_interval_end_h=11
time_interval_end_m=20
interval=2
t_count_threshold = 2


bo1 = generate_time_points(time_interval_start_h, time_interval_start_m, 
                           time_interval_end_h, time_interval_end_m, interval)
freq_list_1 = []
for i in bo1:
    freq_list_1.append(tmp_m2ms.reset_index().set_index('datetime').loc[i])

hist_list_1 = []
for freq in freq_list_1:
    tmp_freq = []
    for row in freq.iterrows():
        tmp = [row[1][3]]*int(row[1][5])
        tmp_freq = tmp_freq + tmp
    hist_list_1.append(tmp_freq)

# print(len(hist_list_1))

vals = {}
for i in range(len(hist_list_1)): 
    for j in hist_list_1[i]:
        if j not in vals.keys():
            vals[j]=1; 
        else: 
            vals[j]=vals[j]+1

# freq_count = dict(zip(his))
# print(vals)


import copy
vals_sorted = copy.deepcopy(sorted(vals.items(), key = 
             lambda vals:(vals[1], vals[0]),reverse = True))
sign_threshold = vals_sorted[1][0]
print(sign_threshold)
# print(vals_sorted)
time_slices=generate_time_slices(time_interval_start_h,time_interval_start_m,time_interval_end_h,
                                 time_interval_end_m,interval)

for i in range(1,len(time_slices)+1):
    time_slice = time_slices[i-1]
    m2m_breakout = tmp_m2ms_sorted.loc[time_slice]
    # keep only instances with strong signal
    m2m_filter_rssi = m2m_breakout[m2m_breakout.rssi >= sign_threshold].copy()
    # print(len(m2m_filter_rssi))
    # Count number of time members were in close proximity
    # We name the count column "weight" so that networkx will use it as weight for the spring layout
    m2m_edges = m2m_filter_rssi.groupby(['member1', 'member2'])[['rssi_weighted_mean']
                                                               ].count().rename(columns={'rssi_weighted_mean':'weight'})
    m2m_edges = m2m_edges[["weight"]].reset_index()
    # Keep strongest edges (threshold set manually)
    m2m_edges = m2m_edges[m2m_edges.weight > interval/2]
    if i == 1: 
        m2m_edges_count = m2m_edges.copy().assign(t_count = [1 for x in range(1,len(m2m_edges['member1'])+1)])
        m2m_edges_count = m2m_edges_count.assign(appeared = [False for x in range(1,len(m2m_edges_count)+1)])
    else: 
        for h in range(1,len(m2m_edges['member1'])):        
            for j in range(1,len(m2m_edges_count['member1'])):
                if m2m_edges.iloc[h,0]==m2m_edges_count.iloc[j,0] and m2m_edges.iloc[h,1]==m2m_edges_count.iloc[j,1]: 
                    m2m_edges_count.iloc[j,3]=m2m_edges_count.iloc[j,3]+1 
                    m2m_edges_count.iloc[j,4]=True
                elif j==len(m2m_edges_count['member1']):
                    m2m_edges_count.append({'member1':m2m_edges.iloc[h,0],'member2':m2m_edges.iloc[h,1],
                                            'weight':m2m_edges.iloc[h,2],'t_count':m2m_edges.iloc[h,3],
                                           'appeared':True})
        for j in m2m_edges_count.index.values:            
            if m2m_edges_count.loc[j,'appeared']==False:
                #print(j)
                if m2m_edges_count.loc[j,'t_count']<t_count_threshold: 
                    m2m_edges_count.drop(j,inplace=True)
                #for x in range(1,len(m2m_edges_count)):
                    #print m2m_edges_count.iloc[x:x+1]
            else: 
                m2m_edges_count.loc[j,'appeared']=False
for x in range(1,len(m2m_edges_count)):
    print m2m_edges_count.iloc[x:x+1]

        

# plot the graph with filtered interactions 
graph=nx.from_pandas_edgelist(m2m_edges_count, "member1", "member2", "t_count")
fig = plt.figure(figsize=(12,10), dpi=120)
draw_graph(graph, graph_layout="spring")
plt.show()
# use interactive network graph to clearly see who is involved in what interactions 
from pyvis.network import Network
net = Network(height="750px", width="100%", bgcolor="#222222", font_color="white")
net.from_nx(graph) 
net.set_options('''
    var options = {
  "nodes": {
    "size": 12
  },
  "edges": {
    "color": {
      "inherit": true
    },
    "smooth": false
  },
  "interaction": {
    "hover": true,
    "keyboard": {
      "enabled": true
    },
    "navigationButtons": true,
    "tooltipDelay": 1000
  },
  "manipulation": {
    "enabled": true,
    "initiallyActive": true
  },
  "physics": {
    "minVelocity": 0.75
  }
}
    ''')
net.show("Interactivenetworkx.html")
   
# incorporate background information into the dynamic graph

# read member background data  
attendees_metadata_filename = "Badge assignments_Attendees_2019.xlsx"
attendees_metadata = pd.read_excel(data_dir+attendees_metadata_filename)
background = pd.DataFrame(columns=['name','badge','background','affiliation'])
background_affiliation = pd.DataFrame(columns=['name','badge','background','affiliation'])
members_metadata = pd.read_csv(data_dir+members_metadata_filename)
background['name'] = members_metadata['member']
background['badge'] = members_metadata['BADGE IP']
for i in background['badge']:
    if i in attendees_metadata['BADGE IP'].values:
        a = background.loc[background['badge'] == i]
        b = attendees_metadata.loc[attendees_metadata['BADGE IP']==i]
        a['background'] = b['Cleaned Primary discipline/field of interest'].values
        a['affiliation'] = b['Affiliation'].values
        background_affiliation = pd.concat([background_affiliation, a])
    else:
        a = background.loc[background['badge']==i]
        background_affiliation = pd.concat([background_affiliation, a])
        

#create a dictionary for member id and background info
bg_dict = {}
for i in range(0,len(background_affiliation)):
    bg_dict.update({background_affiliation.loc[i,'name']: str(background_affiliation.loc[i,'background'])+", "+
                                                          str(background_affiliation.loc[i,'affiliation'])+", "+
                                                          str(i)})
    
#relabel the nodes with the background infomation
graph_bg = nx.relabel_nodes(graph,bg_dict)

# create the graph with filtered interactions 
fig = plt.figure(figsize=(12,10), dpi=150)
draw_graph(graph_bg, graph_layout="spring",node_size=60)
plt.show()<jupyter_output><empty_output><jupyter_text>Interactive Interaction Finding Tool This tool helps find interaction between members in a certain amount of time with designated parameters, such as signal strength using the distribution of signal strength. These parameters can bu customized. To automatively choose a threshold for signal strength, the program analyzes the distribution of frequencies of signal strength and it will take the frequency of the peak -2. -2 for leave some room for fluctuation.<jupyter_code>from ipywidgets import GridspecLayout,Layout,IntSlider,Button
from numpy import std
# import bqplot as bq


start_hr = IntSlider(min=7, max=14, value=9, description="Start hour: ",
                         layout=Layout(width='auto', height='auto')
                    )
start_min = IntSlider(min=0, max=59, value=50, description="Start min: ",
                         layout=Layout(width='auto', height='auto')
                     )
end_hr = IntSlider(min=7, max=14, value=11, description="End hour: ",
                         layout=Layout(width='auto', height='auto')
                  )
end_min = IntSlider(min=0, max=59, value=30,description="End min: ",
                         layout=Layout(width='auto', height='auto')
                   )
interval_threshold = IntSlider(min=2, max=10, value=2,description="Time interval: ",
                         layout=Layout(width='auto', height='auto')
                    )
time_count_threshold = IntSlider(min=0, max=90, value=2,description="Time count threshold: ",
                         layout=Layout(width='auto', height='auto')
                    )
app = GridspecLayout(3, 3, height='100px')

app[0,0]=start_hr
app[0,1]=start_min
app[0,2]=end_hr
app[1,0]=end_min
app[1,1]=interval_threshold
app[1,2]=time_count_threshold



def start_hr_change(change):
    time_interval_start_h=change['new']
#     print(time_interval_start_h)
    
def start_min_change(change):
    time_interval_start_h=change['new']
#     print(time_interval_start_m)
    
def end_hr_change(change):
    time_interval_start_h=change['new']
#     print(time_interval_end_h)
    
def end_min_change(change):
    time_interval_start_h=change['new']
#     print(time_interval_end_m)

def interval_threshold_change(change):
    interval=change['new']
    print(interval)
    
def time_count_threshold_change(change):
    t_count_threshold=change['new']
#     print(t_count_threshold)

start_hr.observe(start_hr_change, names='value')
start_min.observe(start_min_change, names='value')
end_hr.observe(end_hr_change, names='value')
end_min.observe(end_min_change, names='value')
interval_threshold.observe(interval_threshold_change, names='value')
time_count_threshold.observe(time_count_threshold_change, names='value')


draw_graph_button=Button(description='Draw graph!', button_style='info', layout=Layout(height='auto', width='auto'))
app[2,:]=draw_graph_button

display(app)



def draw_graph_button_click_handler(btn_object):
    print('You pressed the {} button!'.format(btn_object.description))
    time_interval_start_h=start_hr.value
    time_interval_start_m=start_min.value
    time_interval_end_h=end_hr.value
    time_interval_end_m=end_min.value
    interval=interval_threshold.value
    t_count_threshold=time_count_threshold.value
    bo1 = generate_time_points(time_interval_start_h, time_interval_start_m, 
                           time_interval_end_h, time_interval_end_m, interval)
    freq_list_1 = []
    for i in bo1:
        freq_list_1.append(tmp_m2ms.reset_index().set_index('datetime').loc[i])

    hist_list_1 = []
    for freq in freq_list_1:
        tmp_freq = []
        for row in freq.iterrows():
            tmp = [row[1][3]]*int(row[1][5])
            tmp_freq = tmp_freq + tmp
        hist_list_1.append(tmp_freq)

    # print(len(hist_list_1))

    vals = {}
    for i in range(len(hist_list_1)): 
        for j in hist_list_1[i]:
            if j not in vals.keys():
                vals[j]=1; 
            else: 
                vals[j]=vals[j]+1
      
    
    import copy
    vals_sorted = copy.deepcopy(sorted(vals.items(), key = 
                 lambda vals:(vals[1], vals[0]),reverse = True))
    sign_threshold = vals_sorted[2][0]
    print('The threshold we find from distribution is '+ str(sign_threshold))
    # print(vals_sorted)
    

    time_slices=generate_time_slices(time_interval_start_h,time_interval_start_m,time_interval_end_h,
                                     time_interval_end_m,interval)
    
    for i in range(1,len(time_slices)+1):
#         print(i)
        time_slice = time_slices[i-1]
        m2m_breakout = tmp_m2ms_sorted.loc[time_slice]
        # keep only instances with strong signal
        m2m_filter_rssi = m2m_breakout[m2m_breakout.rssi >= sign_threshold].copy()
        # print(len(m2m_filter_rssi))
        # Count number of time members were in close proximity
        # We name the count column "weight" so that networkx will use it as weight for the spring layout
        m2m_edges = m2m_filter_rssi.groupby(['member1', 'member2'])[['rssi_weighted_mean']
                                                                   ].count().rename(columns={'rssi_weighted_mean':'weight'})
        m2m_edges = m2m_edges[["weight"]].reset_index()
        # Keep strongest edges (threshold set manually)
        m2m_edges = m2m_edges[m2m_edges.weight > interval/2]
        if i == 1: 
            m2m_edges_count = m2m_edges.copy().assign(t_count = [1 for x in range(1,len(m2m_edges['member1'])+1)])
            m2m_edges_count = m2m_edges_count.assign(appeared = [False for x in range(1,len(m2m_edges_count)+1)])
        else: 
            for h in range(1,len(m2m_edges['member1'])):        
                for j in range(1,len(m2m_edges_count['member1'])):
                    if m2m_edges.iloc[h,0]==m2m_edges_count.iloc[j,0] and m2m_edges.iloc[h,1]==m2m_edges_count.iloc[j,1]: 
                        m2m_edges_count.iloc[j,3]=m2m_edges_count.iloc[j,3]+1 
                        m2m_edges_count.iloc[j,4]=True
                    elif j==len(m2m_edges_count['member1']):
                        m2m_edges_count.append({'member1':m2m_edges.iloc[h,0],'member2':m2m_edges.iloc[h,1],
                                                'weight':m2m_edges.iloc[h,2],'t_count':m2m_edges.iloc[h,3],
                                               'appeared':True})
            for j in m2m_edges_count.index.values:            
                if m2m_edges_count.loc[j,'appeared']==False:
                    #print(j)
                    if m2m_edges_count.loc[j,'t_count']<t_count_threshold: 
                        m2m_edges_count.drop(j,inplace=True)
                    #for x in range(1,len(m2m_edges_count)):
                        #print m2m_edges_count.iloc[x:x+1]
                else: 
                    m2m_edges_count.loc[j,'appeared']=False
#         for x in range(1,len(m2m_edges_count)):
#             print m2m_edges_count.iloc[x:x+1]

    
    graph=nx.from_pandas_edgelist(m2m_edges_count, "member1", "member2", "t_count")
    

    # incorporate background information into the dynamic graph

    # read member background data  
    attendees_metadata_filename = "Badge assignments_Attendees_2019.xlsx"
    attendees_metadata = pd.read_excel(data_dir+attendees_metadata_filename)
    background = pd.DataFrame(columns=['name','badge','background','affiliation'])
    background_affiliation = pd.DataFrame(columns=['name','badge','background','affiliation'])
    members_metadata = pd.read_csv(data_dir+members_metadata_filename)
    background['name'] = members_metadata['member']
    background['badge'] = members_metadata['BADGE IP']
    for i in background['badge']:
        if i in attendees_metadata['BADGE IP'].values:
            a = background.loc[background['badge'] == i]
            b = attendees_metadata.loc[attendees_metadata['BADGE IP']==i]
            a['background'] = b['Cleaned Primary discipline/field of interest'].values
            a['affiliation'] = b['Affiliation'].values
            background_affiliation = pd.concat([background_affiliation, a])
        else:
            a = background.loc[background['badge']==i]
            background_affiliation = pd.concat([background_affiliation, a])


    #create a dictionary for member id and background info
    bg_dict = {}
    for i in range(0,len(background_affiliation)):
        bg_dict.update({background_affiliation.loc[i,'name']: str(background_affiliation.loc[i,'background'])+", "+
                                                              str(background_affiliation.loc[i,'affiliation'])+", "+
                                                              str(i)})

    #relabel the nodes with the background infomation
    graph_bg = nx.relabel_nodes(graph,bg_dict)

    # create the graph with filtered interactions 
    fig = plt.figure(figsize=(12,10), dpi=150)
    draw_graph(graph_bg, graph_layout="spring",node_size=60)
    plt.show()
    
    
draw_graph_button.on_click(draw_graph_button_click_handler)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11744670114797527, 'fraction_numerical': 0.04049451242588621, 'mean_word_length': 3.2585280687617515, 'pattern_counts': {'":': 34, '<': 57, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 28, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '143272', 'license': 'MIT', 'n_tokens_mistral': 11341, 'n_tokens_neox': 10587, 'n_words': 2154, 'path': 'Dynamic_Network_Graph_Exploration.ipynb', 'repo_name': 'VerolaX/Capstone-URMC', 'size': '31726'}