<jupyter_start><jupyter_text>Self Study 4 In this self study we implement and test a simple Markov network model for node prediction, and a Gibbs sampling inference (prediction) process. For this material there is no direct support from scikit learn, so we have to do a little more coding ourselves than before ...For basic network functionalities we use the `networkx` package: https://networkx.github.io/documentation/stable/index.html<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import networkx as nx
from networkx import read_edgelist
import random<jupyter_output><empty_output><jupyter_text>We are using a social network consisting of 71 lawyers. A description of the network and the original data can be found here:http://moreno.ss.uci.edu/data.htmllazegaOf the three different relationships included in the data we will only be using the 'friendship' relation. This is a directed relationship, i.e., friends(a,b) does not necessarily imply friends(b,a) according to the data. We first construct a networkx graph from the file lazega-friends.edges. The node attributes are read from a separate file into a Pandas data frame:<jupyter_code>lazega=read_edgelist('lazega-friends.edges',nodetype=int) 
node_atts=pd.read_csv("lazega-attributes.txt", sep=' ')
print(node_atts.loc[1,'nodeGender'])
display(node_atts)<jupyter_output>1<jupyter_text>Now we annotate the nodes in the graph with some selected attributes (later on, if you want to do more, you can consider additional attributes).<jupyter_code>for i in range(node_atts.shape[0]):
    lazega.add_node(node_atts.loc[i,'nodeID'], gender=node_atts.loc[i,'nodeGender'])
    lazega.add_node(node_atts.loc[i,'nodeID'], office=node_atts.loc[i,'nodeOffice'])
    lazega.add_node(node_atts.loc[i,'nodeID'], true_practice=node_atts.loc[i,'nodePractice'])
    if random.random() > 0.4:
        lazega.add_node(node_atts.loc[i,'nodeID'], observed_practice=node_atts.loc[i,'nodePractice'])
    else:
        lazega.add_node(node_atts.loc[i,'nodeID'], observed_practice=np.nan)<jupyter_output><empty_output><jupyter_text>The 'practice' attribute is going to be our class label that we want to predict. Therfore, we pretend that this attribute is not observed for some randomly selected nodes. We do this by adding a new attribute 'observed_practice' which has either the true practice value, or 'nan' for unobserved.<jupyter_code>random.seed(5)
for i in range(node_atts.shape[0]):
    if random.random() > 0.4:
        lazega.add_node(node_atts.loc[i,'nodeID'], observed_practice=node_atts.loc[i,'nodePractice'])
    else:
        lazega.add_node(node_atts.loc[i,'nodeID'], observed_practice=np.nan)<jupyter_output><empty_output><jupyter_text>The following is a utility function for retrieving a plain numpy array containing the values of a selected attribute for all nodes.<jupyter_code>def get_att_array(G,att_name):
    ret_array=np.zeros(nx.number_of_nodes(G))
    for i,n in enumerate(G.nodes()):
        ret_array[i]=G.nodes[n][att_name]
    return(ret_array)
print(get_att_array(lazega,'observed_practice'))<jupyter_output>[ 1.  2.  2.  1. nan  2. nan nan  1.  2.  1. nan nan nan  1.  1. nan nan
  1. nan nan  2. nan  2.  1.  2.  1.  2.  1.  1. nan nan  1.  2.  2. nan
  1.  1.  1.  1. nan  2. nan nan nan nan  1.  1. nan nan  1.  1.  2. nan
  2. nan nan nan  2.  1.  2.  2. nan  1. nan  2. nan nan nan nan nan]<jupyter_text>**Beware:** lazega.nodes[n] returns nodes according to their key (= nodeID) value, which are the integers 1..71, whereas the arrays returned by get_att_array are indexed 0..70. We can draw the network with node coloring according to a selected attribute as follows:<jupyter_code>nx.draw_kamada_kawai(lazega,with_labels=True,node_color=get_att_array(lazega,'observed_practice'))<jupyter_output>/home/starksultana/anaconda3/lib/python3.7/site-packages/networkx/drawing/nx_pylab.py:579: MatplotlibDeprecationWarning: 
The iterable function was deprecated in Matplotlib 3.1 and will be removed in 3.3. Use np.iterable instead.
  if not cb.iterable(width):<jupyter_text>We now want to predict the missing 'practice' labels using Gibbs sampling on a Markov network model. We will first need to initialize values for the unobserved attributes. **Task 1:** complete the following function:<jupyter_code>def init_practice(): 
   ...:     
   ...:     for i in range(len(get_att_array(lazega,'observed_practice'))): 
   ...:         if (np.isnan(get_att_array(lazega,'observed_practice')[i]))!=True: 
   ...:             #print(get_att_array(lazega,'observed_practice')[i])
   ...:             lazega.add_node(node_atts.loc[i,'nodeID'], predicted_practice=get_att_array(lazega,'observed_practice')[i]) 
   ...:         else:  
   ...:             #print("Hey") 
   ...:             lazega.add_node(node_atts.loc[i,'nodeID'], predicted_practice=random.randint(1,2)) 
   ...:     print(get_att_array(lazega,'predicted_practice')) 
   ...:      
   ...: init_practice()<jupyter_output>[1. 2. 1. 1. 1. 1. 2. 2. 2. 1. 1. 1. 1. 1. 2. 2. 1. 1. 1. 1. 1. 1. 2. 2.
 2. 2. 1. 1. 2. 1. 1. 1. 2. 2. 2. 1. 1. 1. 1. 1. 1. 2. 1. 2. 1. 1. 1. 2.
 1. 2. 2. 2. 1. 1. 1. 1. 2. 1. 2. 2. 2. 2. 2. 2. 1. 2. 1. 1. 1. 1. 1.]<jupyter_text>We next define some potential functions. Mostly we will be using node or edge potentials, i.e., potential functions thatdepend only on the attributes of a single node, or on the attributes of two nodes (connected by an edge). Instead of defining the potential function directly, it is more convenient to first define the log of the potential function. Then, instead of taking a big product, one can first take the sum of relevant potentials, and apply an exponential at the end (cf. slides 12, 20, and 23). E.g., the log of the homophily potential (slide 20) is w in the case A_i=A_j, and 0 otherwise. Also, all real numbers are then permissible return values, without the non-negativity condition.The following are examples of a simple node (log-)potential and an "ising-style" homophily potential. Both potential functions depend on numerical parameters w1,w2. You may start with using these two log-potentials only, and later define additional potential functions.<jupyter_code>def n_log_potential_1(n,w1,w2):
    if n['predicted_practice'] == 1: return w1
    else: return w2
def n_log_potential_2(n1,n2,w1,w2):
    if n1['predicted_practice']==n2['predicted_practice']: return w1
    else: return w2
def n_log_potential_3(n,w1,w2):
    if n['predicted'] == 2: return w1
    else: return w2
print(n_log_potential_1(lazega.nodes[4],-1,1))

print(n_log_potential_2(lazega.nodes[4],lazega.nodes[6],-2,3))

print(lazega.nodes[10])<jupyter_output>-1
3
{'gender': 1, 'office': 1, 'true_practice': 2, 'observed_practice': 2, 'predicted_practice': 2.0}<jupyter_text>**Task 2:** Write a Gibbs-sampling function for re-sampling the 'predicted_practice' attribute value for a node *n*. At this point we need not worry whether for *n* the actual 'practice' value is known or not.<jupyter_code>import math 
def gibbs_sample(n,w1,w2): #should be only n should use all neighbors, but aint working 
    node_potentials1 = []
    node_potentials2 = []
    edge_potentials = [] 
         
    potential_1 = n_log_potential_1(n,w1,w2) 
    node_potentials1.append(potential_1)
    potential_1 = n_log_potential_1(n,w1,w2) 
    node_potentials2.append(potential_1)  
          
    if np.isnan(n['predicted_practice']) != True: 
        neighbors = nx.all_neighbors(lazega,n['predicted_practice'])
    for key in list(neighbors):
        potential_2 = n_log_potential_2(n,lazega.nodes[key],w1,w2)
        edge_potentials.append(potential_2)    
    nodes_sum1 = sum(node_potentials1)
    nodes_sum2 = sum(node_potentials2)
    edges_sum = sum(edge_potentials)
    exp_sum1 = math.exp(0.5*(edges_sum+nodes_sum1))    
    exp_sum2 = math.exp(0.5*(edges_sum+nodes_sum2)) 
 
   
    new_probability=(exp_sum1/exp_sum1+exp_sum2)/100
    print(new_probability)
    if random.uniform(0, 1) > new_probability: 
        n['predicted_practice'] = 2 
    else: 
        n['predicted_practice']= 1 
    #print(n['predicted_practice'])
    return n['predicted_practice']
     
   
gibbs_sample(lazega.nodes[4],-1,1)



        
        
        
    # Iterate over all log-potential functions you want to use
    # For node potentials, evaluate the potential for the given node n
    # For edge potentials, evaluate the potential for all pairs (n,n') where n' is a neighbor in the 'friendship' graph.
    #
    # Since the friendship graph is directed, there are three possibilities of how to do this precisely:
    #        - consider all n' where friends(n,n')
    #        - consider all n' where friends(n',n)
    #        - consider both cases of n'
    # 
    # The method nx.all_neighbors(lazega,n) will return an iterator over both types of neighbors of n, so the third
    # option is the most convenient to use (and maybe also the most sensible)
    #
    # Sum the values of all the potential functions, and take the exponential.
    # This has to be done for both possible values of the 'predicted_practice' value for n
    #
    # You will probably need to add arguments to the gibbs_sample method for the numerical parameters of 
    # the potential functions that you are using.
    # 
    # Calculate the probability for predicted_practice(n) according to the quotient shown on slide 12 (see also slide 23)
    #
    # Set the new value of predicted_practice(n) randomly according to the probabilities you have just computed.<jupyter_output>0.0122313016014843<jupyter_text>Once the resampling of the predicted_practice attribute for a single node is in place, the rest is quite straightforward: **Task 3**: Write a function that performs one round of Gibbs sampling, i.e., re-samples the predicted values for all nodes for which the class attribute is unknown.<jupyter_code>def gibbs_one_round(G): 
    for i in G.nodes(): 
        if (np.isnan(lazega.nodes[i]['observed_practice'])):
            #print(lazega.nodes[i]['observed_practice'])
            lazega.nodes[i]['observed_practice'] = gibbs_sample(lazega.nodes[i],-1,1)
            #print(lazega.nodes[i]['observed_practice'])
    #print(prediction)
    return get_att_array(lazega,'observed_practice')
    
    
          
gibbs_one_round(lazega)<jupyter_output><empty_output><jupyter_text>Now we can put everything together to use our model to predict the 'practice' attribute. **Task 4**: write code for doing the following:<jupyter_code>count_1= 0
count_2= 0
#values=[]
burn_in = 500
for j in range(1000):
    value = gibbs_one_round(lazega)
    if j>burn_in:
       break
       

print(value)
nr_diff_values = 0
for i in range(node_atts.shape[0]):
    if value[i] != get_att_array(lazega,'true_practice')[i]:
        nr_diff_values +=1

accuracy = ((len(lazega.nodes())-nr_diff_values)/len(lazega.nodes()))*100
print(accuracy)

# Perform a number of gibbs_one_round(lazega) sampling steps
#
# (Maybe after a certain number of burn-in iterations): keep count of how often the predicted_practice value of nodes 
# with observed_practice == nan is in the two states 1 or 2.
# 
# Predict the unobserved practice values as the more probably state in the Gibbs sample
#
# Compare your predicted values against the true values<jupyter_output>[1. 2. 2. 1. 1. 2. 1. 1. 1. 2. 1. 1. 2. 1. 1. 1. 1. 1. 1. 2. 2. 2. 2. 2.
 1. 2. 1. 2. 1. 1. 1. 1. 1. 2. 2. 1. 1. 1. 1. 1. 1. 2. 1. 1. 1. 1. 1. 1.
 1. 2. 1. 1. 2. 2. 2. 2. 2. 2. 2. 1. 2. 2. 2. 1. 1. 2. 1. 1. 1. 2. 1.]
83.09859154929578<jupyter_text>Now that things are up and running, we can explore some of the properties of the model and the Gibbs sampling procedure: **Task 5**:  Perform the Gibbs-prediction procedure several times and explore how stable the prediction results are. Apart from the final categorical prediction, you can also consider the actual frequencies of the two 'predicted_practice" states in your samples     Try different settings of the parameter values of the potential functions. Can you find a relationship between the parameter values, and the stability of the Gibbs sampling?      Also try different parameter settings in order to optimize prediction accuracy. In a real application, one would learn the parameter values by likelihood optimization based on the labeled training nodes. This is outside the scope of what we can do here, so let's do a simple grid search over parameter values instead (ugly!) and evaluate by checking accuracy using the known actual labels for the test nodes (knowing that -- in principle -- this is forbidden!).<jupyter_code>##run it several times and note the results
##<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10237587812771332, 'fraction_numerical': 0.029284079248559476, 'mean_word_length': 3.8266666666666667, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 46, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '31243', 'license': 'MIT', 'n_tokens_mistral': 4482, 'n_tokens_neox': 4046, 'n_words': 1420, 'path': 'ML_selfstudy4/MLSelfStudy4-F20 (1).ipynb', 'repo_name': 'Jbarata98/ML_AAU1920', 'size': '12669'}