<jupyter_start><jupyter_text>---_You are currently looking at **version 1.2** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-social-network-analysis/resources/yPcBs) course resource._---  Assignment 4<jupyter_code>import networkx as nx
import pandas as pd
import numpy as np
import pickle<jupyter_output><empty_output><jupyter_text>--- Part 1 - Random Graph IdentificationFor the first part of this assignment you will analyze randomly generated graphs and determine which algorithm created them.<jupyter_code>P1_Graphs = pickle.load(open('A4_graphs','rb'))
P1_Graphs<jupyter_output><empty_output><jupyter_text>`P1_Graphs` is a list containing 5 networkx graphs. Each of these graphs were generated by one of three possible algorithms:* Preferential Attachment (`'PA'`)* Small World with low probability of rewiring (`'SW_L'`)* Small World with high probability of rewiring (`'SW_H'`)Anaylze each of the 5 graphs and determine which of the three algorithms generated the graph.*The `graph_identification` function should return a list of length 5 where each element in the list is either `'PA'`, `'SW_L'`, or `'SW_H'`.*<jupyter_code># import matplotlib.pyplot as plt
# %matplotlib notebook
# plt.figure(figsize=(5, 4))
# G = P1_Graphs[0]
# node_color = [G.degree(v) for v in G]
# nx.draw_networkx(G, node_size=0.1 ,with_labels=False, width=0.15)
# plt.axis('off')
# plt.tight_layout()
# print(nx.average_clustering(G))
# print(nx.average_shortest_path_length(G))
# plt.figure(figsize=(5, 4))
# G = P1_Graphs[1]
# node_color = [G.degree(v) for v in G]
# nx.draw_networkx(G, node_size=0.1 ,with_labels=False, width=0.15)
# plt.axis('off')
# plt.tight_layout()
# print(nx.average_clustering(G))
# print(nx.average_shortest_path_length(G))
# plt.figure(figsize=(5, 4))
# G = P1_Graphs[2]
# node_color = [G.degree(v) for v in G]
# nx.draw_networkx(G, node_size=0.1 ,with_labels=False, width=0.15)
# plt.axis('off')
# plt.tight_layout()
# print(nx.average_clustering(G))
# print(nx.average_shortest_path_length(G))
# plt.figure(figsize=(5, 4))
# G = P1_Graphs[3]
# node_color = [G.degree(v) for v in G]
# nx.draw_networkx(G, node_size=0.1 ,with_labels=False, width=0.15)
# plt.axis('off')
# plt.tight_layout()
# print(nx.average_clustering(G))
# print(nx.average_shortest_path_length(G))
# plt.figure(figsize=(5, 4))
# G = P1_Graphs[4]
# node_color = [G.degree(v) for v in G]
# nx.draw_networkx(G, node_size=0.1 ,with_labels=False, width=0.15)
# plt.axis('off')
# plt.tight_layout()
# print(nx.average_clustering(G))
# print(nx.average_shortest_path_length(G))
# fig, ax = plt.subplots(1, 5, sharex='col', sharey='row')
# histogram = []
# degree_values_list = []
# for G in P1_Graphs:
#     degrees = G.degree()
#     degree_values = sorted(set(degrees.values()))
#     histogram.append([list(degrees.values()).count(i) / float(nx.number_of_nodes(G)) for i in degree_values])
#     degree_values_list.append(degree_values)

# yaxis = 0
# for degrees, histo_bar in zip(degree_values_list, histogram):
#     ax[yaxis].bar(degrees, histo_bar)
#     ax[yaxis].axis('off')
#     yaxis += 1

# plt.show()
from sklearn.linear_model import LinearRegression
import sklearn.metrics as metrics
def Power_Law(X, y):
    X_log = np.log(X).reshape(len(X), 1)
    y_log = np.log(y).reshape(len(y), 1)
    model = LinearRegression()
    model.fit(X_log, y_log)
    y_pred = model.predict(X_log)
    err = metrics.mean_squared_error(y_log, y_pred)
    #print(err)
    if err < 1:
        return True
    return False
def graph_identification():
    
    # Your Code Here
    result = []
    
    for G in P1_Graphs:
        degrees = G.degree()
        degree_values = sorted(set(degrees.values()))
        histogram = [list(degrees.values()).count(i) / float(nx.number_of_nodes(G)) for i in degree_values]
        #print(degree_values.shape)
        #print(histogram.shape)
        if Power_Law(degree_values, histogram):
            result.append('PA')
        elif nx.average_clustering(G) > 0.1:
            result.append('SW_L')
        else:
            result.append('SW_H')
    return result
graph_identification()<jupyter_output>0.520944933119
4.87568614695
3.38638123744
0.478224799361
1.15502360586<jupyter_text>--- Part 2 - Company EmailsFor the second part of this assignment you will be workking with a company's email network where each node corresponds to a person at the company, and each edge indicates that at least one email has been sent between two people.The network also contains the node attributes `Department` and `ManagementSalary`.`Department` indicates the department in the company which the person belongs to, and `ManagementSalary` indicates whether that person is receiving a management position salary.<jupyter_code>G = nx.read_gpickle('email_prediction.txt')

print(nx.info(G))<jupyter_output>Name: 
Type: Graph
Number of nodes: 1005
Number of edges: 16706
Average degree:  33.2458<jupyter_text>Part 2A - Salary PredictionUsing network `G`, identify the people in the network with missing values for the node attribute `ManagementSalary` and predict whether or not these individuals are receiving a management position salary.To accomplish this, you will need to create a matrix of node features using networkx, train a sklearn classifier on nodes that have `ManagementSalary` data, and predict a probability of the node receiving a management salary for nodes where `ManagementSalary` is missing.Your predictions will need to be given as the probability that the corresponding employee is receiving a management position salary.The evaluation metric for this assignment is the Area Under the ROC Curve (AUC).Your grade will be based on the AUC score computed for your classifier. A model which with an AUC of 0.88 or higher will receive full points, and with an AUC of 0.82 or higher will pass (get 80% of the full points).Using your trained classifier, return a series of length 252 with the data being the probability of receiving management salary, and the index being the node id.    Example:            1       1.0        2       0.0        5       0.8        8       1.0            ...        996     0.7        1000    0.5        1001    0.0        Length: 252, dtype: float64<jupyter_code>def salary_predictions():
    
    # Your Code Here
    df = pd.DataFrame(G.nodes(data=True))
    df['Department'] = df[1].apply(lambda x: x['Department'])
    df['ManagementSalary'] = df[1].apply(lambda x: x['ManagementSalary'])
    df.drop(1, axis=1, inplace=True)
    df.rename(columns = {0:'Node'}, inplace=True)
    df_test = df[df['ManagementSalary'].isnull()]
    df['Degree'] = nx.degree_centrality(G).values()
    df['Closeness'] = nx.closeness_centrality(G).values()
    df['Betweenness'] = nx.betweenness_centrality(G).values()
    df['PageRank'] = nx.pagerank(G).values()
    df['Hub Score'] = nx.hits(G)[0].values()
    df['Auth Score'] = nx.hits(G)[1].values()
    
    X_train = df[df['ManagementSalary'].notnull()][['Degree', 'Closeness', 'Betweenness', 'PageRank',
                                                'Hub Score', 'Auth Score', 'Department']]
    Y_train = df[df['ManagementSalary'].notnull()]['ManagementSalary']
    X_test = df[df['ManagementSalary'].isnull()][['Degree', 'Closeness', 'Betweenness', 'PageRank',
                                                'Hub Score', 'Auth Score', 'Department']]
    Y_test = df[df['ManagementSalary'].isnull()]['ManagementSalary']
    
    from sklearn.preprocessing import MinMaxScaler
    scaler = MinMaxScaler()
    X_train = scaler.fit_transform(X_train)
    X_test = scaler.transform(X_test)
    
    from sklearn.neural_network import MLPClassifier
    model = MLPClassifier(hidden_layer_sizes=[5, 5], alpha=5, solver='lbfgs')
    model.fit(X_train, Y_train)
    
    import sklearn.metrics as metrics
    predicted_prob = [x[1] for x in model.predict_proba(X_test)]
    result = pd.Series(predicted_prob, index=Y_test.index)
    
    return result
salary_predictions()<jupyter_output><empty_output><jupyter_text>Part 2B - New Connections PredictionFor the last part of this assignment, you will predict future connections between employees of the network. The future connections information has been loaded into the variable `future_connections`. The index is a tuple indicating a pair of nodes that currently do not have a connection, and the `Future Connection` column indicates if an edge between those two nodes will exist in the future, where a value of 1.0 indicates a future connection.<jupyter_code>future_connections = pd.read_csv('Future_Connections.csv', index_col=0, converters={0: eval})
future_connections[:10]<jupyter_output><empty_output><jupyter_text>Using network `G` and `future_connections`, identify the edges in `future_connections` with missing values and predict whether or not these edges will have a future connection.To accomplish this, you will need to create a matrix of features for the edges found in `future_connections` using networkx, train a sklearn classifier on those edges in `future_connections` that have `Future Connection` data, and predict a probability of the edge being a future connection for those edges in `future_connections` where `Future Connection` is missing.Your predictions will need to be given as the probability of the corresponding edge being a future connection.The evaluation metric for this assignment is the Area Under the ROC Curve (AUC).Your grade will be based on the AUC score computed for your classifier. A model which with an AUC of 0.88 or higher will receive full points, and with an AUC of 0.82 or higher will pass (get 80% of the full points).Using your trained classifier, return a series of length 122112 with the data being the probability of the edge being a future connection, and the index being the edge as represented by a tuple of nodes.    Example:            (107, 348)    0.35        (542, 751)    0.40        (20, 426)     0.55        (50, 989)     0.35                  ...        (939, 940)    0.15        (555, 905)    0.35        (75, 101)     0.65        Length: 122112, dtype: float64<jupyter_code>def new_connections_predictions():
    
    # Your Code Here
    index_order = future_connections[future_connections['Future Connection'].isnull()].index
    #print(index_order)
    df = future_connections.copy()
    df.sort_index(inplace=True)
    
    jacc_coef = [x[2] for x in list(nx.jaccard_coefficient(G))]
    #print("Jaccard Coef.. Done!")
    ra_index = [x[2] for x in list(nx.resource_allocation_index(G))]
    #print("RA Index.. Done!")
    adamic_adar = [x[2] for x in list(nx.adamic_adar_index(G))]
    #print("Adamic Adar.. Done!")
    pref_attach = [x[2] for x in list(nx.preferential_attachment(G))]
    #print("Pref. Attach.. Done!")
    community_common_neigh = [x[2] for x in list(nx.cn_soundarajan_hopcroft(G, community='Department'))]
    #print("Community Common Neighbor.. Done!")
    community_ra = [x[2] for x in list(nx.ra_index_soundarajan_hopcroft(G, community='Department'))]
    #print("Community RA Index.. Done!")
    
    df['jacc_coef'] = jacc_coef
    df['ra_index'] = ra_index
    df['adamic_adar'] = adamic_adar
    df['pref_attach'] = pref_attach
    df['community_common_neigh'] = community_common_neigh
    df['community_ra'] = community_ra
    
    X_train = df[df['Future Connection'].notnull()][['jacc_coef', 'ra_index', 'adamic_adar',
                                                            'pref_attach', 'community_common_neigh', 'community_ra']]
    y_train = df[df['Future Connection'].notnull()]['Future Connection']
    
    X_test = df[df['Future Connection'].isnull()][['jacc_coef', 'ra_index', 'adamic_adar',
                                                            'pref_attach', 'community_common_neigh', 'community_ra']]
    y_test = df[df['Future Connection'].isnull()]['Future Connection']
    
    #print("Train Test Split.. Done!")
    
    from sklearn.preprocessing import MinMaxScaler
    scaler = MinMaxScaler()
    X_train = scaler.fit_transform(X_train)
    X_test = scaler.transform(X_test)
    
    #print("Scaling.. Done!")
    
    from sklearn.neural_network import MLPClassifier
    model = MLPClassifier(hidden_layer_sizes=[10, 5], alpha=5, solver='lbfgs')
    model.fit(X_train, y_train)
    
    #print("Model Fitted")
    
    import sklearn.metrics as metrics
    predicted_prob = [x[1] for x in model.predict_proba(X_test)]
    result = pd.Series(predicted_prob, index=y_test.index)
    result = result.reindex(index_order)
    
    #print("Results are here.")
    
    return result
new_connections_predictions()<jupyter_output>Index([(107, 348), (542, 751),  (20, 426),  (50, 989), (942, 986), (324, 857),
        (13, 710),  (19, 271), (319, 878), (659, 707),
       ...
       (144, 824), (742, 985), (506, 684), (505, 916), (149, 214), (165, 923),
       (673, 755), (939, 940), (555, 905),  (75, 101)],
      dtype='object', length=122112)
<NAME>.. Done!
RA Index.. Done!
<NAME>.. Done!
Pref. Attach.. Done!
Community Common Neighbor.. Done!
Community RA Index.. Done!
Train Test Split.. Done!
Scaling.. Done!
Model Fitted
Results are here.
{'fraction_non_alphanumeric': 0.11766480615477448, 'fraction_numerical': 0.03190526474581385, 'mean_word_length': 3.885408990420044, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '334961', 'license': 'MIT', 'n_tokens_mistral': 4555, 'n_tokens_neox': 4225, 'n_words': 1251, 'path': 'Applied Social Network Analysis in Python/Week4 - Network Evolution/Assignment+4.ipynb', 'repo_name': 'rajatgarg149/Data-Science-Python------Coursera-MICHIGAN-', 'size': '13269'}