<jupyter_start><jupyter_text>Analysis of the attributes<jupyter_code>actors_agg_adj = np.load("sparse_agg_actor_adj.npy")
actors_agg_df = pd.read_pickle("actors_agg_df.pkl")
actors_graph = nx.from_numpy_matrix(actors_agg_adj)
# Change gender
actors_agg_df["gender"] = actors_agg_df["gender"].map(lambda x: "F" if x == 1 else "M")
# Add name and gender as node attribute
name_dict = {}
gender_dict = {}
attributes = actors_agg_df[['actors', 'gender']].values
for index, attr in enumerate(attributes):
    name_dict[index] = attr[0]
    gender_dict[index] = attr[1]
nx.set_node_attributes(actors_graph, name_dict, 'name')
nx.set_node_attributes(actors_graph, gender_dict, 'gender')
# Add eigenvector centrality as node attribute
eigenvector_dict = nx.eigenvector_centrality(actors_graph)
nx.set_node_attributes(actors_graph, eigenvector_dict, 'eigenvector')
seed = 30
# Create communities using community detection library
communities_dict = community.best_partition(actors_graph, random_state = seed)
# Add community as attribute
nx.set_node_attributes(actors_graph, communities_dict, 'community')
# Extract communities to an array of objects
communities = []
for i in range(max(communities_dict.values()) + 1):
    communities.append([actors_graph.node[n] for n in actors_graph.nodes() if actors_graph.node[n]['community'] == i])
# Extract communities to an array of objects
communities = []
for i in range(max(communities_dict.values()) + 1):
    communities.append([actors_graph.node[n] for n in actors_graph.nodes() if actors_graph.node[n]['community'] == i])
# Create dataframe from extracted communities 
communities_df = [pd.DataFrame(n) for n in communities]
naive_communities_df = communities_df
# Find representatives of each class by eigenvector centrality
for index, community in enumerate(communities_df):
    print(f'Representatives of community {index} with a size of {len(community.index)} actors\n{community.nlargest(10, "eigenvector")}\n')<jupyter_output>Representatives of community 0 with a size of 835 actors
                name gender  eigenvector  community
462       <NAME>      M     0.030552          0
525     <NAME>      M     0.030534          0
48     <NAME>      F     0.028134          0
685  <NAME>      F     0.028000          0
150    <NAME>      M     0.027244          0
268    <NAME>      M     0.026938          0
99    <NAME>      F     0.024650          0
172     <NAME>      M     0.023269          0
174      <NAME>      M     0.023267          0
15      <NAME>      F     0.022856          0

Representatives of community 1 with a size of 526 actors
                 name gender  eigenvector  community
156    <NAME>      M     0.043506          1
211  <NAME>      M     0.043262          1
106    <NAME>      F     0.042612          1
381      <NAME>      M     0.042459          1
95       <NAME>      F     0.042313       [...]<jupyter_text>We can observe that in community 2 we have notable A-list actors while in the smallest one (community 7) we have celebrities who might have participated in some movies but are not really actors.  Visualization<jupyter_code># Visualize with colors per community
groups = set(nx.get_node_attributes(actors_graph,'community').values())
mapping = dict(zip(sorted(groups),count()))
nodes = actors_graph.nodes()
colors = [mapping[actors_graph.node[n]['community']] for n in nodes]

# Drawing nodes and edges separately to add colors
plt.figure(figsize=(28, 7))
pos = nx.spring_layout(actors_graph)
ec = nx.draw_networkx_edges(actors_graph, pos, alpha=0.1, width=0.7)
nc = nx.draw_networkx_nodes(actors_graph, pos, nodelist=nodes, node_color=colors, 
                            with_labels=False, node_size=10, cmap=plt.cm.jet)
plt.colorbar(nc)
plt.axis('off')
plt.title("Louvain Communities")
plt.savefig("plots/louvain.pdf")
plt.show()<jupyter_output>/Users/arielalba/anaconda3/envs/network-tour/lib/python3.7/site-packages/networkx/drawing/nx_pylab.py:579: MatplotlibDeprecationWarning: 
The iterable function was deprecated in Matplotlib 3.1 and will be removed in 3.3. Use np.iterable instead.
  if not cb.iterable(width):<jupyter_text>We can observe that community 2 (the famous A-list actors) tend to work together more often, and also with a selection of actors from other communities. Some mid-tier actors are popular and mix with many other communities, while some are stuck in their own community. The celebrities who are not really actors (community 7) are very separated from other actors communities.  Explain communities with logistic regression on signals<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
actors_id_df = actors_agg_df[["actors"]].reset_index()
actors_id_df.head()
communities_df = pd.concat(communities_df).drop(columns=["gender","eigenvector"])
print(len(communities_df))
communities_df.head()
community_labels_df = communities_df.merge(actors_id_df,left_on="name",right_on="actors").drop(columns=["name"]).set_index("actor_id")
print(np.unique(community_labels_df.index))
print(len(np.unique(community_labels_df.index)))
community_labels_df.head()

community_labels_df=community_labels_df.drop(columns=["actors"])
community_labels_df.head()
actors_dataset_df = actors_agg_df.join(community_labels_df)[["budget","popularity","revenue","vote_average","vote_count","community"]]
# Save .npy file to use later
actors_with_communities_dataset = actors_dataset_df.to_numpy()
np.save("actors_with_communities_dataset", actors_with_communities_dataset)
actors_dataset_df.head()<jupyter_output><empty_output><jupyter_text>Save dataset with communities<jupyter_code>actors_dataset_df.to_pickle("actors_dataset_df.pkl")
actors_dataset = actors_dataset_df.values
X = actors_dataset[:,1:-1]
y = actors_dataset[:,-1]
train_features,test_features,train_labels,test_labels=train_test_split(X,y,test_size=0.2)
# Fit a logistic regression model
max_iter = 10000000
std_scaler = StandardScaler()
scaled_train_features = std_scaler.fit_transform(train_features)
model = LogisticRegression(
    solver="liblinear", multi_class="auto", max_iter=max_iter, C=10000
)
model.fit(scaled_train_features, train_labels)
scaled_test_features = std_scaler.transform(test_features)
predict_train = model.predict(scaled_train_features)
predict_test = model.predict(scaled_test_features)
accuracy=accuracy_score(test_labels,predict_test)
print("Accuracy: " +str(accuracy))<jupyter_output>Accuracy: 0.5265822784810127<jupyter_text>Boxplots of signals on communities<jupyter_code>signals = ["budget","popularity","vote_average","vote_count"]
f, axes = plt.subplots(4,1,figsize=(10,15),sharex= True)
for i, signal in enumerate(signals):
    sns.boxplot(x="community", y=signal, data=actors_dataset_df, ax=axes[i]);
    # Uncomment if you want to see every point
    # sns.swarmplot(x="community", y=signal, data=actors_dataset_df, color=".25", ax=axes[i]);
plt.savefig("plots/communities_boxplots.pdf")<jupyter_output><empty_output><jupyter_text>Save for Gephi<jupyter_code># Add name and gender as node attribute
columns_of_type_set = ["cast", "crew", "movie_id", "genres", "keywords", 
                       "original_language", "production_companies", 
                       "production_countries", "spoken_languages", "title",
                       "release_date", "status"]

for col in columns_of_type_set:
    actors_agg_df[col] = actors_agg_df[col].map(gefx_compatible)

actors_agg_df["id"] = actors_agg_df["actors"]
actors_agg_df["name"] = ""

# update names for the most important actors of each community
for index, community in enumerate(naive_communities_df):
    names = community.nlargest(10, "eigenvector")["name"]
    actors_agg_df.loc[actors_agg_df["actors"].isin(names), "name"] = actors_agg_df.loc[actors_agg_df["actors"].isin(names), "actors"]

cols_to_gefx = ["id", "name", "gender", "budget", "genres", "popularity", "revenue", "spoken_languages", "vote_average", "vote_count"]
attrs_to_graph(actors_graph, actors_agg_df[cols_to_gefx])

for index, community in enumerate(naive_communities_df):
    names = community.nlargest(10, "eigenvector")["name"]
    names_dict = actors_agg_df.loc[actors_agg_df["actors"].isin(names), "actors"].to_dict()
    actors_graph = nx.relabel_nodes(actors_graph, names_dict)

# write in gefx format
DATA_PATH = "data"
nx.write_gexf(actors_graph, f"{DATA_PATH}/louvain_graph.gexf")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11017048794826573, 'fraction_numerical': 0.029041740152851263, 'mean_word_length': 5.1018651362984215, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '277318', 'license': 'MIT', 'n_tokens_mistral': 2983, 'n_tokens_neox': 2730, 'n_words': 629, 'path': 'projects/project_ntds_2019/05_analysis_and_visualization.ipynb', 'repo_name': 'eliasmpw/EE-558-ntds', 'size': '8608'}