<jupyter_start><jupyter_text>Exploratory Data Analysis of graph data **Libraries**1. NetworkX: https://networkx.org/2. scikit-network: https://scikit-network.readthedocs.io/en/latest/<jupyter_code># Load modules
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(rc={'figure.figsize':(13,6)})

import networkx as nx
%%bash
cd ../input
ls
# TODO
# - Create synthetic datasets or Find an undirected graph
# - Plot graph, degrees distribution, edge prob distributions
# - Find central nodes<jupyter_output><empty_output><jupyter_text>Datasets You are free to use any **probabilistic graph** available. A small dataset will be given. However, since not manyreal-life probabilistic networks are publicly available, you may work use synthetic datasets as follows: youmay take any undirected network and simply assign probabilities to the edges by using a probabilitydistribution such as uniform, normal, power-law, etc.Two excellent repositories for graph data are:1. http://snap.stanford.edu/data/index.html2. http://networkrepository.com/ **Graph Embedding with Self Clustering: Facebook** http://snap.stanford.edu/data/gemsec-Facebook.html  Collins<jupyter_code># Load datasets
collins_df = pd.read_csv("../input/collins.csv",header=None)
collins_df.columns = ["src","dst", "probability"]
collins_df.head(6)
collins_df.shape
collins_df["probability"].plot.hist(title="Edge Probabilities distribution", bins=100, xlim=[0,1]);
# Create graph
G = nx.from_pandas_edgelist(df=collins_df, source='src', target='dst', edge_attr="probability")
# Print graph properties
print(f" Number of nodes: {G.number_of_nodes()}")
print(f" Number of edges: {G.number_of_edges()}")
print(f" Is directed?: {G.is_directed()}")
# list(G.nodes)
# Plot degree distribution
plt.hist(dict(G.degree()).values(), bins=100)
plt.title("Degrees distribution");
# Plot graph
# nx.draw(G)
# nx.draw_spring(G, node_size=100, node_color="#00C98D", with_labels=False)<jupyter_output><empty_output><jupyter_text>Facebook datasets<jupyter_code>import numpy as np
# Load datasets
artist_edges_df = pd.read_csv("../input/gemsec_facebook_dataset/artist_edges.csv")
artist_edges_df.head()
# Number of edges
artist_edges_df.shape
# Number of nodes
len(np.unique(np.hstack([artist_edges_df["node_1"].values, artist_edges_df["node_2"].values])))<jupyter_output><empty_output><jupyter_text>**Artist Dataset**- Number of edges : 50,515- Number of nodes : 819,306 **Probability distributions**1. Uniform2. Normal3. Power-Law<jupyter_code># Drop self loops
print("Before:", artist_edges_df.shape)
artist_edges_df = artist_edges_df[artist_edges_df["node_1"] != artist_edges_df["node_2"]]
print("After:", artist_edges_df.shape)
artist_edges_df[artist_edges_df["node_1"]==11760]
## 1. Uniform
artist_edges_df["probability"] = np.random.uniform(0,1, size=len(artist_edges_df))
(artist_edges_df["probability"]
 .plot
 .hist(title="Edge Probabilities distribution (Uniform)", bins=100, xlim=[0,1]));
# Save dataset
artist_edges_df.to_csv("../input/artists_uniform.csv", index=False, header=False)
## 2. Normal
probs = np.random.normal(0.5, 0.10, size=len(artist_edges_df))
probs[probs<0]=0
probs[probs>1]=0

artist_edges_df["probability"] = probs
(artist_edges_df["probability"]
 .plot
 .hist(title="Edge Probabilities distribution (Normal)", bins=100, xlim=[0,1]));
# Save dataset
artist_edges_df.to_csv("../input/artists_normal.csv", index=False, header=False)
## 3. Power Law
probs = np.random.power(0.7, size=len(artist_edges_df))
probs[probs<0]=0
probs[probs>1]=0

artist_edges_df["probability"] = probs
(artist_edges_df["probability"]
 .plot
 .hist(title="Edge Probabilities distribution (Power Law)", bins=100, xlim=[0,1]));
# Save dataset
artist_edges_df.to_csv("../input/artists_power_law.csv", index=False, header=False)<jupyter_output><empty_output><jupyter_text>Experiment results<jupyter_code>import plotly.express as px
import plotly.graph_objects as go
import pandas as pd<jupyter_output><empty_output><jupyter_text>Load experiments **Experiments**1. experiments.csv2. experiments_threshold_0.5.csv3. experiments_k_large.csv<jupyter_code># Load experimenation results
experiments_df = pd.read_csv("../experiments/experiments.csv")
experiments_threshold_05_df = pd.read_csv("../experiments/experiments_threshold_0.5.csv")
experiments_k_large_df = pd.read_csv("../experiments/experiments_k_large.csv")
# Check total experiments per file
print("experiments_df:", experiments_df.shape)
print("experiments_threshold_05_df:", experiments_threshold_05_df.shape)
print("experiments_k_large_df:",experiments_k_large_df.shape)
# Check experiments
pd.set_option('display.max_rows', 200)
experiments_df<jupyter_output><empty_output><jupyter_text>**There are 4 different variables (degrees of freedom). I must visualize the efect of each one in running times.**1. Scripts (Algorithms)    * Graphframes    * Rdd    * Best graphframes + rdd2. Cores3. Distribution4. k **`Experiments` file: Degrees of Freedom**1. scripts = ["graphframe_bs.py", "graphframe_fast.py", "graphframe_fast_bv.py", "rdd_bs.py", "rdd_fast.py", "rdd_fast_bv.py"]2. cores = [1,2,8]3. dataset in ["artists_uniform", "artists_normal", "artists_power_law"]4. k in [100, 1000, 10000]<jupyter_code>print(experiments_df.shape)
print(6 * 3 * 3 * 3)
# Check experiments_threshold_05_df
experiments_threshold_05_df<jupyter_output><empty_output><jupyter_text>**Experiments `experiments_threshold_05_df`: Degrees of Freedom*** scripts = ["graphframe_fast.py", "graphframe_fast_bv.py", "rdd_fast.py", "rdd_fast_bv.py"]  * cores = [8]  * dataset in ["artists_normal"]  * k in [100, 1000, 10000]<jupyter_code>print(experiments_threshold_05_df.shape)
print(4*1*1*3)
# Check experiments_k_large_df
experiments_k_large_df<jupyter_output><empty_output><jupyter_text>**Experiments `experiments_k_large_df`: Degrees of Freedom*** scripts = ["graphframe_bs.py", "graphframe_fast.py", "graphframe_fast_bv.py", "rdd_bs","rdd_fast.py", "rdd_fast_bv.py"]* cores = [8]* dataset in ["artists_uniform", "artists_normal", "artists_power_law"]* k = 2_300_000<jupyter_code>print(experiments_k_large_df.shape)
print(6*1*3*1)<jupyter_output>(18, 5)
18<jupyter_text>Prepare Files<jupyter_code># experiments_df
experiments_df = experiments_df.rename(columns={"script":"algorithm",
                                                "time":"time (sec)"})

experiments_df["algorithm"] = experiments_df["algorithm"].apply(lambda x: x.split(".")[0])

experiments_df["distribution"] = experiments_df["dataset"].apply(lambda x: "_".join(x.split("_")[1:]))
# experiments_df
experiments_threshold_05_df = experiments_threshold_05_df.rename(columns={"script":"algorithm",
                                                "time":"time (sec)"})

experiments_threshold_05_df["algorithm"] = experiments_threshold_05_df["algorithm"].apply(lambda x: x.split(".")[0])

experiments_threshold_05_df["distribution"] = experiments_threshold_05_df["dataset"].apply(lambda x: "_".join(x.split("_")[1:]))
# experiments_df
experiments_k_large_df = experiments_k_large_df.rename(columns={"script":"algorithm",
                                                "time":"time (sec)"})

experiments_k_large_df["algorithm"] = experiments_k_large_df["algorithm"].apply(lambda x: x.split(".")[0])

experiments_k_large_df["distribution"] = experiments_k_large_df["dataset"].apply(lambda x: "_".join(x.split("_")[1:]))
# Fix data types
# experiments_df["k"] = experiments_df["k"].astype(str)
# Fastest method
experiments_df.sort_values("time (sec)").head(10)
# Slowest method
experiments_df.sort_values("time (sec)", ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Full experiments **Total graphframes experiments**: 3 rows * 3 columns * 3 lines * 3 points = 81<jupyter_code># 2. Compare graphframes implementations (All distributions)
filtered_dataset = experiments_df[(experiments_df["algorithm"].isin(["graphframe_bs",
                                                                     "graphframe_fast",
                                                                     "graphframe_fast_bv"]))].copy()


filtered_dataset["algorithm"] = (filtered_dataset["algorithm"]
                                 .replace({"graphframe_bs":"baseline",
                                           "graphframe_fast":"optimization 1",
                                           "graphframe_fast_bv":"optimization 2"}))


fig = px.line(filtered_dataset,
                 x="k",
                 y="time (sec)",
                 log_x=True,
                 log_y=False,
                 facet_row="distribution",
                 color="algorithm",
                 facet_col="cores")

fig.update_layout(
    title=f"Experiment times of GraphFrames implementations"
)


for i in range(len(fig.data)):
    fig.data[i].update(mode='markers+lines')

fig.update_layout(
    autosize=False,
    width=1000,
    height=700)
fig.show()<jupyter_output><empty_output><jupyter_text>**Total rdd experiments**: 3 rows * 3 columns * 3 lines * 3 points = 81<jupyter_code># 3. Compare rdd (All distributions)
filtered_dataset = experiments_df[(experiments_df["algorithm"].isin(["rdd_bs",
                                                                     "rdd_fast",
                                                                     "rdd_fast_bv"]))].copy()

filtered_dataset["algorithm"] = (filtered_dataset["algorithm"]
                                 .replace({"rdd_bs":"baseline",
                                           "rdd_fast":"optimization 1",
                                           "rdd_fast_bv":"optimization 2"}))

fig = px.line(filtered_dataset,
                 x="k",
                 y="time (sec)",
                 log_x=True,
                 log_y=False,
                 facet_row="distribution",
                 color="algorithm",
                 facet_col="cores",
                 labels={"rdd_bs": "test", "rdd_fast": "test", "rdd_fast_bv":"r2w43r"})

fig.update_layout(
    title=f"Experiment times of RDD implementations",
    yaxis_title="time (sec)"
)

for i in range(len(fig.data)):
    fig.data[i].update(mode='markers+lines')

fig.update_layout(
    autosize=False,
    width=1000,
    height=700)
fig.show()<jupyter_output><empty_output><jupyter_text>Experiments (threshold=0.5)<jupyter_code>experiments_threshold_05_df
## Compare graphframes_fast and graphframes_fast_bv 
# with threshold=0.8 and threshold=0.5 for Normal distribution

fig = go.Figure()


## --> Extract times:
# bs, optimization 1, optimization 2
# threshold 0.5, 0.8

df_bs_08 = (experiments_df
            [(experiments_df["cores"]==8) &
             (experiments_df["dataset"]=="artists_normal") &             
             (experiments_df["algorithm"] == "graphframe_bs")])

df_opt1_08 = (experiments_df
            [(experiments_df["cores"]==8) &
             (experiments_df["dataset"]=="artists_normal") &             
             (experiments_df["algorithm"] == "graphframe_fast")])

df_opt2_08 = (experiments_df
             [(experiments_df["cores"]==8) &
             (experiments_df["dataset"]=="artists_normal") &             
             (experiments_df["algorithm"] == "graphframe_fast_bv")])


df_opt1_05 = (experiments_threshold_05_df
              [experiments_threshold_05_df["algorithm"] == "graphframe_fast"])


df_opt2_05 = (experiments_threshold_05_df
              [experiments_threshold_05_df["algorithm"] == "graphframe_fast_bv"])




## Add lines

# 1. df_bs_08
fig.add_trace(go.Scatter(
    x = df_bs_08["k"],
    y = df_bs_08["time (sec)"],
    name="baseline"
))


# 2. df_opt1_08
fig.add_trace(go.Scatter(
    x = df_opt1_08["k"],
    y = df_opt1_08["time (sec)"],
    name="optimization 1, threshold=0.8",
    line = dict(color='firebrick')
))

# 3. df_opt2_08
fig.add_trace(go.Scatter(
    x = df_opt2_08["k"],
    y = df_opt2_08["time (sec)"],
    name="optimization 2, threshold=0.8",
    line = dict(color='#00CC96')
))


# 4. df_opt1_05
fig.add_trace(go.Scatter(
    x = df_opt1_05["k"],
    y = df_opt1_05["time (sec)"],
    name="optimization 1, threshold=0.5",
    line = dict(color='firebrick',dash='dash')
))


# 5. df_opt2_05
fig.add_trace(go.Scatter(
    x = df_opt2_05["k"],
    y = df_opt2_05["time (sec)"],
    name="optimization 2, threshold=0.5",
    line = dict(color='#00CC96',dash='dash')
))

fig.update_layout(
    title=f"Experiment times of GraphFrames implementations for threshold=0.8 and threshold=0.5 | Normal Distribution",
    xaxis_title="k",
    yaxis_title="time (sec)",
)

fig.update_xaxes(type="log")

fig.show()
## Compare rdd_fast and rdd_fast_bv
# with threshold=0.8 and threshold=0.5 for Normal distribution

fig = go.Figure()


## --> Extract times:
# bs, optimization 1, optimization 2
# threshold 0.5, 0.8

df_bs_08 = (experiments_df
            [(experiments_df["cores"]==8) &
             (experiments_df["dataset"]=="artists_normal") &             
             (experiments_df["algorithm"] == "rdd_bs")])

df_opt1_08 = (experiments_df
            [(experiments_df["cores"]==8) &
             (experiments_df["dataset"]=="artists_normal") &             
             (experiments_df["algorithm"] == "rdd_fast")])

df_opt2_08 = (experiments_df
             [(experiments_df["cores"]==8) &
             (experiments_df["dataset"]=="artists_normal") &             
             (experiments_df["algorithm"] == "rdd_fast_bv")])


df_opt1_05 = (experiments_threshold_05_df
              [experiments_threshold_05_df["algorithm"] == "rdd_fast"])


df_opt2_05 = (experiments_threshold_05_df
              [experiments_threshold_05_df["algorithm"] == "rdd_fast_bv"])




## Add lines

# 1. df_bs_08
fig.add_trace(go.Scatter(
    x = df_bs_08["k"],
    y = df_bs_08["time (sec)"],
    name="baseline"
))


# 2. df_opt1_08
fig.add_trace(go.Scatter(
    x = df_opt1_08["k"],
    y = df_opt1_08["time (sec)"],
    name="optimization 1, threshold=0.8",
    line = dict(color='firebrick')
))

# 3. df_opt2_08
fig.add_trace(go.Scatter(
    x = df_opt2_08["k"],
    y = df_opt2_08["time (sec)"],
    name="optimization 2, threshold=0.8",
    line = dict(color='#00CC96')
))


# 4. df_opt1_05
fig.add_trace(go.Scatter(
    x = df_opt1_05["k"],
    y = df_opt1_05["time (sec)"],
    name="optimization 1, threshold=0.5",
    line = dict(color='firebrick',dash='dash')
))


# 5. df_opt2_05
fig.add_trace(go.Scatter(
    x = df_opt2_05["k"],
    y = df_opt2_05["time (sec)"],
    name="optimization 2, threshold=0.5",
    line = dict(color='#00CC96',dash='dash')
))

fig.update_layout(
    title=f"Experiment times of RDD implementations for threshold=0.8 and threshold=0.5 | Normal Distribution",
    xaxis_title="k",
    yaxis_title="time (sec)",
)

fig.update_xaxes(type="log")

fig.show()<jupyter_output><empty_output><jupyter_text>Experiments (k = triangles + 1) Compare `optimization 1` and `optimization 2` (with and without broadcast variable) Αυτά τα πειράματα δεν έτρεξαν λόγω μνήμης.```java.lang.OutOfMemoryError```<jupyter_code># experiments_df[(experiments_df["cores"] == 8) &
#                (experiments_df["algorithm"] == "graphframe_bs")]
experiments_k_large_df
# Compare baseline implementations for k> total triangles
# for Normal distribution

fig = go.Figure()


## Add Bars

# 1. graphframe_bs
fig.add_trace(go.Bar(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_bs", "distribution"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_bs", "time (sec)"].values,
    name="Baseline"
))


# 2. graphframe_fast
fig.add_trace(go.Bar(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_fast", "distribution"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_fast", "time (sec)"].values,
    name="Optimization 1"
))


# 3. graphframe_fast_bv
fig.add_trace(go.Bar(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_fast_bv", "distribution"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_fast_bv", "time (sec)"].values,
    name="Optimization 2"
))


fig.update_layout(
    title=f"Stress test: GraphFrames implementations for k > total number of triangles",
    xaxis_title="distribution",
    yaxis_title="time (sec)",
)

fig.show()
# Compare baseline implementations for k> total triangles
# for Normal distribution

fig = go.Figure()


## Add Bars

# 1. graphframe_bs
fig.add_trace(go.Bar(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_bs", "distribution"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_bs", "time (sec)"].values,
    name="Baseline"
))


# 2. graphframe_fast
fig.add_trace(go.Bar(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_fast", "distribution"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_fast", "time (sec)"].values,
    name="Optimization 1"
))


# 3. graphframe_fast_bv
fig.add_trace(go.Bar(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_fast_bv", "distribution"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_fast_bv", "time (sec)"].values,
    name="Optimization 2"
))


fig.update_layout(
    title=f"Stress test: RDD implementations for k > total number of triangles",
    xaxis_title="distribution",
    yaxis_title="time (sec)",
)

fig.show()<jupyter_output><empty_output><jupyter_text>GraphFrames vs RDD<jupyter_code># 2. Compare graphframes implementations (All distributions)
filtered_dataset = experiments_df[(experiments_df["cores"] == 8) &
                                  (experiments_df["algorithm"].isin(["graphframe_fast",
                                                                     "rdd_fast",
                                                                     "rdd_fast_bv"]))].copy()


filtered_dataset["algorithm"] = (filtered_dataset["algorithm"]
                                 .replace({"graphframe_fast":"optimization 1 - GraphFrames",
                                           "rdd_fast":"optimization 1 - RDD",
                                           "rdd_fast_bv":"optimization 2 - RDD"}))


fig = px.line(filtered_dataset,
                 x="k",
                 y="time (sec)",
                 log_x=True,
                 log_y=True,
                 facet_col="distribution",
                 color="algorithm")

fig.update_layout(
    title=f"Results comparison: GraphFrames vs RDD implementations"
)


for i in range(len(fig.data)):
    fig.data[i].update(mode='markers+lines')

fig.update_layout(
    autosize=False,
    width=1000,
    height=500)
fig.show()
## Compare graphframes_fast and graphframes_fast_bv 
# with threshold=0.8 and threshold=0.5 for Normal distribution

fig = go.Figure()


## --> Extract times:
# bs, optimization 1, optimization 2
# threshold 0.5, 0.8

df_bs_08 = (experiments_df
            [(experiments_df["cores"]==8) &
             (experiments_df["dataset"]=="artists_normal") &             
             (experiments_df["algorithm"] == "graphframe_bs")])

df_opt1_08 = (experiments_df
            [(experiments_df["cores"]==8) &
             (experiments_df["dataset"]=="artists_normal") &             
             (experiments_df["algorithm"] == "graphframe_fast")])

df_opt2_08 = (experiments_df
             [(experiments_df["cores"]==8) &
             (experiments_df["dataset"]=="artists_normal") &             
             (experiments_df["algorithm"] == "graphframe_fast_bv")])


df_opt1_05 = (experiments_threshold_05_df
              [experiments_threshold_05_df["algorithm"] == "graphframe_fast"])


df_opt2_05 = (experiments_threshold_05_df
              [experiments_threshold_05_df["algorithm"] == "graphframe_fast_bv"])




## Add lines

# 1. df_bs_08
fig.add_trace(go.Scatter(
    x = df_bs_08["k"],
    y = df_bs_08["time (sec)"],
    name="baseline"
))


# 2. df_opt1_08
fig.add_trace(go.Scatter(
    x = df_opt1_08["k"],
    y = df_opt1_08["time (sec)"],
    name="optimization 1, threshold=0.8",
    line = dict(color='firebrick')
))

# 3. df_opt2_08
fig.add_trace(go.Scatter(
    x = df_opt2_08["k"],
    y = df_opt2_08["time (sec)"],
    name="optimization 2, threshold=0.8",
    line = dict(color='#00CC96')
))


# 4. df_opt1_05
fig.add_trace(go.Scatter(
    x = df_opt1_05["k"],
    y = df_opt1_05["time (sec)"],
    name="optimization 1, threshold=0.5",
    line = dict(color='firebrick',dash='dash')
))


# 5. df_opt2_05
fig.add_trace(go.Scatter(
    x = df_opt2_05["k"],
    y = df_opt2_05["time (sec)"],
    name="optimization 2, threshold=0.5",
    line = dict(color='#00CC96',dash='dash')
))


fig.update_xaxes(type="log")


fig.update_layout(
    title=f"Results comparison: GraphFrames vs RDD",
    xaxis_title="distribution",
    yaxis_title="time (sec)",
)

fig.show()<jupyter_output><empty_output><jupyter_text>Legacy plots<jupyter_code>## Compare rdd_fast and rdd_fast_bv
# with threshold=0.8 and threshold=0.5 for Normal distribution

fig = go.Figure()


## Add lines

# 1. graphframe_fast
fig.add_trace(go.Scatter(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_fast", "dataset"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_fast", "time (sec)"].values,
    name="Graphframes, optimization 1",
    line = dict(color='#00CC96')
))


# 2. graphframe_fast_bv
fig.add_trace(go.Scatter(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_fast_bv", "dataset"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="graphframe_fast_bv", "time (sec)"].values,
    name="Graphframes, optimization 2",
    line = dict(color='#00CC96',dash='dash')
))


# 1. rdd_fast
fig.add_trace(go.Scatter(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_fast", "dataset"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_fast", "time (sec)"].values,
    name="RDD, optimization 1",
    line = dict(color='royalblue')
))


# 2. rdd_fast_bv
fig.add_trace(go.Scatter(
    x = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_fast_bv", "dataset"].values,
    y = experiments_k_large_df.loc[experiments_k_large_df["algorithm"]=="rdd_fast_bv", "time (sec)"].values,
    name="RDD, optimization 2",
    line = dict(color='royalblue',dash='dash')
))



fig.update_layout(
    title=f"Compare Optimization 1 - Optimization 2 (Broadcast variable) | k > total number of triangles",
    xaxis_title="distribution",
    yaxis_title="time (secs.)",
)

fig.show()
# 
# for cores=8 

filtered_dataset = experiments_df[(experiments_df["cores"]==8) &
                                  (experiments_df["algorithm"].isin(["graphframe_fast", "graphframe_fast_bv"]))]

fig = px.line(filtered_dataset,
                 x="k",
                 y="time (secs.)",
                 log_x=True,
                 log_y=True,
                 color="algorithm",
                 facet_col="dataset")

fig.update_layout(
    title=f"Test",
)

for i in range(len(fig.data)):
    fig.data[i].update(mode='markers+lines')
    
fig.show()
# fig.for_each_annotation(lambda a: a.update(text="time (sec.)"))
# 1. Compare graphframes implementations (Single distribution)
filtered_dataset = experiments_df[(experiments_df["dataset"]=="artists_uniform") &
                                  (experiments_df["script"].isin(["graphframe_bs.py",
                                                                  "graphframe_fast.py",
                                                                  "graphframe_fast_bv.py"]))]

fig = px.line(filtered_dataset,
                 x="k",
                 y="time",
                 log_x=True,
                 log_y=True,
                 color="script",
                 facet_col="cores")

fig.update_layout(
    title=f"Results of graphframes implementations - Uniform edge-probability distribution",
    yaxis_title="time (sec.)",
)


for i in range(len(fig.data)):
    fig.data[i].update(mode='markers+lines')
    
fig.show()
# Compare graphframe_fast,graphframe_fast_bv, rdd_fast,rdd_fast_bv implementations
# for cores=8 

filtered_dataset = experiments_df[(experiments_df["cores"]==8) &
                                  (experiments_df["script"].isin(["graphframe_fast.py",
                                                                  "graphframe_fast_bv.py",
                                                                  "rdd_fast.py",
                                                                  "rdd_fast_bv.py"]))]

fig = px.line(filtered_dataset,
                 x="k",
                 y="time",
                 log_x=True,
                 log_y=True,
                 color="script",
                 facet_col="dataset")

fig.update_layout(
    title=f"graphframe_fast vs rdd_vs implementations for 8 cores",
)

for i in range(len(fig.data)):
    fig.data[i].update(mode='markers+lines')
    
fig.show()
# 2. Compare rdd implementations
filtered_dataset = experiments_df[(experiments_df["dataset"]=="artists_uniform") &
                                  (experiments_df["script"].isin(["rdd_bs.py",
                                                                  "rdd_fast.py",
                                                                  "rdd_fast_bv.py"]))]

fig = px.line(filtered_dataset,
                 x="k",
                 y="time",
                 log_x=True,
                 log_y=True,
                 color="script",
                 facet_col="cores",
                 line_dash_map="dataset")

fig.update_layout(
    title=f"Graphframes implementations for uniform edge distribution",
)

for i in range(len(fig.data)):
    fig.data[i].update(mode='markers+lines')
    
fig.show()
# 2. Compare distributions for graphframes implementations
filtered_dataset = experiments_df[(experiments_df["cores"]==8) &
                                  (experiments_df["script"].isin(["graphframe_bs.py",
                                                                  "graphframe_fast.py",
                                                                  "graphframe_fast_bv.py"]))]

fig = px.line(filtered_dataset,
                 x="k",
                 y="time",
                 log_x=True,
                 log_y=True,
                 color="script",
                 facet_col="dataset")

fig.update_layout(
    title=f"Compare distributions for graphframes implementations and cores=8",
)

for i in range(len(fig.data)):
    fig.data[i].update(mode='markers+lines')
    
fig.show()
# 2. Compare distributions for rdd implementation
filtered_dataset = experiments_df[(experiments_df["cores"]==8) &
                                  (experiments_df["script"].isin(["rdd_bs.py",
                                                                  "rdd_fast.py",
                                                                  "rdd_fast_bv.py"]))]

fig = px.line(filtered_dataset,
                 x="k",
                 y="time",
                 log_x=True,
                 log_y=True,
                 color="script",
                 facet_col="dataset")

fig.update_layout(
    title=f"Graphframes implementations for uniform edge distribution",
)

for i in range(len(fig.data)):
    fig.data[i].update(mode='markers+lines')
    
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14126326500944905, 'fraction_numerical': 0.021914522459659833, 'mean_word_length': 3.217811158798283, 'pattern_counts': {'":': 18, '<': 66, '<?xml version=': 0, '>': 74, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 71, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '596375', 'license': 'MIT', 'n_tokens_mistral': 9464, 'n_tokens_neox': 8879, 'n_words': 1475, 'path': 'notebooks/EDA_graph_data.ipynb', 'repo_name': 'papaemman/Mining-of-Massive-Datasets-AUTh', 'size': '27516'}