<jupyter_start><jupyter_text>Describing the dataIn this notebook, we quickly describe the data of the experimental results.This data is extracted from an sqlite database much more data is contained but it is somehow messy because it evolved with the project.  InstancesThe full instances can be found in the sqlite database.Here we only save some general information.* The graph id which can also be used to get the actual instance from the database* The number of vertices in the instance* The number of edges in the instance* The type of the instance (random or celestial)<jupyter_code>import pandas as pd
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
import math
sns.set_theme() # we use seaborn for beautiful and simple plots
instances = pd.read_json("./data/instances.json")
instances.sample(5)<jupyter_output><empty_output><jupyter_text>SolutionsThe full solutions can again be found in the sqlite database using the solution id.Here we only state the important information about it.* The graph id which denotes for which instance this is a solution* The solution id which can be used to retrieve the actual solution from the database* The objective values for MS, TE, and BE.* The general technique of the solver* The concrete solver (including the objective)* The solution type (the objective it has been optimized for)* The date the solution has been computed* The runtime it took to compute it* If it has an optimality certificate (only possible for MIPs or CPs)The data has already been cleared of duplicates. For every solver and instance, only the last entry is available.<jupyter_code>solutions = pd.read_json("./data/solutions.json")
solutions.sample(10)<jupyter_output><empty_output><jupyter_text>Getting a quick overview of the instance distributionIt is always good to know, how the instances are distributed.For this, we filter the instance for those that are actually solved.<jupyter_code>must_have_solvers = ["CP-1 (MS)"]#, "GA (TE)"]
solvers_for_graph_id = solutions.groupby(["graph_id"])["solver"].unique()
def are_superset(s):
    return set(must_have_solvers).issubset(set(s))
graph_ids_to_consider = list(solvers_for_graph_id[solvers_for_graph_id.apply(are_superset)].index)
results = solutions.merge(instances, left_on="graph_id", right_on="graph_id")[["graph_id", "instance_type", "MS", "TE", "BE", "edges_amount", "vertices_amount"]]
results = results[results["graph_id"].isin(graph_ids_to_consider)]
results = results.groupby(["graph_id"]).min()
results
important_instances = instances[instances["graph_id"].isin(set(results.index))]
import networkx as nx
plt.figure(figsize=(5,3.6))
plt.title("Instance distribution")

t = important_instances.copy()
t.rename(inplace=True, columns={"instance_type": "Type"})
t.sort_values(["Type"], inplace=True)
sns.scatterplot(data=t, y="edges_amount", x="vertices_amount", hue="Type", size="Type", alpha=0.8, sizes=[15, 7])

ns = list(t["vertices_amount"].unique())
ns.sort()
ns = pd.DataFrame(index=ns, data={"100%": [nx.complete_graph(n).number_of_edges() for n in ns], "50%": [0.5*nx.complete_graph(n).number_of_edges() for n in ns]})
sns.lineplot(data=ns, palette=["grey", "grey"])
plt.xlabel("Number of vertices")
plt.ylabel("Number of edges")
plt.title("")
plt.ylim(0, 800)
plt.legend(loc='lower right', framealpha=0.5)
plt.tight_layout(pad=0.01)
#plt.tight_layout()
plt.savefig("./output/plot_instance_distribution.pdf")
plt.savefig("./output/plot_instance_distribution.png", dpi=300)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11429381735677822, 'fraction_numerical': 0.008791832104367556, 'mean_word_length': 6.734649122807017, 'pattern_counts': {'":': 3, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '350970', 'license': 'MIT', 'n_tokens_mistral': 1118, 'n_tokens_neox': 1063, 'n_words': 362, 'path': 'notebooks/00_describing_the_data.ipynb', 'repo_name': 'ahillbs/minimum_scan_cover', 'size': '3526'}