<jupyter_start><jupyter_text>Networks: structure, evolution & processes**Internet Analytics - Lab 2**---**Group:** W**Names:*** <NAME>* <NAME>* <NAME>--- Instructions*This is a template for part 1 of the lab. Clearly write your answers, comments and interpretations in Markodown cells. Don't forget that you can add $\LaTeX$ equations in these cells. Feel free to add or remove any cell.**Please properly comment your code. Code readability will be considered for grading. To avoid long cells of codes in the notebook, you can also embed long python functions and classes in a separate module. Don’t forget to hand in your module if that is the case. In multiple exercises, you are required to come up with your own method to solve various problems. Be creative and clearly motivate and explain your methods. Creativity and clarity will be considered for grading.* --- 2.1 Exploration of real networks Exercise 2.1<jupyter_code># need to import a few things
import networkx as nx
import matplotlib.pyplot as plt

# helper function
def plot_degree_distribution(G):
    degs = list()
    for n in G.nodes():
        deg = G.degree(n)
        if deg >= len(degs):
            degs = degs + [0] * (deg - len(degs) + 1)
        degs[deg] += 1
    return degs
# Part 1 : read the given file into a list
graph1 = nx.read_edgelist('../data/network1.csv', delimiter=',', nodetype=int, encoding="utf-8")
# Part 2 : Compute the number of edges and nodes in the graph.
print("Number of nodes :")
print(len(graph1.nodes()))
print("Number of edges :")
print(len(graph1.edges()))<jupyter_output><empty_output><jupyter_text>Exercise 2.2:  Node degree distribution<jupyter_code># helper function
def plot_degree_distribution(G):
    degs = list()
    for n in G.nodes():
        deg = G.degree(n)
        if deg >= len(degs):
            degs = degs + [0] * (deg - len(degs) + 1)
        degs[deg] += 1
    return degs
# compute the node degree distribution
degree_list = plot_degree_distribution(graph1)

plt.plot(degree_list)
plt.yscale('log')
plt.xscale('log')
plt.xlabel('Degree')
plt.ylabel('Number of nodes')
plt.title('Plot of node degrees for network1')
plt.show();<jupyter_output><empty_output><jupyter_text>The PlotThe plot we choose to show the node degree distribution is a log-log plot, because there are a lot of nodes and thus makes a graph hard to read without a log scale (as it becomes exponential, and is stuck to the axes). We can also see that the ditribution ressembles greatly a Pareto distribution, with a very big shape parameter (here 8000) and a very little scale parameter (here 1).  Exercise 2.3: Giant component<jupyter_code>nb_connected = nx.number_connected_components(graph1)
print("There are", nb_connected, "connected components, as returned by a method of the graph.");

giant = sorted(nx.connected_component_subgraphs(graph1), key=len)
print("The biggest component has ", len(giant[-1].nodes()),
      "nodes, when the second largest has",len(giant[-2].nodes()),"nodes. So there is a giant component")<jupyter_output><empty_output><jupyter_text>Exercise 2.4: Short paths & Small-worlds If we think about the fact that we have a giant component that is not connected to other components in any way, it is quite obvious that network1 is in fact not a small world, as there would be no way to get to another component from the giant component in six degrees.Nevertheless, let plot the shortest path length between the nodes of the graph. To keep the computation time we will only look for shortest path that have a length of at most 6.<jupyter_code>shorts = nx.all_pairs_shortest_path_length(graph1,6)
short_distrib = list()
let_six = 0; """number of node pair with a shortest path lesser than 6"""
shortest_path_count = 0;
for component in giant:
    node_count = len(component.nodes());
    shortest_path_count += (node_count**2 - node_count)/2
for n in shorts.values():
    let_six += len(n.values())
    for deg in n.values():
        if deg >= len(short_distrib):
            short_distrib = short_distrib + [0] * (deg - len(short_distrib) + 1)
        short_distrib[deg] += 1
        
 
plt.plot(short_distrib)
plt.xlabel('Shortest path lenght')
plt.ylabel('Number of node pairs')
plt.title('Plot of the length of shortest path between any pair of node of network1')
plt.show();
print("In network1 ",let_six*100/shortest_path_count,"% of the shortest path have a length lesser or equal than 6.")<jupyter_output><empty_output><jupyter_text>It seem that the distribution of the shortest path length follow a normal law with a mean of 4. We can see that despite most nodse of the graph having a degree lesser than 3, the distance between two nodes is also lesser than 6 most of the time. If we forget than the graph count many dijoint component, i.e. exist some paire of node you cannot connect no matter how long your path is, we can say our graph is a small world.  Exercise 2.5: Network comparison<jupyter_code>#Reading the graph
graph2 = nx.read_edgelist('../data/network2.csv', delimiter=',', nodetype=int, encoding="utf-8")

#Computing its size
print("The network is composed of", len(graph2.nodes()), "nodes and", len(graph2.edges()), "edges.")

#Computing degree distribution
degree_list2 = plot_degree_distribution(graph2)
plt.plot(degree_list2)
plt.xlabel('Degree')
plt.ylabel('Number of nodes')
plt.title('Plot of node degrees for network2')
plt.show();

#Computing the connected components
nb_connected2 = nx.number_connected_components(graph2)
print("There are", nb_connected2, "connected components, as returned by a method of the graph.");

giant2 = sorted(nx.connected_component_subgraphs(graph2), key=len)
print("The biggest component has ", len(giant2[-1].nodes()),
      "nodes, when the second largest has",len(giant2[-2].nodes()),"nodes. So there is a giant component")

#Ploting shortest path
shorts2 = nx.all_pairs_shortest_path_length(graph2,6)
short_distrib2 = list()
let_six2 = 0; """number of node pair with a shortest path lesser than 6"""
shortest_path_count2 = 0;

for component in giant2:
    node_count2 = len(component.nodes());
    shortest_path_count2 += (node_count2**2 - node_count2)/2
    
for n in shorts2.values():
    let_six2 += len(n.values())
    for deg in n.values():
        if deg >= len(short_distrib2):
            short_distrib2 = short_distrib2 + [0] * (deg - len(short_distrib2) + 1)
        short_distrib2[deg] += 1
        
 
plt.plot(short_distrib)
plt.xlabel('Shortest path lenght')
plt.ylabel('Number of node pairs')
plt.title('Plot of the length of shortest path between any pair of node of network2')
plt.show();
print("In network2 ",let_six2*100/shortest_path_count2,"% of the shortest path have a length lesser or equal than 6.")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10191366266132622, 'fraction_numerical': 0.016466399643969738, 'mean_word_length': 4.499184339314845, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '622311', 'license': 'MIT', 'n_tokens_mistral': 2109, 'n_tokens_neox': 2003, 'n_words': 797, 'path': 'lab02/submision/ix-lab2/notebooks/ix-lab2-notebook-part1.ipynb', 'repo_name': 'Battleman/InternetAnalyticsW', 'size': '6762'}