<jupyter_start><jupyter_text>Tutorial This is a brief tutorial of basic Metagraph usage.First, we import Metagraph:<jupyter_code>import metagraph as mg<jupyter_output><empty_output><jupyter_text>Inspecting Types and Available AlgorithmsThe default resolver automatically pulls in all registered Metagraph plugins.The default resolver also links itself with the `metagraph` namespace, allowingusers to ignore the default resolver in most cases.<jupyter_code>res = mg.resolver<jupyter_output><empty_output><jupyter_text>A hierarchy of available types is automatically added as properties on `res`.<jupyter_code>dir(res.types)<jupyter_output><empty_output><jupyter_text>Alternatively, simply access the types directly from `mg`.Most attributes of `res` are exposed as attributes on `mg` for convenience.<jupyter_code>dir(mg.types)<jupyter_output><empty_output><jupyter_text>Two important concepts in Metagraph are abstract types and concrete types. Abstract types describe a generic kind of data container with potentially many equivalent representations.Concrete types describe a specific data object which fits under the abstract type category.One can think of abstract types as data container specifications and concrete types as implementations of those specifications.For each abstract type, there are several concrete types.Within a single abstract type, all concrete types are able to represent equivalent data, but in a different format or data structure.Here we show the concrete types which represent `Graphs`:<jupyter_code>dir(mg.types.Graph)<jupyter_output><empty_output><jupyter_text>Available algorithms are listed under `mg.algos` and grouped by categories.Note that these are also available under `res.algos`.<jupyter_code>dir(mg.algos)
dir(mg.algos.traversal)<jupyter_output><empty_output><jupyter_text>Example UsageLet's see how to use Metagraph by first constructing a graph from an edge list.Begin with an input csv file representing an edge list and weights.<jupyter_code>data = """
Source,Destination,Weight
0,1,4
0,3,2
0,4,7
1,3,3
1,4,5
2,4,5
2,5,2
2,6,8
3,4,1
4,7,4
5,6,4
5,7,6
"""<jupyter_output><empty_output><jupyter_text>Read in the csv file and convert to a Pandas `DataFrame`.<jupyter_code>import pandas as pd
import io
csv_file = io.StringIO(data)
df = pd.read_csv(csv_file)<jupyter_output><empty_output><jupyter_text>This `DataFrame` represents a graph’s edges, but Metagraph doesn’t know that yet. To use the `DataFrame` within Metagraph, we first need to convert it into an `EdgeMap`.A `PandasEdgeMap` takes a `DataFrame` plus the labels of the columns representing source, destination, and weight. With these, Metagraph will know how to interpret the `DataFrame` as an `EdgeMap`.<jupyter_code>em = mg.wrappers.EdgeMap.PandasEdgeMap(df, 'Source', 'Destination', 'Weight', is_directed=False)
em.value<jupyter_output><empty_output><jupyter_text>Convert EdgeMap to a Graph`Graphs` and `EdgeMaps` have many similarities, but `Graphs` are more powerful. `Graphs` can have weights on the nodes, not just on the edges. `Graphs` can also have isolate nodes (nodes with no edges), which `EdgeMaps` cannot have.Most Metagraph algorithms take a `Graph` as input, so we will convert our `PandasEdgeMap` into a `Graph`. In this case, it will become a `NetworkXGraph`.<jupyter_code>g = mg.algos.util.graph.build(em)
g
g.value.edges(data=True)<jupyter_output><empty_output><jupyter_text>Translate to other Graph formatsBecause Metagraph knows how to interpret `g` as a `Graph`, we can easily convert it other `Graph` formats.Let's convert it to a `ScipyGraph`. This format stores the edges and weights in a scipy.sparse matrix along with a numpy array mapping the position to a NodeId (in case the nodes are not sequential from 0..n). Any node weights are stored in a separate numpy array.<jupyter_code>g2 = mg.translate(g, mg.wrappers.Graph.ScipyGraph)
g2<jupyter_output><empty_output><jupyter_text>The matrix is accessed using `g2.value`. The node list is accessed using `.node_list`.We can verify the weighs and edges by inspecting the sparse adjacency matrix directly.<jupyter_code>g2.value.toarray()<jupyter_output><empty_output><jupyter_text>We can also convert `g` into an adjacency matrix representation using a `GrblasGraph`. This also stores the edges and node weights separately.<jupyter_code>g3 = mg.translate(g, mg.types.Graph.GrblasGraphType)
g3
g3.value<jupyter_output><empty_output><jupyter_text>We can also visualize the graph.<jupyter_code>import grblas
grblas.io.draw(g3.value)<jupyter_output><empty_output><jupyter_text>Inspect the steps required for translationsRather than actually converting `g` into other formats, let’s ask Metagraph how it will do the conversion. Each conversion requires a translator (written by plugin developers) to convert between the two formats. However, even if there isn’t a direct translator between two formats, Metagraph will find a path and take several translation steps as needed to perform the task.The mechanism for viewing the plan is to invoke the translation from ``mg.plan.translate`` rather than ``mg.translate``. Other than the additional ``.plan``, the call signature is identical.In this first example, there is a direct function which translates between `NetworkXGraphType` and `ScipyGraphType`.<jupyter_code>mg.plan.translate(g, mg.types.Graph.ScipyGraphType)<jupyter_output><empty_output><jupyter_text>---In this next example, there is no direct function which convert `NetworkXGraphType` into a `GrblasGraphType`. Instead, we have to first convert to `ScipyGraphType` and then to `GrblasGraphType` before finally arriving at our desired format.While Metagraph will do the conversion automatically, understanding the steps involved helps users plan for expected computation time and memory usage. If needed, plugin developers can write a plugin to provide a direct translation path.<jupyter_code>mg.plan.translate(g, mg.types.Graph.GrblasGraphType)<jupyter_output><empty_output><jupyter_text>Algorithm Example 1: Breadth First SearchAlgorithms are described initially in an abstract definition. For bfs_iter, we take a `Graph` and return a `Vector` indicating the NodeIDs in the order visited.After the abstract definition is written, multiple concrete implementations are written to operate on concrete types.Let's look at the signature and specific implementations available for bfs_iter.<jupyter_code>mg.algos.traversal.bfs_iter.signatures<jupyter_output>Signature:
	(graph: Graph({}), source_node: NodeID, depth_limit: int = -1) -> Vector({})
Implementations:
	(graph: ScipyGraphType({}), source_node: NodeID, depth_limit: int = -1) -> NumpyVectorType({})
	(graph: NetworkXGraphType({}), source_node: NodeID, depth_limit: int = -1) -> NumpyVectorType({})<jupyter_text>We see that there are two implementations available, each with a different type of input graph.---Let's perform a breadth-first search with our different representations of `g`. We should get approximately the same answer no matter which implementation is chosen (same NodeIDs within each depth level of the traversal).<jupyter_code>cc = mg.algos.traversal.bfs_iter(g, 0)
cc
cc2 = mg.algos.traversal.bfs_iter(g2, 0)
cc2<jupyter_output><empty_output><jupyter_text>---Similar to how we can view the plan for translations, we can view the plan for algorithms.No translation is needed because we already have a concrete implementation which takes a `NetworkXGraph` as input.<jupyter_code>mg.plan.algos.traversal.bfs_iter(g, 0)<jupyter_output><empty_output><jupyter_text>---In the next example, `g2` also satisfies a concrete implementation, so no input translation is required.<jupyter_code>mg.plan.algos.traversal.bfs_iter(g2, 0)<jupyter_output><empty_output><jupyter_text>Algorithm Example 2: PagerankLet's look at the same pieces of information, but for pagerank. Pagerank takes a `Graph` and returns a `NodeMap` indicating the rank value of each node in the graph.First, let's verify the signature and the implementations available.We see that there are two implementations available, taking a `NetworkXGraph` or `GrblasGraph` as input.<jupyter_code>mg.algos.centrality.pagerank.signatures<jupyter_output>Signature:
	(graph: Graph({'edge_type': 'map', 'edge_dtype': ('float', 'int')}), damping: float = 0.85, maxiter: int = 50, tolerance: float = 1e-05) -> NodeMap({})
Implementations:
	(graph: NetworkXGraphType({}), damping: float = 0.85, maxiter: int = 50, tolerance: float = 1e-05) -> PythonNodeMapType({})
	(graph: GrblasGraphType({}), damping: float = 0.85, maxiter: int = 50, tolerance: float = 1e-05) -> GrblasNodeMapType({})<jupyter_text>---Let's look at the steps required in the plan if we start with a `ScipyGraph`. Then let's perform the computation.We see that the `ScipyGraph` will need to be translated to a `GrblasGraph` in order to call the algorithm. **Metagraph will do this for us automatically.**<jupyter_code>mg.plan.algos.centrality.pagerank(g2)
pr = mg.algos.centrality.pagerank(g2)
pr<jupyter_output><empty_output><jupyter_text>The result is a `GrblasNodeMap` which can be inspected by looking at the underlying `.value`.<jupyter_code>pr.value<jupyter_output><empty_output><jupyter_text>Let's translate it to a numpy array.<jupyter_code>pr_array = mg.translate(pr, mg.types.NodeMap.NumpyNodeMapType)
pr_array.value<jupyter_output><empty_output><jupyter_text>**Helpful tip**: The translation type can also be specified as a string<jupyter_code>mg.translate(pr, "NumpyNodeMap")<jupyter_output><empty_output><jupyter_text>Now let's verify that we get the same answer with the NetworkX implementation of Pagerank. We can ensure the NetworkX implementation is called by passing in a NetworkXGraph. Because no translationsare required, it will choose that implementation.The result is a `PythonNodeMapType`, which is simply a Python `dict`.<jupyter_code>pr2 = mg.algos.centrality.pagerank(g)
pr2<jupyter_output><empty_output><jupyter_text>Translate to a numpy array and verify the same results (within tolerance)<jupyter_code>pr2_array = mg.translate(pr2, "NumpyNodeMap")
pr2_array.value
abs(pr2_array - pr_array.value) < 1e-15<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08653182042427232, 'fraction_numerical': 0.009274790330537741, 'mean_word_length': 6.598200899550225, 'pattern_counts': {'":': 0, '<': 108, '<?xml version=': 0, '>': 113, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '46480', 'license': 'Apache-2.0', 'n_tokens_mistral': 3091, 'n_tokens_neox': 2998, 'n_words': 1093, 'path': 'docs/getting_started/tutorial.ipynb', 'repo_name': 'jim22k/metagraph', 'size': '10135'}