<jupyter_start><jupyter_text>Load in a single data file and visualize the tree and dataframe<jupyter_code># Path to a Caliper JSON-split file
caliper_file = "../data/lulesh-1node/lulesh-annotation-profile-1core-nompi.json"

# Read file into a Hatchet GraphFrame using from_caliper_json() API
gf = ht.GraphFrame.from_caliper_json(caliper_file)
# Display numeric metric columns (such as inclusive and exlusive time), which can be visualized on the tree
gf.show_metric_columns()
# Print the terminal tree representation using exclusive time
print(gf.tree(metric_column="time"))
# Get the man page of different parameters for Hatchet's tree function
help(gf.tree)
# Print the dataframe of metrics
display(gf.dataframe)<jupyter_output><empty_output><jupyter_text>Filter and squash a tree<jupyter_code># Filter the tree to keep nodes with a time value greater than 300,000 sec
filter_func = lambda x: x["time"] > 300e3
filter_gf = gf.filter(filter_func,
                      squash=False)
display(HTML(filter_gf.dataframe.to_html()))
squash_gf = filter_gf.squash()
# Compare size of input graph and filtered graph
print(f"Input Graph Size                       : {len(gf.graph)}")
print(f"Result (after filter/squash) Graph Size: {len(squash_gf.graph)}")
# Print the resulting tree
print(squash_gf.tree(metric_column="time"))
# Another filter example by function name, filter and squash in a single step
squash_gf = gf.filter(lambda x: x["name"] in ("main", "lulesh.cycle"))
print(squash_gf.tree(metric_column="time"))<jupyter_output><empty_output><jupyter_text>Filter using callpath query language<jupyter_code># Aggregate metrics across ranks (in this case, we only have data for rank 0, so this will
# drop the rank index column)
gf.drop_index_levels(function=np.max)

# Define call path query
query = [{"name": "LagrangeNodal"}, "*"]

# Filter and squash the tree
query_gf = gf.filter(query, squash=True)

# Print the resulting tree
print(query_gf.tree(metric_column="time"))<jupyter_output><empty_output><jupyter_text>Aggregate data across ranks and/or threads<jupyter_code># Path to a Caliper file
caliper_file = "../data/lulesh-1node/lulesh-annotation-profile-27cores.json"

# Read file into a Hatchet GraphFrame
gf = ht.GraphFrame.from_caliper_json(caliper_file)

display(HTML(gf.dataframe.to_html()))
gf.drop_index_levels(function=np.max)
display(gf.dataframe)
# Print the resulting dataframe of metrics
display(HTML(gf.dataframe.to_html()))<jupyter_output><empty_output><jupyter_text>Calculate speedup between two trees<jupyter_code># Add cali-query to PATH
cali_query_path = "Caliper-2.5.0/install/bin"
os.environ["PATH"] += os.pathsep + cali_query_path

# Define caliper query for filtering profile
grouping_attribute = "prop:nested"
default_metric = "avg#inclusive#sum#time.duration"
query = "select %s,sum(%s) group by %s format json-split" % (grouping_attribute, default_metric, grouping_attribute)
# Path to Caliper files
cali_file1 = "../data/cDPu64825TuLB5ujG_0.cali"
cali_file2 = "../data/cjDCIuaXAoayBi9Lr_2.cali"

# Read Caliper files into a Hatchet GraphFrame using from_caliper() API
gf = ht.GraphFrame.from_caliper(cali_file1, query)   # 1 rank
gf2 = ht.GraphFrame.from_caliper(cali_file2, query)  # 64 ranks
# Print the tree representation using the exclusive time metric
print(gf.tree(metric_column="time (inc)"))
# Print the tree representation using the exclusive time metric
print(gf2.tree(metric_column="time (inc)"))
# Compute the speedup of 1 rank to 64 ranks
gf3 = gf / gf2
# Print the resulting tree
# Two things to note here:
# 1) The MPI nodes are annotated with a green arrow that points to the right. This indicates that those
#    nodes exist only in the right tree (i.e., 64 ranks). By right tree, we are referring to the position
#    in the equation gf3 = gf / gf2 as shown in the above cell.
# 2) Nodes with good speedup are highlighted in red, but may be preferred to color these nodes in green.
print(gf3.tree(metric_column="time (inc)"))
# Print resulting tree, but reverse the color scheme, so red identifies nodes with poor scaling (low values)
print(gf3.tree(metric_column="time (inc)",
               invert_colormap=True))
# Print the resulting dataframe of metrics
display(gf3.dataframe)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09943714821763602, 'fraction_numerical': 0.011960600375234522, 'mean_word_length': 5.531393568147013, 'pattern_counts': {'":': 1, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '168882', 'license': 'MIT', 'n_tokens_mistral': 1445, 'n_tokens_neox': 1354, 'n_words': 429, 'path': 'notebooks/0-basic-api.ipynb', 'repo_name': 'mpokorny/hatchet-tutorial', 'size': '4264'}