<jupyter_start><jupyter_text>HEAT CLUSTERMAP VISUALIZATIONS Given a tree structure, this notebook gets heat clustermap visualizations of it. In particular, given a jet (the set of its constituents), visualization for the different trees obtained with each clustering algorithm (and the truth tree) are displayed. There are 2 functions for visualization:a) function heat_dendrogram: Create  a heat dendrogram clustermap. If two jets are given as input:- 1) If only one jet is given as input, both rows and columns are ordered according to that jet tree.- 2) If truthJet and recluster_jet1, the visualization shows the heat data map of the truth jet, with columns ordered according to the truth jet and rows according to recluster_jet1.- 3) If recluster_jet1 and recluster_jet2, the visualization shows the heat data map of recluster_jet1, with columns ordered according to the recluster_jet1 and rows according to recluster_jet2.    b) function dendrogramDiff:    Given two jet algorithms heat data matrices, we reorder the heat matrices according to the truth jet order (order in which leaves are accessed when traversing the truth tree) and take the difference.    Both functions have the same arguments:- truthJet: Truth jet dictionary- recluster_jet1: reclustered jet 1- recluster_jet2: reclustered jet 2- full_path: Bool. If True, then use the total number of steps to connect a pair of leaves as the heat data. If False, then Given a pair of jet constituents {i,j} and the number of steps needed for each constituent to reach their closest common ancestor {Si,Sj}, the heat map scale represents the maximum number of steps, i.e. max{Si,Sj}.- FigName: Dir and location to save a plot.     Usage:- Input_dir: select the directory with the jets dictionaries.- Input_jet: select the jet filename for the visualizations.- Optional: input a name and location to save an image as the "outFilename" argument of the "heat_dendrogram" or "dendrogramDiff" functions.Note: The length of the connections among nodes for dendrogram diagrams shown at the sides of the clustermaps, is given by:- The distance measure d_ij between nodes, for the (Kt, CA, Antikt} algorithms - An integer starting at 0 and increasing by one each time a node is added, for the truth case.<jupyter_code># Filename and dir  to save a plot
# outFilename="plots/heatClustermap/figTruthTruth_singlepath_jet2.jpg"
#Data dir
input_dir= 'data/'
# input_dir ="../ToyJetsShower/data/"
# nput_jet = 'tree_13_truth'
input_jet = 'tree_10_truth'
with open(input_dir+ str(input_jet) + '.pkl', "rb") as fd: 
    truth_jet = pickle.load(fd, encoding='latin-1')[0]

jet_name = ('_').join(input_jet.split('_')[-3:-1])
truth_jet["name"]=jet_name
truth_jet["algorithm"]="truth"

display(truth_jet["name"])
# Number of jet constituents
linkageList.runTraverse_jet(truth_jet, draw_tree=True)
len(truth_jet["outers_list"])
# Recluster jet
recluster = True
if recluster:
    reclusterKt = reclusterTree.recluster(truth_jet, alpha=1)
    reclusterAntikt = reclusterTree.recluster(truth_jet, alpha=-1)
    reclusterCA = reclusterTree.recluster(truth_jet, alpha=0)
    
# Note: The variable alpha determines the clustering algorithm, with {1,0,-1} for the {Kt, CA, Antikt} algorithms respectively
# Load reclustered jets
with open(input_dir+ str(truth_jet["name"])+'_1' + '.pkl', "rb") as fd: reclusterKt = pickle.load(fd, encoding='latin-1')
with open(input_dir+ str(truth_jet["name"])+'_-1' + '.pkl', "rb") as fd: reclusterAntikt = pickle.load(fd, encoding='latin-1')
with open(input_dir+ str(truth_jet["name"])+'_0' + '.pkl', "rb") as fd: reclusterCA = pickle.load(fd, encoding='latin-1')<jupyter_output><empty_output><jupyter_text>SAME TREE BOTH IN ROWS AND COLUMNS<jupyter_code>heatClustermap.heat_dendrogram(
    truthJet=truth_jet,
    full_path=False, 
    FigName=None)
heatClustermap.heat_dendrogram(
    recluster_jet1=reclusterKt,
    full_path=False, 
    FigName=None)
heatClustermap.heat_dendrogram(
    recluster_jet1=reclusterCA,
    full_path=False, 
    FigName=None)
heatClustermap.heat_dendrogram(
    recluster_jet1=reclusterAntikt,
    full_path=False, 
    FigName=None)<jupyter_output>2019-08-05 12:58:20,050:INFO::alpha row: -1 -- alpha column: -1<jupyter_text>DIFFERENT TREE IN ROWS AND COLUMNS<jupyter_code>heatClustermap.heat_dendrogram(
    truthJet=truth_jet,
    recluster_jet1 = reclusterKt, 
    full_path=False, 
    FigName=None)
heatClustermap.heat_dendrogram(
    truthJet=truth_jet,
    recluster_jet1 = reclusterCA, 
    full_path=False, 
    FigName=None)
heatClustermap.heat_dendrogram(
    truthJet=truth_jet,
    recluster_jet1 = reclusterAntikt, 
    full_path=False, 
    FigName=None)
heatClustermap.heat_dendrogram(
    recluster_jet1 = reclusterKt, 
    recluster_jet2 = reclusterCA,
    full_path=False, 
    FigName=None)
heatClustermap.heat_dendrogram(
    recluster_jet1 = reclusterAntikt, 
    recluster_jet2 = reclusterKt,
    full_path=False, 
    FigName=None)<jupyter_output>2019-08-05 12:20:31,325:INFO::alpha row: 1 -- alpha column: -1<jupyter_text>DENDROGRAM DIFFERENCES Given two jet algorithms heat data matrices, we reorder the heat matrices according to the truth jet order (order in which leaves are accessed when traversing the truth tree) and take the difference. Values close to cero mean that the trees substructe is closer between trees.<jupyter_code>heatClustermap.dendrogramDiff(
    truthJet=truth_jet,
    recluster_jet1=reclusterKt,
    full_path=False, 
    FigName=None)
heatClustermap.dendrogramDiff(
    truthJet=truth_jet,
    recluster_jet1=reclusterCA,
    full_path=False, 
    FigName=None)
heatClustermap.dendrogramDiff(
    truthJet=truth_jet,
    recluster_jet1=reclusterAntikt,
    full_path=False, 
    FigName=None)
heatClustermap.dendrogramDiff(
    truthJet=None,
    recluster_jet1=reclusterCA,
    recluster_jet2 = reclusterKt,
    full_path=False, 
    FigName=None)
heatClustermap.dendrogramDiff(
    truthJet=None,
    recluster_jet1=reclusterKt,
    recluster_jet2 = reclusterAntikt,
    full_path=False, 
    FigName=None)<jupyter_output>2019-08-05 12:22:19,437:INFO::(recluster jet2 - recluster jet1) heat data
{'fraction_non_alphanumeric': 0.08501300390117035, 'fraction_numerical': 0.018042912873862157, 'mean_word_length': 5.146853146853147, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '325760', 'license': 'MIT', 'n_tokens_mistral': 2158, 'n_tokens_neox': 2000, 'n_words': 600, 'path': 'examples/visualizeClusterMap.ipynb', 'repo_name': 'SebastianMacaluso/ReclusterTreeAlgorithms', 'size': '6152'}