<jupyter_start><jupyter_text>Research Question 1 - Order-Invariant AssumptionSince the deep-learning model receives knowledge paths without any rank-related information, intuitively, the attention mechanism can only judge the quality of a knowledge path based on its content. Therefore, the ordering of the knowledge paths should not impact the dependent variable. In other words, the attention mechanism will not favor knowledge paths in certain positions; this could occur when the position's average quality of the knowledge path is systematically higher than others' average quality.Based on intuition, the project has held the order-invariant assumption during the development of other systems. This analysis aims to verify the validity of this assumption.<jupyter_code>import pickle

with open("data/human.pkl", "rb") as human, open("data/comet.pkl", "rb") as comet:
    experiments = pickle.load(human) + pickle.load(comet)

print(f"Total number of experiments: {len(experiments)}")<jupyter_output>Total number of experiments: 14<jupyter_text>`float32`, used by the system, has a precision of up to 7 significant figures. Under the order-invariant assumption, all observations in a sample should converge to a single value within the theoretical limitation of `float32`.<jupyter_code>def float32_precision_test(observations):
    observations = observations.apply(lambda x: float(format(x, ".7g")))
    
    return observations.nunique() == 1

def float32_aggrement_test(observations):
    for digit in range(7, -1, -1):
        if observations.apply(lambda x: float(format(x, f".{digit}g"))).nunique() == 1:
            return digit<jupyter_output><empty_output><jupyter_text>Analysis on Prediction Vector<jupyter_code>import pandas as pd

HUMAN_NEEDS = ("status", "approval", "tranquility", "competition", "health", "family", "romance", "food",
               "independence","power", "order", "curiosity", "serenity", "honor", "belonging", "contact",
               "savings", "idealism", "rest")

predictions = pd.DataFrame(
    ((e.story, e.sentence, e.character, i.human_size, i.machine_size, need, value)
     for e in experiments for i in e.instances for s in i.samples
     for need, value in zip(HUMAN_NEEDS, s.result)),
    columns=("story", "line", "character", "human_size", "machine_size", "need", "value")
)

predictions<jupyter_output><empty_output><jupyter_text>Since this analysis only focuses on the system's behavior, the effects of human-like knowledge paths are removed (`human_size == 0`); since this analysis only focuses on the order of knowledge paths, the effects of knowledge path selection are removed (`machine_size == machine_size.max()`).<jupyter_code>predictions = predictions.groupby(["story", "line", "character"])\
    .apply(lambda x: x[(x.human_size == 0) & (x.machine_size == x.machine_size.max())])\
    .reset_index(drop=True)[["story", "line", "character", "need", "value"]]

predictions<jupyter_output><empty_output><jupyter_text>Float 32 TestThe test described above is performed for each inference instance on each human need category. The result indicates whether the observations follow the order-invariant assumption or not.<jupyter_code>predictions.groupby(["story", "line", "character", "need"]).value\
    .apply(float32_precision_test)\
    .to_frame("expected")\
    .pivot_table("expected", ["story", "line", "character"], "need")
predictions.groupby(["story", "line", "character", "need"]).value\
    .apply(float32_aggrement_test)\
    .rename("aggrement")\
    .value_counts()\
    .plot(kind="pie", title="Aggrement on N Significant Figures")<jupyter_output><empty_output><jupyter_text>Base on the result, not all observations converge within the limit of 7 significant figures - invalidating the order-invariant assumption for prediction vector. However, all observations converge within the limit of 4 significant figures. Thus, although there is a detectable influence, it does not significantly impact the model's prediction.  Analysis on Attention<jupyter_code>import itertools

attentions = pd.DataFrame(
    ((e.story, e.sentence, e.character, i.human_size, i.machine_size, p.content, p.index, p.weight)
     for e in experiments for i in e.instances for s in i.samples
     for p in itertools.chain(s.human_paths, s.machine_paths)
     if i.human_size == 0),
    columns=("story", "line", "character", "human_size", "machine_size", "path", "index", "value")
)

attentions
attentions = attentions.groupby(["story", "line", "character"])\
    .apply(lambda x: x[(x.human_size == 0) & (x.machine_size == x.machine_size.max())])\
    .reset_index(drop=True)[["story", "line", "character", "path", "index", "value"]]

attentions<jupyter_output><empty_output><jupyter_text>Float 32 TestThe test described above is performed for each inference instance on each knowledge path. The result indicates whether the observations follow the order-invariant assumption or not.<jupyter_code>attentions.groupby(["story", "line", "character", "path"]).value\
    .apply(float32_precision_test)\
    .to_frame("expected")
attentions.groupby(["story", "line", "character", "path"]).value\
    .apply(float32_aggrement_test)\
    .rename("aggrement")\
    .value_counts()\
    .plot(kind="pie", title="Aggrement on N Significant Figures")<jupyter_output><empty_output><jupyter_text>Base on the result, not all observations converge within the limit of 7 significant figures - invalidating the order-invariant assumption for prediction vector. However, all observations converge within the limit of 5 significant figures. Thus, although there is a detectable influence, it does not significantly impact the model's prediction.  Preference Based on PositionAs described above, the attention mechanism may favor knowledge path certain positions. This analysis investigates the validity of the statement. [The distance correlation test](https://en.wikipedia.org/wiki/Distance_correlation) is used since no assumption on the type of correlation (i.e., linear, quadratic, exponential, etc.) should be made.<jupyter_code>from dcor.independence import distance_covariance_test

def classification(p):
    if p < 0.001:
        return "Significant at 0.001"
    elif p < 0.01:
        return "Significant at 0.01"
    elif p < 0.05:
        return "Significant at 0.05"
    
    return "Not significant"

attentions.groupby(["story", "line", "character", "path"])\
    .apply(lambda x: distance_covariance_test(x.index, x.value)[0])\
    .apply(classification)\
    .rename("significance")\
    .value_counts()\
    .plot(kind="pie", title="Distance Correlation Test on Path's Index and Path's Attention")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1168189306574809, 'fraction_numerical': 0.00883630372921971, 'mean_word_length': 5.166204986149585, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '317579', 'license': 'MIT', 'n_tokens_mistral': 1988, 'n_tokens_neox': 1892, 'n_words': 643, 'path': 'Research Question 1.ipynb', 'repo_name': 'Dralan1912/HAICOR-Experiment', 'size': '6677'}