@inproceedings{tseng-etal-2024-semantic,
    title = "The Semantic Relations in {LLM}s: An Information-theoretic Compression Approach",
    author = "Tseng, Yu-Hsiang  and
      Chen, Pin-Er  and
      Lian, Da-Chen  and
      Hsieh, Shu-Kai",
    editor = "Dong, Tiansi  and
      Hinrichs, Erhard  and
      Han, Zhen  and
      Liu, Kang  and
      Song, Yangqiu  and
      Cao, Yixin  and
      Hempelmann, Christian F.  and
      Sifa, Rafet",
    booktitle = "Proceedings of the Workshop: Bridging Neurons and Symbols for Natural Language Processing and Knowledge Graphs Reasoning (NeusymBridge) @ LREC-COLING-2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.neusymbridge-1.2",
    pages = "8--21",
    abstract = "Compressibility is closely related to the predictability of the texts from the information theory viewpoint. As large language models (LLMs) are trained to maximize the conditional probabilities of upcoming words, they may capture the subtlety and nuances of the semantic constraints underlying the texts, and texts aligning with the encoded semantic constraints are more compressible than those that do not. This paper systematically tests whether and how LLMs can act as compressors of semantic pairs. Using semantic relations from English and Chinese Wordnet, we empirically demonstrate that texts with correct semantic pairings are more compressible than incorrect ones, measured by the proposed compression advantages index. We also show that, with the Pythia model suite and a fine-tuned model on Chinese Wordnet, compression capacities are modulated by the model{'}s seen data. These findings are consistent with the view that LLMs encode the semantic knowledge as underlying constraints learned from texts and can act as compressors of semantic information or potentially other structured knowledge.",
}

Compressibility is closely related to the predictability of the texts from the information theory viewpoint. As large language models (LLMs) are trained to maximize the conditional probabilities of upcoming words, they may capture the subtlety and nuances of the semantic constraints underlying the texts, and texts aligning with the encoded semantic constraints are more compressible than those that do not. This paper systematically tests whether and how LLMs can act as compressors of semantic pairs. Using semantic relations from English and Chinese Wordnet, we empirically demonstrate that texts with correct semantic pairings are more compressible than incorrect ones, measured by the proposed compression advantages index. We also show that, with the Pythia model suite and a fine-tuned model on Chinese Wordnet, compression capacities are modulated by the model{'}s seen data. These findings are consistent with the view that LLMs encode the semantic knowledge as underlying constraints learned from texts and can act as compressors of semantic information or potentially other structured knowledge.
Tseng, Yu-Hsiang Chen, Pin-Er Lian, Da-Chen Hsieh, Shu-Kai
The Semantic Relations in LLMs: An Information-theoretic Compression Approach