@inproceedings{
futrell2023natural,
title={Natural Language Systematicity from a Constraint on Excess Entropy},
author={Richard Futrell},
booktitle={NeurIPS 2023 workshop: Information-Theoretic Principles in Cognitive Systems},
year={2023},
url={https://openreview.net/forum?id=52jORfVm1S}
}
Natural language is systematic: utterances are composed of individually meaningful parts which are typically concatenated together. I argue that natural-language-like systematicity arises in codes when they are constrained by excess entropy, the mutual information between the past and the future of a process. In three examples, I show that codes with natural-language-like systematicity have lower excess entropy than matched alternatives.
Natural Language Systematicity from a Constraint on Excess Entropy
Richard Futrell