Selection models of language production support informed text partitioning: an intuitive and practical, bag of phrases framework for text analysis
The task of text segmentation, or 'chunking, may occur at many levels in text analysis, depending on whether it is most beneficial to break it down by paragraphs of a book, sentences of a paragraph, etc. Here, we focus on a fine grained segmentation task, which we refer to as text partitioning, where we apply methodologies to segment sentences or clauses into phrases, or lexical constructions of one or more words. In the past, we have explored (uniform) stochastic text partitioning a process on the gaps between words whereby each space assumes one from a binary state of fixed (word binding) or broken (word separating) by some probability. In that work, we narrowly explored perhaps the most naive version of this process: random, or, uniform stochastic partitioning, where all word word gaps are prescribed a uniformly set breakage probability, q. Under this framework, the breakage probability is a tunable parameter, and was set to be pure uniform: q 1/2. In this work, we explore phrase frequency distributions under variation of the parameter q, and define non uniform, or informed stochastic partitions, where q is a function of surrounding information. Using a crude but effective function for q, we go on to apply informed partitions to over 20,000 English texts from the Project Gutenberg eBooks database. In these analyses, we connect selection models to generate a notion of fit goodness for the 'bag of terms' (words or phrases) representations of texts, and find informed (phrase) partitions to be an improvement over the q 1 (word) and q 1/2 (phrase) partitions in most cases. This, together with the scalability of the methods proposed, suggests that the bag of phrases model should more often than not be implemented in place of the bag of words model, setting the stage for a paradigm shift in feature selection, which lies at the foundation of text analysis methodology.
ArXiv