This paper addresses the problem of determining the distance between two regular languages. It will show how to expand Jaccard distance, which works on finite sets, to potentially-infinite regular languages. The entropy of a regular language plays a large role in the extension. Much of the paper is spent investigating the entropy of a regular language. This includes addressing issues that have required previous authors to rely on the upper limit of Shannon's traditional formulation of entropy, because its limit does not always exist. The paper also includes proposing a new limit based formulation for the entropy of a regular language and proves that formulation to both exist and be equivalent to Shannon's original formulation (when it exists). Additionally, the proposed formulation is shown to equal an analogous but formally quite different notion of topological entropy from Symbolic Dynamics -- consequently also showing Shannon's original formulation to be equivalent to topological entropy. Surprisingly, the natural Jaccard-like entropy distance is trivial in most cases. Instead, the entropy sum distance metric is suggested, and shown to be granular in certain situations.
Chomsky and Miller's seminal paper on regular languages @cite_21 does not address distances between regular languages. It uses Shannon's notion of channel capacity (equation 7 from @cite_21 ) for the entropy of a regular language: @math
@cite_21
1998314653 2035875679 2144638632 2121189333
We use an information-theoretic notion, namely, (Shannon) information rate, to generalize common syntactic similarity metrics (like Hamming distance and longest common subsequences) between strings to ones between languages. We show that the similarity metrics between two regular languages are computable. We further study self-similarity of a regular language under various similarity metrics. As far as semantic similarity is concerned, we study the amplitude of an automaton, which intuitively characterizes how much a typical execution of the automaton fluctuates. Finally, we investigate, through experiments, how to measure similarity between two real-world programs using Lempel-Ziv compression on the runs at the assembly level. Let L be an irreducible regular language. Let W be a non-empty set of words (or sub-words) of L and denote by LW = v ∈ L:w ⊏ v, ∀w ∈ W the language obtained from L by forbidding all the words w in W. Then the entropy decreases strictly: ent(LW) < ent(L). In this note we present a new proof of this fact, based on a method of Gromov, which avoids the Perron-Frobenius theory. This result applies to the regular languages of finitely generated free groups and an additional application is presented. We show how appropriately chosen functions f which we call distinguishing can be used to make deterministic finite automata backward deterministic. This idea can be exploited to design regular language classes called f-distinguishable which are identifiable in the limit from positive samples. Special cases of this approach are the k-reversible and terminal distinguishable languages, as discussed in Angluin (J. Assoc. Comput. Mach. 29 (3) (1982) 741), Fernau (Technical Report WSI-99-23, Universitat Tubingen (Germany), Wilhelm-Schickard-Institut fur Informatik, 1999, Short version published in the proceedings of AMAI 2000, see http: rutcor.rutgers.edu amai aimath00 AcceptedCont.htm, Proc. 15th Internat. Conf. on Pattern Recognition (ICPR 2000), Vol. 2, IEEE Press, New York, 2000, pp. 125?128), Radhakrishnan (Ph.D. Thesis, Department of Computer Science and Engineering, Indian Institute of Technology, Bombay, India, 1987), Radhakrishnan and Nagaraja (IEEE Trans. Systems, Man Cybernet. 17 (6) (1987) 982). Moreover, we show that all regular languages may be approximated in the setting introduced by Kobayashi and Yokomori (in: K. P. Jantke, T. Shinohara, Th. Zeugmann (Eds.), Proc. Sixth Internat. Conf. Algorithmic Learning Theory (ALT?95), Lecture Notes in Computer Science Lecture Notes in Artificial Intelligence, Vol. 997, Springer, Berlin, 1995, pp. 298?312), (Theoret. Comput. Sci. 174 (1997) 251?257) by any class of f-distinguishable languages. Observe that the class of all function-distinguishable languages is equal to the class of regular languages. In this paper, we consider the capacity C of the binary deletion channel for the limiting case where the deletion probability p goes to 0. It is known that for any p < 1 2, the capacity satisfies C ≥ 1−H(p), where H is the standard binary entropy. We show that this lower bound is essentially tight in the limit, by providing an upper bound C ≤ 1−(1−o(1))H(p), where the o(1) term is understood to be vanishing as p goes to 0. Our proof utilizes a natural counting argument that should prove helpful in analyzing related channels.