This paper proposes a novel algorithm which learns a formal regular grammar from real-world continuous data, such as videos or other streaming data. Learning latent terminals, non-terminals, and productions rules directly from streaming data allows the construction of a generative model capturing sequential structures with multiple possibilities. Our model is fully differentiable, and provides easily interpretable results which are important in order to understand the learned structures. It outperforms the state-of-the-art on several challenging datasets and is more accurate for forecasting future activities in videos. We plan to open-source the code.
Chomsky grammars @cite_2 @cite_3 are designed to represent functional linguistic relationships. They have found wide applications in defining programming languages, natural language understanding, and understanding of images and videos .
@cite_3 @cite_2
2115912164 2279996368 1568155371 877909479
The first part of this article gives a brief overview of the four levels of the Chomsky hierarchy, with a special emphasis on context-free and regular languages. It then recapitulates the arguments why neither regular nor context-free grammar is sufficiently expressive to capture all phenomena in the natural language syntax. In the second part, two refinements of the Chomsky hierarchy are reviewed, which are both relevant to the extant research in cognitive science: the mildly context-sensitive languages (which are located between context-free and context-sensitive languages), and the sub-regular hierarchy (which distinguishes several levels of complexity within the class of regular languages). We study disambiguating of pronoun references in Winograd Schemas, which are part of the Winograd Schema Challenge, a proposed replacement for the Turing test. In particular we consider sentences where the pronoun can be resolved to both antecedents without semantic violations in world knowledge, that means for both readings of the sentence there is a possible consistent world. Nevertheless humans will strongly prefer one answer, which can be explained by pragmatic effects described in Relevance Theory. We state formal optimization criteria based on principles of Relevance Theory in a simplification of Roger Schank's graph framework for natural language understanding. We perform experiments using Answer Set Programming and report the usefulness of our criteria for disambiguation and their sensitivity to parameter variations. Combinatory Categorial Grammar (CCG) offers a new approach to the theory of natural language grammar. Coordination, relativization, and related prosodic phenomena have been analyzed in CCG in terms of a radically revised notion of surface structure. CCG surface structures do not exhibit traditional notions of syntactic dominance and command, and do not constitute an autonomous level of representation. Instead, they reflect the computations by which a sentence may be realized or analyzed, to synchronously define a predicate-argument structure, or logical form. Surface Structure and Interpretation shows that binding and control can be captured at this level, preserving the advantages of CCG as an account of coordination and unbounded dependency.The core of the book is a detailed treatment of extraction, a focus of syntactic research since the early work of Chomsky and Ross. The topics addressed include the sources of subject-object asymmetries and phenomena attributed to the Empty Category Principle (ECP), asymmetric islands, parasitic gaps, and the relation of coordination and extraction, including their interactions with binding theory. In his conclusion, the author relates CCG to other categorial and type-driven approaches and to proposals for minimalism in linguistic theory.Linguistic Inquiry Monograph No. 30 Recently, joint video-language modeling has been attracting more and more attention. However, most existing approaches focus on exploring the language model upon on a fixed visual model. In this paper, we propose a unified framework that jointly models video and the corresponding text sentences. The framework consists of three parts: a compositional semantics language model, a deep video model and a joint embedding model. In our language model, we propose a dependency-tree structure model that embeds sentence into a continuous vector space, which preserves visually grounded meanings and word order. In the visual model, we leverage deep neural networks to capture essential semantic information from videos. In the joint embedding model, we minimize the distance of the outputs of the deep video model and compositional language model in the joint space, and update these two models jointly. Based on these three parts, our system is able to accomplish three tasks: 1) natural language generation, and 2) video retrieval and 3) language retrieval. In the experiments, the results show our approach outperforms SVM, CRF and CCA baselines in predicting Subject-Verb-Object triplet and natural sentence generation, and is better than CCA in video retrieval and language retrieval tasks.