Recently, there has been a lot of interest in automatically generating descriptions for an image. Most existing language-model based approaches for this task learn to generate an image description word by word in its original word order. However, for humans, it is more natural to locate the objects and their relationships first, and then elaborate on each object, describing notable attributes. We present a coarse-to-fine method that decomposes the original image description into a skeleton sentence and its attributes, and generates the skeleton sentence and attribute phrases separately. By this decomposition, our method can generate more accurate and novel descriptions than the previous state-of-the-art. Experimental results on the MS-COCO and a larger scale Stock3M datasets show that our algorithm yields consistent improvements across different evaluation metrics, especially on the SPICE metric, which has much higher correlation with human ratings than the conventional metrics. Furthermore, our algorithm can generate descriptions with varied length, benefiting from the separate control of the skeleton and attributes. This enables image description generation that better accommodates user preferences.
Parsing of a sentence is the process of analyzing the sentence according to a set of grammar rules, and generating a rooted parse tree that represents the syntactic structure of the sentence @cite_1 . There is some language-model-based work that parses the captions for better sentence encoding. For example, Socher al @cite_17 proposed the Dependency Tree-RNN, which uses dependency trees to embed sentences into a vector space, and then performs caption retrieval with the embedded vector. Unfortunately, the model is unable to generate novel sentences.
@cite_1 @cite_17
2791908423 2121127625 2035585180 2296308987
Semantic parsing is the process of mapping a natural language sentence into a formal representation of its meaning. In this work we use the neural network approach to transform natural language sentence into a query to an ontology database in the SPARQL language. This method does not rely on handcraft-rules, high-quality lexicons, manually-built templates or other handmade complex structures. Our approach is based on vector space model and neural networks. The proposed model is based in two learning steps. The first step generates a vector representation for the sentence in natural language and SPARQL query. The second step uses this vector representation as input to a neural network (LSTM with attention mechanism) to generate a model able to encode natural language and decode SPARQL. This article considers approaches which rerank the output of an existing probabilistic parser. The base parser produces a set of candidate parses for each input sentence, with associated probabilities that define an initial ranking of these parses. A second model then attempts to improve upon this initial ranking, using additional features of the tree as evidence. The strength of our approach is that it allows a tree to be represented as an arbitrary set of features, without concerns about how these features interact or overlap and without the need to define a derivation or a generative model which takes these features into account. We introduce a new method for the reranking task, based on the boosting approach to ranking problems described in (1998). We apply the boosting method to parsing the Wall Street Journal treebank. The method combined the log-likelihood under a baseline model (that of Collins [1999]) with evidence from an additional 500,000 features over parse trees that were not included in the original model. The new model achieved 89.75 F-measure, a 13 relative decrease in F measure error over the baseline model's score of 88.2 . The article also introduces a new algorithm for the boosting approach which takes advantage of the sparsity of the feature space in the parsing data. Experiments show significant efficiency gains for the new algorithm over the obvious implementation of the boosting approach. We argue that the method is an appealing alternative—in terms of both simplicity and efficiency—to work on feature selection methods within log-linear (maximum-entropy) models. Although the experiments in this article are on natural language parsing (NLP), the approach should be applicable to many other NLP problems which are naturally framed as ranking tasks, for example, speech recognition, machine translation, or natural language generation. levels, such as Pet6fi's text grammar and Rumelhart's schemata for stories. By defining permissible interrelations between constituents of larger texts at all levels of abstraction, these create meaning structures organized on both hierarchical and associative principles, rather than on a sequential basis. The automated "understanding" process consists of instantiating an appropriate set of event templates by associating a given text with the ERGO inventory of event templates and const ucting a network of instanti ted templates based on intertemplate relational rules and constraints. The network of instantiated t mplates thus represents the information content of the unstructured original text, and provides the structured input for the event record data bas . Details of the automated understanding process presented here in the abstract are given in the follo ing section. 3.2 Processing Principles. The process of data base generation involves two major functions: (1) co tent analysis of the incoming text (2) event record synthesis (or production) The first involves constructing a meaning representation of the text and the second the extraction of relevant information and its storage in a data base record. The major focus of ERGO is on reports of a particular class of events which describe aircraft movements. The unit of analysis is therefore the report, a textual unit consisting of one or more par graphs, each containing one or more sentences. The first step in the analytical process involves a lexical lookup: a lexical entry contains morphological, syntactic and semantic information, on the lines of Sager (1973) and (1973). Each sentence is then subjected to a syntactic analysis by means of an Augmented Transition Network (ATN) parser (Woods, 1970). Since event templates are based on propositional structures, the analytical We present a two-stage framework to parse a sentence into its Abstract Meaning Representation (AMR). We first use a dependency parser to generate a dependency tree for the sentence. In the second stage, we design a novel transition-based algorithm that transforms the dependency tree to an AMR graph. There are several advantages with this approach. First, the dependency parser can be trained on a training set much larger than the training set for the tree-to-graph algorithm, resulting in a more accurate AMR parser overall. Our parser yields an improvement of 5 absolute in F-measure over the best previous result. Second, the actions that we design are linguistically intuitive and capture the regularities in the mapping between the dependency structure and the AMR of a sentence. Third, our parser runs in nearly linear time in practice in spite of a worst-case complexity ofO(n 2 ).