We address two challenges of probabilistic topic modelling in order to better estimate
 the probability of a word in a given context, i.e., P(wordjcontext) : (1) No
 Language Structure in Context: Probabilistic topic models ignore word order by
 summarizing a given context as a “bag-of-word” and consequently the semantics
 of words in the context is lost. In this work, we incorporate language structure
 by combining a neural autoregressive topic model (TM) with a LSTM based language
 model (LSTM-LM) in a single probabilistic framework. The LSTM-LM
 learns a vector-space representation of each word by accounting for word order
 in local collocation patterns, while the TM simultaneously learns a latent representation
 from the entire document. In addition, the LSTM-LM models complex
 characteristics of language (e.g., syntax and semantics), while the TM discovers
 the underlying thematic structure in a collection of documents. We unite two complementary
 paradigms of learning the meaning of word occurrences by combining
 a topic model and a language model in a unified probabilistic framework, named
 as ctx-DocNADE. (2) Limited Context and/or Smaller training corpus of documents:
 In settings with a small number of word occurrences (i.e., lack of context)
 in short text or data sparsity in a corpus of few documents, the application of TMs
 is challenging. We address this challenge by incorporating external knowledge
 into neural autoregressive topic models via a language modelling approach: we
 use word embeddings as input of a LSTM-LM with the aim to improve the wordtopic
 mapping on a smaller and/or short-text corpus. The proposed DocNADE
 extension is named as ctx-DocNADEe.

 We present novel neural autoregressive topic model variants coupled with neural
 language models and embeddings priors that consistently outperform state-of-theart
 generative topic models in terms of generalization (perplexity), interpretability
 (topic coherence) and applicability (retrieval and classification) over 6 long-text
 and 8 short-text datasets from diverse domains. Probabilistic topic models, such as LDA BID1 , Replicated Softmax (RSM) (Salakhutdinov & Hinton, 2009 ) and Document Neural Autoregressive Distribution Estimator (DocNADE) variants BID12 BID34 BID15 BID8 are often used to extract topics from text collections, and predict the probabilities of each word in a given document belonging to each topic. Subsequently, they learn latent document representations that can be used to perform natural language processing (NLP) tasks such as information retrieval (IR), document classification or summarization. However, such probabilistic topic models ignore word order and represent a given context as a bag of its words, thereby disregarding semantic information.To motivate our first task of extending probabilistic topic models to incorporate word order and language structure, assume that we conduct topic analysis on the following two sentences: When estimating the probability of a word in a given context (here: P ("bear"|context)), traditional topic models do not account for language structure since they ignore word order within the context and are based on "bag-of-words" (BoWs) only. In this particular setting, the two sentences have the same unigram statistics, but are about different topics. On deciding which topic generated the word "bear" in the second sentence, the preceding words "market falls" make it more likely that it was generated by a topic that assigns a high probability to words related to stock market trading, where "bear territory" is a colloquial expression in the domain. In addition, the language structure (e.g., syntax and semantics) is also ignored. For instance, the word "bear" in the first sentence is a proper noun and subject while it is an object in the second. In practice, topic models also ignore functional words such as "into", which may not be appropriate in some scenarios.Recently, BID23 have shown that a deep contextualized LSTM-based language model (LSTM-LM) is able to capture different language concepts in a layer-wise fashion, e.g., the lowest layer captures language syntax and topmost layer captures semantics. However, in LSTM-LMs the probability of a word is a function of its sentence only and word occurrences are modeled in a fine granularity. Consequently, LSTM-LMs do not capture semantics at a document level. To this end, recent studies such as TDLM BID14 , Topic-RNN (Dieng et al., 2016) and TCNLM BID32 have integrated the merits of latent topic and neural language models (LMs); however, they have focused on improving LMs with global (semantics) dependencies using latent topics.Similarly, while bi-gram LDA based topic models BID31 BID33 and n-gram based topic learning BID15 can capture word order in short contexts, they are unable to capture long term dependencies and language concepts. In contrast, DocNADE variants BID12 BID8 ) learns word occurrences across documents i.e., coarse granularity (in the sense that the topic assigned to a given word occurrence equally depends on all the other words appearing in the same document); however since it is based on the BoW assumption all language structure is ignored. In language modeling, BID17 have shown that recurrent neural networks result in a significant reduction of perplexity over standard n-gram models.Contribution 1: We introduce language structure into neural autoregressive topic models via a LSTM-LM, thereby accounting for word ordering (or semantic regularities), language concepts and long-range dependencies. This allows for the accurate prediction of words, where the probability of each word is a function of global and local (semantics) contexts, modeled via DocNADE and LSTM-LM, respectively. The proposed neural topic model is named as contextualized-Document Neural Autoregressive Distribution Estimator (ctx-DocNADE) and offers learning complementary semantics by combining joint word and latent topic learning in a unified neural autoregressive framework. For instance, FIG0 (left and middle) shows the complementary topic and word semantics, based on TM and LM representations of the term "fall". Observe that the topic captures the usage of "fall" in the context of stock market trading, attributed to the global (semantic) view.While this is a powerful approach for incorporating language structure and word order in particular for long texts and corpora with many documents, learning from contextual information remains challenging in settings with short texts and few documents, since (1) limited word co-occurrences or little context (2) significant word non-overlap in such short texts and (3) small training corpus of documents lead to little evidence for learning word co-occurrences. However, distributional word representations (i.e. word embeddings) BID22 have shown to capture both the semantic and syntactic relatedness in words and demonstrated impressive performance in NLP tasks.For example, assume that we conduct topic analysis over the two short text fragments: Deal with stock index falls and Brace for market share drops. Traditional topic models with "BoW" assumption will not be able to infer relatedness between word pairs such as (falls, drops) due to the lack of word-overlap and small context in the two phrases. However, in the distributed embedding space, the word pairs are semantically related as shown in FIG0 (left). DISPLAYFORM0 Related work such as BID26 employed web search results to improve the information in short texts and BID24 introduced word similarity via thesauri and dictionaries into LDA. BID5 and BID20 integrated word embeddings into LDA and Dirichlet Multinomial Mixture (DMM) BID21 models. Recently, BID8 extends DocNADE by introducing pre-trained word embeddings in topic learning. However, they ignore the underlying language structure, e.g., word ordering, syntax, etc. In addition, DocNADE and its extensions outperform LDA and RSM topic models in terms of perplexity and IR.Contribution 2: We incorporate distributed compositional priors in DocNADE: we use pre-trained word embeddings via LSTM-LM to supplement the multinomial topic model (i.e., DocNADE) in learning latent topic and textual representations on a smaller corpus and/or short texts. Knowing similarities in a distributed space and integrating this complementary information via a LSTM-LM, a topic representation is much more likely and coherent.Taken together, we combine the advantages of complementary learning and external knowledge, and couple topic-and language models with pre-trained word embeddings to model short and long text documents in a unified neural autoregressive framework, named as ctx-DocNADEe. Our approach learns better textual representations, which we quantify via generalizability (e.g., perplexity), interpretability (e.g., topic extraction and coherence) and applicability (e.g., IR and classification).To illustrate our two contributions, we apply our modeling approaches to 7 long-text and 8 short-text datasets from diverse domains and demonstrate that our approach consistently outperforms state-ofthe-art generative topic models. Our learned representations, result in a gain of: (1) 4.6% (.790 vs .755) in topic coherence, (2) 6.5% (.615 vs .577) in precision at retrieval fraction 0.02, and (3) 4.4% (.662 vs .634) in F 1 for text classification, averaged over 6 long-text and 8 short-text datasets.When applied to short-text and long-text documents, our proposed modeling approaches generate contextualized topic vectors, which we name textTOvec. The code is available at https: //github.com/pgcool/textTOvec. In this work, we have shown that accounting for language concepts such as word ordering, syntactic and semantic information in neural autoregressive topic models helps to better estimate the probability of a word in a given context. To this end, we have combined a neural autoregressive topic-(i.e., DocNADE) and a neural language (e.g., LSTM-LM) model in a single probabilistic framework with an aim to introduce language concepts in each of the autoregressive steps of the topic model. This facilitates learning a latent representation from the entire document whilst accounting for the local dynamics of the collocation patterns, encoded in the internal states of LSTM-LM. We further augment this complementary learning with external knowledge by introducing word embeddings. Our experimental results show that our proposed modeling approaches consistently outperform stateof-the-art generative topic models, quantified by generalization (perplexity), topic interpretability (coherence), and applicability (text retrieval and categorization) on 15 datasets.Label: training Instructors shall have tertiary education and experience in the operation and maintenance of the equipment or sub-system of Plant. They shall be proficient in the use of the English language both written and oral. They shall be able to deliver instructions clearly and systematically. The curriculum vitae of the instructors shall be submitted for acceptance by the Engineer at least 8 weeks before the commencement of any training.Label: maintenance The Contractor shall provide experienced staff for 24 hours per Day, 7 Days per week, throughout the Year, for call out to carry out On-call Maintenance for the Signalling System.Label: cables Unless otherwise specified, this standard is applicable to all cables which include single and multi-core cables and wires, Local Area Network (LAN) cables and Fibre Optic (FO) cables.Label: installation The Contractor shall provide and permanently install the asset labels onto all equipment supplied under this Contract. The Contractor shall liaise and co-ordinate with the Engineer for the format and the content of the labels. The Contractor shall submit the final format and size of the labels as well as the installation layout of the labels on the respective equipment, to the Engineer for acceptance.Label: operations, interlocking It shall be possible to switch any station Interlocking capable of reversing the service into "Auto-Turnaround Operation". This facility once selected shall automatically route Trains into and out of these stations, independently of the ATS system. At stations where multiple platforms can be used to reverse the service it shall be possible to select one or both platforms for the service reversal. TAB10 : Perplexity scores for different λ in Generalization task: Ablation over validation set labels are not used during training. The class labels are only used to check if the retrieved documents have the same class label as the query document. To perform document retrieval, we use the same train/development/test split of documents discussed in data statistics (experimental section) for all the datasets during learning.See TAB1 for the hyperparameters in the document retrieval task.
Unified neural model of topic and language modeling to introduce language structure  in topic models for contextualized topic vectors 
{"src": "scitldr-AIC_train", "paper_id": "rkgoyn09KQ"}