Statistics about n-grams (i.e., sequences of contiguous words or other tokens in text documents or other string data) are an important building block in information retrieval and natural language processing. In this work, we study how n-gram statistics, optionally restricted by a maximum n-gram length and minimum collection frequency, can be computed efficiently harnessing MapReduce for distributed data processing. We describe different algorithms, ranging from an extension of word counting, via methods based on the Apriori principle, to a novel method Suffix- that relies on sorting and aggregating suffixes. We examine possible extensions of our method to support the notions of maximality closedness and to perform aggregations beyond occurrence counting. Assuming Hadoop as a concrete MapReduce implementation, we provide insights on an efficient implementation of the methods. Extensive experiments on The New York Times Annotated Corpus and ClueWeb09 expose the relative benefits and trade-offs of the methods.
Given their role in NLP, multiple efforts @cite_23 @cite_10 @cite_2 @cite_18 @cite_45 have looked into @math -gram statistics computation. While these approaches typically consider document collections of modest size, recently @cite_32 and @cite_29 targeted web-scale data. Among the aforementioned work, @cite_18 is closest to ours, also focusing on less frequent @math -grams and using a cluster of machines. However, they only consider @math -grams consisting of up to eleven words and do not provide details on how their methods can be adapted to MapReduce. Yamamoto and Church @cite_13 augment suffix arrays, so that the collection frequency of substrings in a document collection can be determined efficiently. Bernstein and Zobel @cite_3 identify long @math -grams as a means to spot co-derivative documents. @cite_31 and @cite_21 describe the @math -gram statistics made available by Google and Microsoft, respectively. Zhai @cite_7 gives details on the use of @math -gram statistics in language models. @cite_1 demonstrated recently that @math -gram time series are powerful tools to understand the evolution of culture and language.
@cite_13 @cite_18 @cite_7 @cite_29 @cite_21 @cite_1 @cite_32 @cite_3 @cite_45 @cite_23 @cite_2 @cite_31 @cite_10
2159077101 2951714314 1539309091 2104210067
The authors present an algorithm to adapt a n-gram language model to a document as it is dictated. The observed partial document is used to estimate a unigram distribution for the words that already occurred. Then, they find the closest n-gram distribution to the static n-gram distribution (using the discrimination information distance measure) that satisfies the marginal constraints derived from the document. The resulting minimum discrimination information model results in a perplexity of 208 instead of 290 for the static trigram model on a document of 321 words. > We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned Kneser-Ney 5-gram model achieves perplexity 67.6; a combination of techniques leads to 35 reduction in perplexity, or 10 reduction in cross-entropy (bits), over that baseline. The benchmark is available as a code.google.com project; besides the scripts needed to rebuild the training held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models. We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned KneserNey 5-gram model achieves perplexity 67.6. A combination of techniques leads to 35 reduction in perplexity, or 10 reduction in cross-entropy (bits), over that baseline. The benchmark is available as a code.google.com project; besides the scripts needed to rebuild the training held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models. Some models of textual corpora employ text generation methods involving n-gram statistics, while others use latent topic variables inferred using the "bag-of-words" assumption, in which word order is ignored. Previously, these methods have not been combined. In this work, I explore a hierarchical generative probabilistic model that incorporates both n-gram statistics and latent topic variables by extending a unigram topic model to include properties of a hierarchical Dirichlet bigram language model. The model hyperparameters are inferred using a Gibbs EM algorithm. On two data sets, each of 150 documents, the new model exhibits better predictive accuracy than either a hierarchical Dirichlet bigram language model or a unigram topic model. Additionally, the inferred topics are less dominated by function words than are topics discovered using unigram statistics, potentially making them more meaningful.