In the past several decades, many authorship attribution studies have used computational methods to determine the authors of disputed texts. Disputed authorship is a common problem in Classics, since little information about ancient documents has survived the centuries. Many scholars have questioned the authenticity of the final chapter of Xenophon's Cyropaedia, a 4th century B.C. historical text. In this study, we use N-grams frequency vectors with a cosine similarity function and word frequency vectors with Naive Bayes Classifiers (NBC) and Support Vector Machines (SVM) to analyze the authorship of the Cyropaedia. Although the N-gram analysis shows that the epilogue of the Cyropaedia differs slightly from the rest of the work, comparing the analysis of Xenophon with analyses of Aristotle and Plato suggests that this difference is not significant. Both NBC and SVM analyses of word frequencies show that the final chapter of the Cyropaedia is closely related to the other chapters of the Cyropaedia. Therefore, this analysis suggests that the disputed chapter was written by Xenophon. This information can help scholars better understand the Cyropaedia and also demonstrates the usefulness of applying modern authorship analysis techniques to classical literature.
A 1982 study of the Corpus Lysiacum did use word frequencies as one of the features to distinguish texts attributed to Lysias. This method used chi-squared tests to provide a distance measure. Chi-squared tests assume that features are independent, but independence is often not true in grammar. The study of the Corpus Lysiacum additionally involved parsing the text by hand, which is extremely time consuming and error prone @cite_2 .
@cite_2
2015201047 2005492507 1973550659 2030720628
In the process of establishing the information theory, C. E. Shannon proposed the Markov process as a good model to characterize a natural language. The core of this idea is to calculate the frequencies of strings composed of n characters (n-grams), but this statistical analysis of large text data and for a large n has never been carried out because of the memory limitation of computer and the shortage of text data. Taking advantage of the recent powerful computers we developed a new algorithm of n-grams of large text data for arbitrary large n and calculated successfully, within relatively short time, n-grams of some Japanese text data containing between two and thirty million characters. From this experiment it became clear that the automatic extraction or determination of words, compound words and collocations is possible by mutually comparing n-gram statistics for different values of n. We present a statistical model of feature occurrence over time, and develop tests based on classical hypothesis testing for significance of term appearance on a given date. Using additional classical hypothesis testing we are able to combine these terms to generate “topics” as defined by the Topic Detection and Tracking study. The groupings of terms obtained can be used to automatically generate an interactive timeline displaying the major events and topics covered by the corpus. To test the validity of our technique we extracted a large number of these topics from a test corpus and had human evaluators judge how well the selected features captured the gist of the topics, and how they overlapped with a set of known topics from the corpus. The resulting topics were highly rated by evaluators who compared them to known topics. Statistical methods to extract translational equivalents from non-parallel corpora hold the promise of ensuring the required coverage and domain customisation of lexicons as well as accelerating their compilation and maintenance. A challenge for these methods are rare, less common words and expressions, which often have low corpus frequencies. However, it is rare words such as newly introduced terminology and named entities that present the main interest for practical lexical acquisition. In this article, we study possibilities of improving the extraction of low-frequency equivalents from bilingual comparable corpora. Our work is carried out in the general framework which discovers equivalences between words of different languages using similarities between their occurrence patterns found in respective monolingual corpora. We develop a method that aims to compensate for insufficient amounts of corpus evidence on rare words: prior to measuring cross-language similarities, the method uses same-language corpus data to model co-occurrence vectors of rare words by predicting their unseen co-occurrences and smoothing rare, unreliable ones. Our experimental evaluation demonstrates that the proposed method delivers a consistent and significant improvement on the conventional approach to this task. Syntactic information about a corpus of linguistic or pictorial data can be discovered by analyzing the statistics of the data. Given a corpus of text, one can measure the tendencies of pairs of words to occur in common contexts, and use these measurements to define clusters of words. Applied to basic English text, this procedure yields clusters which correspond very closely to the traditional parts of speech (nouns, verbs, articles, etc.). For FORTRAN text, the clusters obtained correspond to integers, operations, etc.; for English text regarded as a sequence of letters (or of phonemes) rather than words, the vowels and the consonants are obtained as clusters. Finally, applied to the gray shades in a digitized picture, the procedure yields slice levels which appear to be useful for figure extraction.