Prior use of machine learning in genre classification used a list of labels as classification categories . However , genre classes are often organised into hierarchies , e.g. , covering the subgenres of fiction . In this paper we present a method of using the hierarchy of labels to improve the classification accuracy . As a testbed for this approach we use the Brown Corpus as well as a range of other corpora , including the BNC , HGC and Syracuse . The results are not encouraging : apart from the Brown corpus , the improvements of our structural classifier over the flat one are not statistically significant . We discuss the relation between structural learning performance and the visual and distributional balance of the label hierarchy , suggesting that only balanced hierarchies might profit from structural learning . Automatic genre identification ( AGI ) can be traced to the mid-1990s ( Karlgren and Cutting , 1994 ; Kessler et al . , 1997 ) , but this research became much more active in recent years , partly because of the explosive growth of the Web , and partly because of the importance of making genre distinctions in NLP applications . In Information Retrieval , given the large number of web pages on any given topic , it is often difficult for the users to find relevant pages that are in the right genre ( Vidulin et al . , 2007 ) . As for other applications , the accuracy of many tasks , such as machine translation , POS tagging ( Giesbrecht and Evert , 2009 ) or identification of discourse relations ( Webber , 2009 ) relies of defining the language model suitable for the genre of a given text . For example , the accuracy of POS tagging reaching 96.9 % on newspaper texts drops down to 85.7 % on forums ( Giesbrecht and Evert , 2009 ) , i.e. , every seventh word in forums is tagged incorrectly . This interest in genres resulted in a proliferation of studies on corpus development of web genres and comparison of methods for AGI . The two corpora commonly used for this task are KI-04 ( Meyer zu Eissen and Stein , 2004 ) and Santinis ( Santini , 2007 ) . The best results reported for these corpora ( with 10-fold cross-validation ) reach 84.1 % on KI-04 and 96.5 % accuracy on Santinis ( Kanaris and Stamatatos , 2009 ) . In our research ( Sharoff et al . , 2010 ) we produced even better results on these two benchmarks ( 85.8 % and 97.1 % , respectively ) . However , this impressive accuracy is not realistic in vivo , i.e. , in classifying web pages retrieved as a result of actual queries . One reason comes from the limited number of genres present in these two collections ( eight genres in KI-04 and seven in Santinis ) . As an example , only front pages of online newspapers are listed in Santinis , but not actual newspaper articles , so once an article is retrieved , it can not be assigned to any class at all . Another reason why the high accuracy is not useful concerns the limited number of sources in each collection , e.g. , all FAQs in Santinis come from either a website with FAQs on hurricanes or another one with tax advice . In the end , a classifier built for FAQs on this training data relies on a high topic-genre correlation in this particular collection and fails to spot any other FAQs . There are other corpora , which are more diverse in the range of their genres , such as the fifteen genres of the Brown Corpus ( Kučera and Francis , 1967 ) or the seventy genres of the BNC ( Lee , 2001 ) , but because of the number of genres in them and the diversity of documents within each genre , the accuracy of prior work on these collections is much less impressive . For example , Karlgren and Cutting ( 1994 ) using linear discriminant analysis achieve an accuracy of 52 % without us-ing cross-validation ( the entire Brown Corpus was used as both the test set and training set ) , with the accuracy improving to 65 % when the 15 genres are collapsed into 10 , and to 73 % with only 4 genres ( Figure 1 ) . This result suggests the importance of the hierarchy of genres . Firstly , making a decision on higher levels might be easier than on lower levels ( fiction or non-fiction rather than science fiction or mystery ) . Secondly , we might be able to improve the accuracy on lower levels , by taking into account the relevant position of each node in the hierarchy ( distinguishing between reportage or editorial becomes easier when we know they are safely under the category of press ) . This paper explores a way of using information on the hierarchy of labels for improving fine-grained genre classification . To the best of our knowledge , this is the first work presenting structural genre classification and distance measures for genres . In Section 2 we present a structural reformulation of Support Vector Machines ( SVMs ) that can take similarities between different genres into account . This formulation necessitates the development of distance measures between different genres in a hierarchy , of which we present three different types in Section 3 , along with possible estimation procedures for these distances . We present experiments with these novel structural SVMs and distance measures on three different corpora in Section 4 . Our experiments show that structural SVMs can outperform the non-structural standard . However , the improvement is only statistically significant on the Brown corpus . In Section 5 we investigate potential reasons for this , including the ( im)balance of different genre hierarchies and problems with our distance measures . In this paper , we have evaluated structural learning approaches to genre classification using sev-eral different genre distance measures . Although we were able to improve on non-structural approaches for the Brown corpus , we found it hard to improve over flat SVMs on other corpora . As potential reasons for this negative result , we suggest that current genre hierarchies are either not of sufficient depth or are visually or distributionally imbalanced . We think further investigation into the relationship between hierarchy balance and structural learning is warranted . Further investigation is also needed into the appropriateness of n-gram features for genre identification as well as good measures of genre distance . In the future , an important task would be the refinement or unsupervised generation of new hierarchies , using information theoretic or data-driven approaches . For a full assessment of hierarchical learning for genre classification , the field of genre studies needs a testbed similar to the Reuters or 20 Newsgroups datasets used in topic-based IR with a balanced genre hierarchy and a representative corpus of reliably annotated webpages . With regard to algorithms , we are also interested in other formulations for structural SVMs and their large-scale implementation as well as the combination of different distance measures , for example in ensemble learning .
Existing genre classification methods achieve high accuracy without regarding hierarchical structures exhibiting unrealistic experimental setups such as a limited number of genres and sources.
They propose a structural reformulation of the Support Vector Machine to take hierarchical information of genres into account by using similarities between different genres.
The proposed model outperforms non-hierarchical models on only one corpus and they discuss that it may be due to insufficient depth or inbalance of hierarchies.