In this paper we examine how the differences in modelling between different data driven systems performing the same NLP task can be exploited to yield a higher accuracy than the best individual system . We do this by means of an experiment involving the task of morpho-syntactic wordclass tagging . Four well-known tagger generators ( Hidden Markov Model , Memory-Based , Transformation Rules and Maximum Entropy ) are trained on the same corpus data . After comparison , their outputs are combined using several voting strategies and second stage classifiers . All combination taggers outperform their best component , with the best combination showing a 19.1 % lower error rate than the best individual tagger . In all Natural Language Processing ( NLP ) systems , we find one or more language models which are used to predict , classify and/or interpret language related observations . Traditionally , these models were categorized as either rule-based / symbolic or corpusbased / probabilistic . Recent work ( e.g. Brill 1992 ) has demonstrated clearly that this categorization is in fact a mix-up of two distinct Categorization systems : on the one hand there is the representation used for the language model ( rules , Markov model , neural net , case base , etc . ) and on the other hand the manner in which the model is constructed ( hand crafted vs. data driven ) . Data driven methods appear to be the more popular . This can be explained by the fact that , in general , hand crafting an explicit model is rather difficult , especially since what is being modelled , natural language , is not ( yet ) wellunderstood . When a data driven method is used , a model is automatically learned from the implicit structure of an annotated training corpus . This is much easier and can quickly lead to a model which produces results with a ' reasonably ' good quality . Obviously , ' reasonably good quality ' is not the ultimate goal . Unfortunately , the quality that can be reached for a given task is limited , and not merely by the potential of the learning method used . Other limiting factors are the power of the hard-and software used to implement the learning method and the availability of training material . Because of these limitations , we find that for most tasks we are ( at any point in time ) faced with a ceiling to the quality that can be reached with any ( then ) available machine learning system . However , the fact that any given system can not go beyond this ceiling does not mean that machine learning as a whole is similarly limited . A potential loophole is that each type of learning method brings its own ' inductive bias ' to the task and therefore different methods will tend to produce different errors . In this paper , we are concerned with the question whether these differences between models can indeed be exploited to yield a data driven model with superior performance . In the machine learning literature this approach is known as ensemble , stacked , or combined classifiers . It has been shown that , when the errors are uncorrelated to a sufficient degree , the resulting combined classifier will often perform better than all the individual systems ( Ali and Pazzani 1996 ; Chan and Stolfo 1995 ; Tumer and Gosh 1996 ) . The underlying assumption is twofold . First , the combined votes will make the system more robust to the quirks of each learner 's particular bias . Also , the use of information about each individual method 's behaviour in principle even admits the possibility to fix collective errors . We will execute our investigation by means of an experiment . The NLP task used in the experiment is morpho-syntactic wordclass tagging . The reasons for this choice are several . First of all , tagging is a widely researched and well-understood task ( cf . van Halteren ( ed . ) 1998 ) . Second , current performance levels on this task still leave room for improvement : ' state of the art ' performance for data driven automatic wordclass taggers ( tagging English text with single tags from a low detail tagset ) is 96 - 97 % correctly tagged words . Finally , a number of rather different methods are available that generate a fully functional tagging system from annotated text . Our experiment shows that , at least for the task at hand , combination of several different systems allows us to raise the performance ceiling for data driven systems . Obviously there is still room for a closer examination of the differences between the combination methods , e.g. the question whether Memory-Based combination would have performed better if we had provided more training data than just Tune , and of the remaining errors , e.g. the effects of inconsistency in the data ( cf . Ratnaparkhi 1996 on such effects in the Penn Treebank corpus ) . Regardless of such closer investigation , we feel that our results are encouraging enough to extend our investigation of combination , starting with additional component taggers and selection strategies , and going on to shifts to other tagsets and/or languages . But the investigation need not be limited to wordclass tagging , for we expect that there are many other NLP tasks where combination could lead to worthwhile improvements .
Different data driven approaches tend to produce different errors and their qualities are limited due to the learning method and available training material.
They propose to combine four different modelling methods for the task of morpho-syntactic wordclass tagging by using several voting strategies and second stage classifiers.
All combinations outperform the best component, with the best one showing a 19.1% lower error rate and raising the performance ceiling.