Computer Science
ComputerScience,1970-2008-Computer Science-1997.jsonl.gz.gz
Computer Science
282a192afc56e4fe09655f9a04326eea217b19a6
Domain Adaptation with Clustered Language Models

In this paper, a method of domain adaptation for clustered language models is developed. It is based on a previously developed clustering algorithm, but with a modified optimisation criterion. The results are shown to be slightly superior to the previously published 'Fillup' method, which can be used to adapt standard n-gram models. However, the improvement both methods give compared to models built from scratch on the adaptation data is quite small (less than 11% relative improvement in word error rate). This suggests that both methods are still unsatisfactory from a practical point of view.


Introduction
Current large vocabulary speech recognition systems can achieve good performance on domains for which large quantities (e.g. millions of words) of textual data are available to train a language model. In real world applications, however, this is quite often not the case. The issue of language model domain adaptation is therefore of great practical importance.
One approach to tackle this problem is to try to learn from an analogy to the speaker dependence issue: current systems perform well by training speaker independent models, which can then be adapted with relatively little data from a given speaker (see [8]). Can the same approach be applied to language model adaptation?
In section 2, previous work in this area is reviewed and a rough working definition of domain is given. A method to perform domain adaptation with clustered language models is then developed (Section 3). Experimental results to evaluate the method are given in Section 4, followed by conclusions in Section 5.

Background
In order to make the description of domain adaptation more precise, a definition of domain is needed. One might be tempted to define domain in the sense of semantic topic. However, texts might differ in other aspects (e.g. style), which could still require language model adaptation. A more general definition of domain, more in line with the term sublanguage, is therefore required. According to [10], there are many different definitions of the term, but most of them seem to agree on the following characteristics of a sublanguage: 1. it is part of a natural language 2. it is of a specialised form 3. it behaves like a complete language 4. it is used in special circumstances (e.g. expert communication) 5. it is limited to a particular subject domain Some of these points seem very useful for the concept of domain (2,4), others less so (1). What properties should an acceptable definition of domain have?
The following spring to mind: • there should be a continuum (e.g. an infinite number) of domains • each domain may contain an infinite number of elements (e.g. documents/sentences/words) • for a given element, one should be able to decide whether or not it belongs to a given domain Domain adaptation can be divided into static and dynamic domain adaptation, depending on the time scale used to perform adaptation. Dynamic adaptation tries to capture phenomena with a shorter time scale (e.g. topic shifts) and is performed on line, whereas static adaptation can be used to perform a onetime shift from one domain to another and is performed off line. Previous work has shown improvements by using both dynamic adaptation of n-gram models ( [7], [5], [2], [9], [6], [13], [3], [4]) and by using static adaptation of n-gram models ( [9], [6], [1], [13]). Since the 'Fillup' method presented in [1] gives better performance than linear interpolation, the 'Fillup' method is used as method of comparison for the adaptive clustering, which will be developed in the next section.

Adaptive Clustering
The task of a language model is to calculate p(w i |c i ), the probability of the next word being w i given the current context c i . Language models differ in the way this probability is modelled and how the context c i is defined. A quite general model proposed in [12] makes use of a state mapping function S and a category mapping function G. The idea behind the state mapping S : c− > s c = S(c) is to assign each of the large number of possible contexts c ∈ C to one of a smaller number of context-equivalent states. Similarly, the category mapping G : w− > g w = G(w) assigns each of the large number of possible words w ∈ V to one of a smaller number of categories (similar to parts of speech). The probability of the next word is then calculated as In order to determine S and G automatically, a clustering algorithm as shown in Figure 1 can be used. It is a greedy, hill-climbing algorithm that moves Algorithm 1: Clustering() start with initial clustering functions S, G iterate until some convergence criterion is met for all w ∈ V and c ∈ C for all g ′ w ∈ G and s ′ c ∈ S calculate the difference in the optimisation criterion when w/c is moved from g w /s c to g ′ w /s ′ c move the w/c to the g ′ w /s ′ c that results in the biggest improvement in optimisation criterion End Clustering Figure 1: The clustering algorithm elements to the best available choice at any given time. Based on equation 1 and on the leaving-one-out likelihood of the model generating the training data, an optimisation criterion can be derived (see [14] for a detailed description). Let N (e) denote the number of times event e appeared in the training data, let B denote the smoothing parameter used for absolute discounting ( [11]), and let n 0 , n 1 , n + denote the number of pairs (s, g) that have appeared zero, one and one or more times in the training data. The resulting optimisation criterion F (as derived in [14]) is The basic building block in the derivation of equation 2 is the likelihood of one event in the training corpus, as estimated from the training corpus in which this one event has been removed (leaving-one-out likelihood). The main idea behind the adaptive clustering is to use as basic building block the likelihood of one event in S Adapt , as estimated from a linear interpolation of counts from S Back and S Adapt from which this one event has been removed. The motivation for this is that the clustering can thus optimise the perplexity on S Adapt , while having access to a linear combination of counts from S Back and S Adapt .
Let N A (e) (N B (e)) denote the number of times event e appeared in S Adapt (S Back ). Define N C (e) to be the linear interpolation of the two counts where Round(x) returns the integer nearest to x. The only events that can contribute to the optimisation function are events that occur at least once in S Adapt (because, as explained above, the likelihood of S Adapt is taken as optimisation function). However, their probability is calculated based on the combined counts. Therefore, the smoothing has to apply to the combined counts. Define n bi,0 , n bi,1 , n bi,+ as the number of pairs (s, g) that have a combined count N C (s, g) of 0, of 1, and larger than 0. In order to introduce absolute discounting for the unigram estimates as well, also define n s,0 , n s,1 , n s,+ as the number of states s that have a combined count N C (s) of 0, of 1, and larger than 0 (similarly, define n g,0 etc. for the unigram estimates involving g). Changing equation 2 according to the basic idea outlined above, this leads to (ng,0+1) ).
By using the same clustering algorithm as before, but with F Adapt instead of F as optimisation criterion, language model domain adaptation can be performed.

Results
In order to test different adaptation methods, two textual samples S Back and S Adapt and acoustic testing data from D Adapt are required. Since the WSJ domain has the associated acoustic data, it is used as D Adapt . As D Back , the patent domain (PAT) was chosen, for which a large sample S Adapt (about 35 million words are used) is also available from the LDC as part of the TIPSTER database.
The recognition system is a state-of-the-art HMM based system (continuous densities, mixtures, triphones). All experiments are based on bigram language models, either clustered (500 clusters) or backoff (singleton bigrams were ignored). The different methods evaluated were • ClustAdapt: a model built with the adaptive clustering presented in the previous section; the initial starting point for the clustering is taken to be the clustering produced by Back Cl ; one global λ parameter was used and optimised iteratively at the end of each iteration; For all methods except Back Bo and Back Cl , the vocabulary was defined to be all the words that appeared in S Adapt , plus additional words from S Back until 20K words were reached. For Back Bo and Back Cl , the vocabulary consisted of the 20K most frequent words in S Back . Because of this difference, the perplexities of Back Bo and Back Cl are not directly comparable to those of the other models. For each method and a given amount of adaptation material, the perplexity of the resulting model was calculated on a held-out section of S Adapt and a recognition run was performed on the acoustic data.   Table 1 gives the results of the two baseline methods, which do not use any of the adaptation material. The high perplexities show that the PAT and WSJ domains are considerably different. The rate of out-of-vocabulary words is about 15%, which is one reason for the very high error rate. Tables 2, 3, 4 and 5 give the results for the different methods and different amounts of adaptation material.

Model
Comparing Table 3 to Table 2, one can seen that Adapt Cl is more robust than Adapt Bo and it leads to better recognition results for almost the entire range of adaptation material. This is consistent with previous results (see [15]), which showed that clustered models are more robust in terms of perplexity.
Comparing Table 4 to Table 3, one can see that F illup outperforms Adapt Cl in almost all cases.
By looking at

Conclusion
Compared to the success of some methods for acoustic adaptation, the results obtained here are somewhat disappointing. In particular, they seem to suggest that the improvements from the adaptation techniques compared to starting from scratch on the adaptation data become quite small when several tens of thousands of words are available 1 . One reason for this could be the fact that the acoustic space has an underlying distance metric and thus allows the comparison of two elements. Moreover, one can specify the kind of transformations one would want the adaptation to be able to perform. Both of these points seem more difficult in the case of language model adaptation. Even though the adaptation method for clustered language models developed in this paper gives slightly better results than the 'Fillup' method, the accuracies obtained with the adaptive clustering and the 'Fillup' method are still very low compared to the about 80% or more the system can achieve with a backoff bigram trained on about 40 million words of the WSJ corpus. Both adaptation methods are therefore still unsatisfactory from a practical point of view.