When not to use MT and other translation tools

The letter "L" often crops up in such decision-making. Love, literature, law...these are some of the areas where translation tools, particularly MT programs might struggle to convey meaning in a useful way. Another "L" word is length; when your system is lexically challenged, the length of the document to be translated will be a key factor in determining whether the time spent entering new terms will be repaid. Creativity is another consideration. Sometimes the brief will be to "live and breathe" the environment of the target audience, to create a "new original", to "surpass the original". Even a good MT program will tie the translator's hands then; revisers tinker, cut and paste, and scrub out complete sentences only when the program has made a complete mess. Translation memory tools regurgitate old translations; not much help if freshness and spontaneity are what is wanted. This intervention will take a hopefully entertaining look at what we may or may not expect our translation tools to deliver. Hook and Hatton Ltd. Hook and Hatton Ltd is a family-owned company providing specialist language services. Over the past four years it has concentrated on supplying MT services via e-mail. It delivers Dutch-English translations with its own MT program and also provides German-English and French-English MT output using third-party programs. Hook & Hatton Ltd 34 Central Avenue Whitehills Northampton NN2 8DZ Telephone: + 44 1604 847278 Fax: + 44 1604 821486 E-mail: Hook_Hatton@compuserve.com 34 EAMT Workshop, Copenhagen, May 1997 When not to use MT and other translation tools Discussion paper presented by Terence Lewis Hook & Hatton Ltd The question we are asking is when not to use translation tools. The first answer might be : when they provide unusable output. Usability, of course, is relative it varies according to the purpose of the document, both for the author and the reader. These purposes may differ: a document that expresses a burning issue for the author may be considered irrelevant by the reader. Or a discarded draft dashed off by a specialist may be an essential piece in a particular reader's intellectual jigsaw puzzle. There are many cheap-and-cheerful translation packages on the market. The following translation was produced by one of them: German: In jedem Fall ist eine Abstraktion und Reduzierung der Objektinformation vorzunehmen. Eine derartige Objektbeschreibung wird als Modellierung bezeichnet. Möglichkeiten zur Datenakquisition bieten Meßmaschinen, Methoden der Photogrammetrie und geodätische Verfahren. English: An abstraction and reduction of object information is to be carried out always. A such object description is designated as a modeling. Possibilities for the data acquisition offer measuring machines, means of photogrammetry and geodesic procedures.

Discussion paper presented by Terence Lewis Hook & Hatton Ltd
The question we are asking is when not to use translation tools. The first answer might be : when they provide unusable output. Usability, of course, is relative -it varies according to the purpose of the document, both for the author and the reader. These purposes may differ: a document that expresses a burning issue for the author may be considered irrelevant by the reader. Or a discarded draft dashed off by a specialist may be an essential piece in a particular reader's intellectual jigsaw puzzle. There are many cheap-and-cheerful translation packages on the market. The following translation was produced by one of them: German: In jedem Fall ist eine Abstraktion und Reduzierung der Objektinformation vorzunehmen. Eine derartige Objektbeschreibung wird als Modellierung bezeichnet. Möglichkeiten zur Datenakquisition bieten Meßmaschinen, Methoden der Photogrammetrie und geodätische Verfahren.
English: An abstraction and reduction of object information is to be carried out always. A such object description is designated as a modeling. Possibilities for the data acquisition offer measuring machines, means of photogrammetry and geodesic procedures.

Figure 1
This is an example of very poor MT output that is still of some informational value to a reader with no knowledge of the source language. If that reader finds it gives him the knowledge he is seeking, he should use the MT software in question; it does not, he should leave it on the shelf.
The above piece of information may fill a gap in somebody's knowledge. We should probably decide against using translation tools if they leave large gaps in our knowledge. A text of foreign specialist terms punctuated with translated determiners, verbs and adjectives is hardly going to enlighten and inform. This degree of inadequacy is less likely in a controlled or restricted domain. Here, it is nearly always possible to generate useful machine translation output. By definition there is near-total lexical, syntactic and semantic coverage. Weather forecasts, avalanche bulletins, job descriptions; motorway traffic flow: in all these areas, few terms are involved and the output is likely to be reliable and valuable. Can we get useful output if the input file is outside the domain? Can we create customised dictionaries? Does the grammar provide coverage of all possible syntax? That depends on how the system has been designed.
There would be little point in submitting an aircraft maintenance manual to the TAUM METEO system.
In practice, use of any MT program does not make much sense if the lexical deficit is disproportionately large in relation to the length of the text. If we have a text of, say, less than 5 pages with more than 30 unknown words it is usually more cost-effective to have the text translated by a human translator than to go through the business of inputting all those new entries. But that's a judgement call. Sometimes, if the subject is likely to crop up again the dictionary update time can be treated as a sound investment.
Most purchasers of MT programs are likely to use their software in a general translation environment. Experience has shown us that there are number of disciplines where we are better off using other approaches to translation than MT. Not a few of these seem to begin with the letter "L". If we take a basic PC translation package straight out of the box it will give a varyingly useful translation of a technical text. Legal texts, however, seem to provide problems for all systems that are not specifically designed to deal with them. Figure 2 exemplifies a number of these problems.  Sentences including phrases such as "how the products developed are manufactured or the processes developed are applied" would confound most off-the-shelf German-English packages, although the French-English package used here gets that right after completely misreading "parties". In fact, if we look at the above example we see that different sentences cause different problems for the two languages. Of course, the ridiculous German translation of "joint research and development" could easily be avoided by coding the four words as a single entity. Generally speaking legal texts contain a multiplicity of clauses, often without any commas. This causes problems when translating into or from languages in which the verb is thrown to the end of the clause after a relative pronoun or a conjunction. These problems are highlighted by Figure 3.

English original:
The vendor who may not be an alien shall not deposit the proceeds of the sale in an account held outside the national territory save in such circumstances as may be determined in sub-section 3 to which reference is also made in chapter 4 under the heading "Recognized accounts" and only subject to satisfactory verification of full compliance with the statutory currency transfer requirements by all bodies duly appointed for this purpose by the regulatory authority or any other agency having the powers to appoint bodies to verify compliance with the statutory currency transfer requirements or any other pertinent requirements.

Figure 3
The lack of commas also makes semantic disambiguation more difficult in the above text. Of course, the combination of MT and TM approaches would certainly reduce likelihood of errors. Strings such as "shall not deposit the proceeds of the sale" and "save in such circumstances" could be entered into the translation archive and would then be delivered up as fixed entities; infelicities such as "sparen in solchen Umständen" would be systematically avoided. In our experience, this is certainly the most cost-effective way to "machine-translate" large volumes of legal text.
With legal translation it is often a question of building a bridge between two quite different legal systems. False friends can very easily send the reader off down the wrong path. The distinction between the Spanish Escritura de compraventa and the contrato privado de compraventa is quite alien to English law. The idea that you can legally purchase a property under a private agreement while the title deed remains in someone else's name would strike most English lawyers as a preposterous proposition. Yet it is common practice in rural Spain. So even if an MT system got the translation of EAMT Workshop, Copenhagen, May 1997 a term right we have to ask ourselves what use the translation would be to a reader who did not understand these differences between English and Spanish legal systems. Another aspect of legal translation is that sometimes much hangs upon the precise interpretation of a single world. In view of this, one would have to question the usefulness of a computer-translated legal text unless it were thoroughly revised by somebody with a knowledge of the two legal systems involved.
Anyway, law is boring and it's the middle of the afternoon. So let's get onto love or lustboth "L" words. Reading recently about the Systran/Seiko strategic alliance aimed at developing a new generation of handheld translators -I just love the name handheld translators -I had the image of this hapless young man wandering around the night clubs of Europe clutching his brand-new Systran/Seiko handheld. A conversation something like this might take place:  Perhaps the result would be so amusing that it might actually serve as a conversation starter! (Author's note: This slide was intended as a humorous interlude in my talk; I have left it in the write-up as it serves to make a serious point about the unsuitability of many modes of linguistic discourse for computer processing).
Talking of bars or night clubs makes me think of cross-dressing and that brings up the problem of cross-disciplinary documents. Such would be a legal document concerning highly specialised photogrammetric equipment to be used in geological surveys of rock faces. If the MT system does not allow the user to select at least three customisable subject dictionaries, the computer version of such a document would be of questionable value and the translation would be more usefully entrusted completely to a human translator.
We have talked a little about domains and disciplines. Another factor that plays a part in the decision on whether to use MT or any other translation tool is the purpose of the document and the purpose of the translation. A translation may face numerous challenges and constraints: • The translation may be needed to assess the creativity of the original.
• A creative translation may be required.
• The translation should surpass the original.
• The translation will become the new original.
• Cultural bridging is required.
Let us look at each of these points. If we need to assess the creativity of the originalsay a sales brochure, it's unlikely that the computer translation is going to capture this. Only a human translator would be able to recognise creativity and interpret it for the target language. Then, the translation itself may need to be creative. It may need to relate to the target audience, live and breathe the culture of the target audience. No matter how good or bad a computer translation is, it will shape the final draft of the document. In fact, the better it is, the less likely it is that the reviser will make any drastic changes. In our experience, the good machine translation seduces the reviser into complacency. If the translation is grammatically and semantically correct, the MT reviser is loath to scrub it out and recast it sentence by sentence.
Another thing the MT program cannot do is to assess how much of what is assumed in the source text needs to be brought out or clarified for the target audience. Figure 5 shows a translation generated by a computer program followed by a translation produced the corporate translation department of the company concerned. The first difference we notice is in the title: The "Call on the Vecht" is a literal translation of the Dutch. The human translator probably felt that the target audience outside Holland would not necessarily know that the Vecht is a river so he wrote "Call from the banks of the Vecht". There is nothing wrong with "Nearly a century later", but "Now, nearly a century later" is far more engaging and somehow establishes a rapport with the reader. I happen to know that the human translation was produced by a Dutchman. As an Englishman, if I were starting the translation from scratch I would have departed far more radically from the source text while still retaining the overall meaning. If, on the other hand, I were given the MT output, I would be inclined to do little more than tinker with it. The result would be a grammatically correct, wooden piece of prose which would fail to fulfil the purposes invested in the Dutch original by that company's PR department. The history of the location in Maarsen (formerly ACF chemistry) goes back to 1895. In that year a small chemical activity was started locally. In 1902 the first quinine production followed. Nearly a century later quinine is still being produced. It is extracted from the bark of the cinchona. Plantations for this evergreen plant are to be found in Indonesia. Africa, South-East Asia and South America. DSM Andeno Maarssem extracts from the bark the pure quinine and produces also (in a chemical process) quinine.

Call from the banks of the Vecht DSM Andeno Maarsen: tradition and ambition
The history of the Maarsen complex (formerly ACF Chemie) started in the year 1895, when a small chemicals operation was started at this site. This was followed in 1902 by the start of quinine production. Now, nearly a century later, the Maarsen facilities still produce quinine, which is extracted from chinchona bark. The evergreen chinchona trees are grown in plantations in Indonesia, Africa, South-east Asia and South America. DSM Andeno Maarsen extracts from chinchona bark pure quinine and.via a chemical route, also produces quinine.

Figure 5
In some situations, the translation will become the new original. This might happen in a multinational company where the author does not write the corporate language well enough for his report to serve as a master document which will be translated into many languages. If the requirement is for a vibrant, confident, authoritative text, the translation would better be done by a human translator from the outset. I already touched on the idea of cultural bridging when I referred to the River Vecht. It involves identifying concepts for which there is no precise equivalent in the target language or references which the intended readership might not fully understand. If we are translating the term "Notaire" from French into German, no cultural bridging is needed. But the "Notaire" does not fully correspond to the English "notary" and it would be misleading if not outright wrong to translate it as such. If a document 40 EAMT Workshop, Copenhagen, May 1997 contains a large number of terms which would have to be explained in detail to the target audience, it is probably more useful to have it translated by a human translator. On the other hand, if we were designing a system to be used exclusively for legal translations such explanations and distinctions would ideally be included in our sub-language dictionary module. Situational decision-making is as critical when assigning tasks to a set of tools as it is when giving out jobs in the translation tool.
Basically, however, it all comes down to money. If Deep Blue can search through the ramifications of hundreds of thousands of chess moves in seconds, it is possible to search corpora containing all possible combinations of a limited number of words and select the statistically most appropriate solutions to a particular lexical or semantic problems. Why hasn't anyone built a commercial program to do this? There seem to be faster ways to make a buck. (Author: This was another aside in my talk, which I have included here although it does not contribute anything to my general argument).
In work environments, translation managers have to make the most cost-effective use of the translator's time. If a translator can produce 1000 words of fully checked text an hour using MT or TM to generate a draft document and is costing me, his employer, around £20 an hour, and I can sell that text at the rate of £150 per thousand words, that translation tool is giving me a gross margin of £130 per 1000 words.
When considering whether to use or not to use translation tools in a professional translation environment we have to decide whether: -a translator with a workbench tool -a translator revising MT output -or a translator who knows the subject and uses dictation software is going to produce the greatest quantity of acceptable translation at the lowest cost.
We come back to the question of what kind of translation is needed. Even poor MT output can be rendered usable with enough post-editing. If the customer will pay £50 per thousand words for lightly revised computer output and the translator can post-edit at a rate of 2000 words an hour, I am grossing less than when I can sell "ready-to-use translation" at £150 per thousand words. But if the customer will pay £15/1000 for pretty raw computer output and the system can produce 20,000 words an hour, that is far more profitable. At the end of the day, the question of when and when not to use MT comes down to what the customer wants and what he's willing to pay for.


Domain Adaptation with Clustered Language Models

In this paper, a method of domain adaptation for clustered language models is developed. It is based on a previously developed clustering algorithm, but with a modified optimisation criterion. The results are shown to be slightly superior to the previously published 'Fillup' method, which can be used to adapt standard n-gram models. However, the improvement both methods give compared to models built from scratch on the adaptation data is quite small (less than 11% relative improvement in word error rate). This suggests that both methods are still unsatisfactory from a practical point of view.

Introduction
Current large vocabulary speech recognition systems can achieve good performance on domains for which large quantities (e.g. millions of words) of textual data are available to train a language model. In real world applications, however, this is quite often not the case. The issue of language model domain adaptation is therefore of great practical importance.
One approach to tackle this problem is to try to learn from an analogy to the speaker dependence issue: current systems perform well by training speaker independent models, which can then be adapted with relatively little data from a given speaker (see [8]). Can the same approach be applied to language model adaptation?
In section 2, previous work in this area is reviewed and a rough working definition of domain is given. A method to perform domain adaptation with clustered language models is then developed (Section 3). Experimental results to evaluate the method are given in Section 4, followed by conclusions in Section 5.

Background
In order to make the description of domain adaptation more precise, a definition of domain is needed. One might be tempted to define domain in the sense of semantic topic. However, texts might differ in other aspects (e.g. style), which could still require language model adaptation. A more general definition of domain, more in line with the term sublanguage, is therefore required. According to [10], there are many different definitions of the term, but most of them seem to agree on the following characteristics of a sublanguage: 1. it is part of a natural language 2. it is of a specialised form 3. it behaves like a complete language 4. it is used in special circumstances (e.g. expert communication) 5. it is limited to a particular subject domain Some of these points seem very useful for the concept of domain (2,4), others less so (1). What properties should an acceptable definition of domain have?
The following spring to mind: • there should be a continuum (e.g. an infinite number) of domains • each domain may contain an infinite number of elements (e.g. documents/sentences/words) • for a given element, one should be able to decide whether or not it belongs to a given domain Domain adaptation can be divided into static and dynamic domain adaptation, depending on the time scale used to perform adaptation. Dynamic adaptation tries to capture phenomena with a shorter time scale (e.g. topic shifts) and is performed on line, whereas static adaptation can be used to perform a onetime shift from one domain to another and is performed off line. Previous work has shown improvements by using both dynamic adaptation of n-gram models ( [7], [5], [2], [9], [6], [13], [3], [4]) and by using static adaptation of n-gram models ( [9], [6], [1], [13]). Since the 'Fillup' method presented in [1] gives better performance than linear interpolation, the 'Fillup' method is used as method of comparison for the adaptive clustering, which will be developed in the next section.

Adaptive Clustering
The task of a language model is to calculate p(w i |c i ), the probability of the next word being w i given the current context c i . Language models differ in the way this probability is modelled and how the context c i is defined. A quite general model proposed in [12] makes use of a state mapping function S and a category mapping function G. The idea behind the state mapping S : c− > s c = S(c) is to assign each of the large number of possible contexts c ∈ C to one of a smaller number of context-equivalent states. Similarly, the category mapping G : w− > g w = G(w) assigns each of the large number of possible words w ∈ V to one of a smaller number of categories (similar to parts of speech). The probability of the next word is then calculated as In order to determine S and G automatically, a clustering algorithm as shown in Figure 1 can be used. It is a greedy, hill-climbing algorithm that moves Algorithm 1: Clustering() start with initial clustering functions S, G iterate until some convergence criterion is met for all w ∈ V and c ∈ C for all g ′ w ∈ G and s ′ c ∈ S calculate the difference in the optimisation criterion when w/c is moved from g w /s c to g ′ w /s ′ c move the w/c to the g ′ w /s ′ c that results in the biggest improvement in optimisation criterion End Clustering Figure 1: The clustering algorithm elements to the best available choice at any given time. Based on equation 1 and on the leaving-one-out likelihood of the model generating the training data, an optimisation criterion can be derived (see [14] for a detailed description). Let N (e) denote the number of times event e appeared in the training data, let B denote the smoothing parameter used for absolute discounting ( [11]), and let n 0 , n 1 , n + denote the number of pairs (s, g) that have appeared zero, one and one or more times in the training data. The resulting optimisation criterion F (as derived in [14]) is The basic building block in the derivation of equation 2 is the likelihood of one event in the training corpus, as estimated from the training corpus in which this one event has been removed (leaving-one-out likelihood). The main idea behind the adaptive clustering is to use as basic building block the likelihood of one event in S Adapt , as estimated from a linear interpolation of counts from S Back and S Adapt from which this one event has been removed. The motivation for this is that the clustering can thus optimise the perplexity on S Adapt , while having access to a linear combination of counts from S Back and S Adapt .
Let N A (e) (N B (e)) denote the number of times event e appeared in S Adapt (S Back ). Define N C (e) to be the linear interpolation of the two counts where Round(x) returns the integer nearest to x. The only events that can contribute to the optimisation function are events that occur at least once in S Adapt (because, as explained above, the likelihood of S Adapt is taken as optimisation function). However, their probability is calculated based on the combined counts. Therefore, the smoothing has to apply to the combined counts. Define n bi,0 , n bi,1 , n bi,+ as the number of pairs (s, g) that have a combined count N C (s, g) of 0, of 1, and larger than 0. In order to introduce absolute discounting for the unigram estimates as well, also define n s,0 , n s,1 , n s,+ as the number of states s that have a combined count N C (s) of 0, of 1, and larger than 0 (similarly, define n g,0 etc. for the unigram estimates involving g). Changing equation 2 according to the basic idea outlined above, this leads to (ng,0+1) ).
By using the same clustering algorithm as before, but with F Adapt instead of F as optimisation criterion, language model domain adaptation can be performed.

Results
In order to test different adaptation methods, two textual samples S Back and S Adapt and acoustic testing data from D Adapt are required. Since the WSJ domain has the associated acoustic data, it is used as D Adapt . As D Back , the patent domain (PAT) was chosen, for which a large sample S Adapt (about 35 million words are used) is also available from the LDC as part of the TIPSTER database.
The recognition system is a state-of-the-art HMM based system (continuous densities, mixtures, triphones). All experiments are based on bigram language models, either clustered (500 clusters) or backoff (singleton bigrams were ignored). The different methods evaluated were • ClustAdapt: a model built with the adaptive clustering presented in the previous section; the initial starting point for the clustering is taken to be the clustering produced by Back Cl ; one global λ parameter was used and optimised iteratively at the end of each iteration; For all methods except Back Bo and Back Cl , the vocabulary was defined to be all the words that appeared in S Adapt , plus additional words from S Back until 20K words were reached. For Back Bo and Back Cl , the vocabulary consisted of the 20K most frequent words in S Back . Because of this difference, the perplexities of Back Bo and Back Cl are not directly comparable to those of the other models. For each method and a given amount of adaptation material, the perplexity of the resulting model was calculated on a held-out section of S Adapt and a recognition run was performed on the acoustic data.   Table 1 gives the results of the two baseline methods, which do not use any of the adaptation material. The high perplexities show that the PAT and WSJ domains are considerably different. The rate of out-of-vocabulary words is about 15%, which is one reason for the very high error rate. Tables 2, 3, 4 and 5 give the results for the different methods and different amounts of adaptation material.

Model
Comparing Table 3 to Table 2, one can seen that Adapt Cl is more robust than Adapt Bo and it leads to better recognition results for almost the entire range of adaptation material. This is consistent with previous results (see [15]), which showed that clustered models are more robust in terms of perplexity.
Comparing Table 4 to Table 3, one can see that F illup outperforms Adapt Cl in almost all cases.
By looking at

Conclusion
Compared to the success of some methods for acoustic adaptation, the results obtained here are somewhat disappointing. In particular, they seem to suggest that the improvements from the adaptation techniques compared to starting from scratch on the adaptation data become quite small when several tens of thousands of words are available 1 . One reason for this could be the fact that the acoustic space has an underlying distance metric and thus allows the comparison of two elements. Moreover, one can specify the kind of transformations one would want the adaptation to be able to perform. Both of these points seem more difficult in the case of language model adaptation. Even though the adaptation method for clustered language models developed in this paper gives slightly better results than the 'Fillup' method, the accuracies obtained with the adaptive clustering and the 'Fillup' method are still very low compared to the about 80% or more the system can achieve with a backoff bigram trained on about 40 million words of the WSJ corpus. Both adaptation methods are therefore still unsatisfactory from a practical point of view.