Improving the freebsd Translation Tools
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Block, Warren
Translation of FreeBSD’s English documentation into other languages is extremely important to users and the FreeBSD project itself. The old methods used to translate FreeBSD documentation are very labor-intensive. Existing translators have difficulty keeping translations up to date, new translators are discouraged by the depth of expertise required to translate, and even writers working on the English original documents must take extra steps to accommodate translations. The gettext PO translation system automates much of the work and provides new abilities to make translation easier and faster, to share the work with other members of a translation team, and to reuse previous work. The PO translation tools have now been implemented for translating FreeBSD books and articles and this new method has begun to revitalize our translation efforts. Here we describe the benefits of translation, the difficulties with the old method, the benefits of the new system, and some potential challenges and possibilities for the future.
 My name is Warren Block. I should come over here where I can see the screen. I am a FreeBSD Documentation Committer. I have been using FreeBSD since before 1998 or so. In 2011, I got a documentation commit bit. In 2014, I joined the mysterious and misunderstood documentation engineering team. We're going to talk about improving the FreeBSD translation tools. Of course, the first question that comes up is, why bother? Or maybe to put it more politely, is the effort of translating worth the result? And I would say there's a huge portion of the world full of people who could benefit from using FreeBSD and can't because they don't speak English even as a second or third language. There's a huge benefit that we can bring to them. Secondly, I would say that translators are usually the ambassadors into a new area, region, or country. They are not just translators. They use it. They use whatever they have translated. If you can make the job easier for translators, you can also increase the number of people who are using your system in those areas. And finally, you can enlarge the community. Who knows what Metcalfe's law is? You might not know it by that name. Roughly paraphrased, this Bob Metcalfe, the guy who invented Ethernet. Metcalfe's law states paraphrased, the value of a telecommunications network is proportional to the number of nodes in that network. In other words, the more nodes, the more valuable it is. With the Internet, of course, if you had only two nodes on it, it would not be that useful. But if you have a million, it's much more useful. And I say that FreeBSD is a telecommunications network, and the people are the nodes in that network. The more users, the more people discovering bugs, the more people submitting patches, the more people adding features, enhancing documentation, and it increases the value of FreeBSD for the very same people who are doing all that work. So it's a feedback loop. And that, I say, is the value of translation. And so when we talk about the FreeBSD documentation, which is what we're talking about here, there are really several different categories. There are the books and articles which are marked up in DocBook XML. That's the handbook, the very famous handbook. But we also have several other books that are really good. We have the Porter's Handbook, which describes how to port programs. And it's large, and it's detailed, and it is continuously updated, and it's great. Go read it. We have a number of articles also that are written in DocBook XML. Sorry, I shouldn't say written in, marked up with DocBook XML. And of course, DocBook XML gives us the ability to render that source into a number of output formats like HTML or PDF or EPUB or PostScript or ASCII, even plain ASCII. And we have all those. Secondly, we have our man pages. 10 or 20 years ago, some of the open source world kind of didn't quite see the advantage of man pages or didn't really understand it and kind of took a wrong turn. But we in the BSD community, this is a major feature for us. A man page is not a tutorial. It's meant to be a quick reference for assistive men. You forget the exact format of a command or something. You can look in the man page, and there it is. And we have a huge resource in those man pages. They are continuously updated and added to. And those are used in the M.Doc 7 markup language. They're mostly, there are some old ones using various horrible combinations of Roth and Troff and all those. I don't even know all the variations of those. There's anyway. And there are others. The documentation project, part of our domain, includes the source. If there is an error in say spelling or grammar or clarity in source files, we are allowed to edit those. It text files any file, really. So that's the sweep of the free BSD documentation project. For this particular thing, we will be concentrating on the books and articles in Doc McXML. And before we talk about that, we will talk about a fun concept called white space. What is white space? I'm going to pause there while you think about that. White space are the characters between letters that are invisible. Spaces, tabs, carriage returns, line feeds. There are, if I said there, there are several in HTML like non-breaking space. There are probably 100 in UTF-8, I'm sure. The problem for us with these is you can't see them, really. You only see the effect. And so when we try and explain that to people about how white space is a special thing, we say see that there, say no, that's it. It's invisible. And it's difficult to explain because of that. And the reason it's a problem for us is because translators using our old translation method worked by commits. And a commit just shows that a file changed. It is very difficult to tell if it was just white space that changed or if it was the content, the words themselves that changed. And so translators could frequently find themselves retranslating something where the words had not changed. And it was, the translation is a difficult enough job that we don't want to add more to that job for translators. Additionally, because we have to separate changes in content and white space, it adds a job for writers working in the English language because I can find numerous content errors in a document. Oh, this word is spelled wrong. This paragraph makes no sense. I can fix all that, but I can't change white space. I can't rewrap the paragraph so the lines are the right length. I can't delete or add spaces or blank lines. I can't touch white space. So I fix the content and I commit it. And then I fix all the white space changes and only white space changes and commit that with a note to translators that they can ignore this because there's no content changes in it. And then inevitably, when I'm doing that white space fix, I find a few more content errors. So I fix those and commit those. And half the time, that means I need to go back and do another white space commit. So white space is a problem for everybody. And this leads straight into the old translation method that we used, which is simple in concept. It is for all intents and purposes the same as if you were porting a program and keeping it up to date with the upstream version. So you see that the upstream version has changed and you're working by commit. So it's like, okay, I'll see the first commit, the change. I will take that. I will port that to my version of it, translate it in this case. You have to work in the order those commits occurred. You can't say, oh, there was a big bug fix. I want that, but I can't do that because I've got to do these 50 commits that occurred before it. And you don't have any control over the size of the work you're going to do. One commit might be a single character change. One might be 50 pages. You have no control over that as a translator. And of course, we're talking about volunteers who are trying to do a difficult job already and they come in and say, okay, I'm willing to translate for you. And okay, here you go. The next commit you need to work on is this one. It's a rewrite of the whole chapter. And you have to finish it all before you commit it and nobody else can work on that document until that particular commit has been entered. So it's not, translators don't have, I wouldn't say any, they have very little control of how much work they have to do, which is not conducive to volunteer labor. And here is an example. And don't worry if you can't read this very well because all this is, this is the German translation of the X11 chapter of the FreeBSD Handbook. And here is how the German translators keep track of the latest revision in English that they have translated. They put a comment at the beginning of it with that commit number, which is 48082, I believe. And the rest of it is their translated version. So this German version has been translated up to, it is in sync with the English version up to 48082. As a translator then, your next question is, well, show me the next diff I have to translate. And to do that involves a fairly involved, oh, it's 48083 would be the next commit and you have to search all between those. This whole line is to find the next commit that needs to be translated. It turns out you have to have a pretty good familiarity with subversion to do that. So now you're not just dealing with translating, now you need to know the version control system in a non-trivial way. And once you find that, it spits out the commit message and then the diff. And of course, this is an English diff. It shows the English original and the changes. And this particular one was not huge, but notice it says 29 more lines not shown. So it's not trivial. And you need, as a translator, you need to translate all of this. You can't do part of it, you have to do all of it. And many translators will say, well, I was going to do this quick thing and now I don't have time to do that and so it might sit for a while. And another question as a translator you might ask would be, well, how many commits do I have to work through to get this thing up to the current version to where it's totally in sync? And again, you need to know a fairly involved command for the version control system. And this lists the commits that have occurred to that file since that last version number that has been translated. However, looking at this, you can't tell. Some of these might be white space only commits that you don't have to translate at all. Some might be a rewrite of the entire chapter which throws out all the work you've already done. And again, it's not conducive to volunteers, especially people doing technically difficult work. At some point then finally, as a translator, you will find yourself looking at something like this. This is the English current version of the handbook chapter on X11 and the German translation, which is not quite up to that level yet. Now think for a second about the different lengths of sentences in different languages. Some of us, like Americans, we think German words are really long. They aren't always. But the translation team has their own rules about how to wrap paragraphs so that their text is easy to read for their translators. Well, what that means is this English document and this German document do not correspond to one-to-one at all. There is no way you can say find the part in the German document that corresponds to where this change occurred in the English document because that's where I have to make my translations. It doesn't. This is just two editors side by side, which scroll individually. So people come to us and they say, I would like to help translate your documentation. And we say to them, okay, do you know Doc McXML? No. Do you know Subversion? No. Okay, here's a book, the FreeBSD Documentation Project Primer, that describes all that. Read it and then come back when you're done. And then we will be happy to accept the gift you're offering us of your difficult work on translating this intense technical documentation. And many of them turn around and say, too much for me. And rightly so. The interesting part of this for me is that we have existing translation teams that have kept up for years using this method. And that's dedication. I can't believe they've done that. So over the last few years, several people have worked on using GetText. Just to review, GetText was a system that was written to write programs that didn't have to be recompiled to work in different languages. The programmer or possibly a separate translator would translate all the prompts and text strings out of the program into a separate file. And the user would set their locale on their machine. They run the program. It appears translated to them. It's magic. It doesn't sound like that particular system would work well for documentation because documentation is lines of text and paragraphs. But it actually turns out to work pretty well. And many other large open source projects are using it. And closed source for all I know. And the way that works is that translators work on these PO files. A program comes with you run a program that extracts translatable strings from the XML document, in our case, the DocBook XML documents. It extracts translatable strings from that file, generates a PO file. The translator runs a PO editor which shows the English string and the translated string side by side with a one-to-one correspondence. There is no searching and there is no fighting with a version control system. White space in these is generally not an issue. It can come up sometimes, but let's put it this way. It's far less of an issue because strings are sorted to be translated and white space changes are XML separators. Is that what you would call it? Delimiters? And so much of the time changes in white space are invisible because it's just ignored by the XML. Finally there is translation memory. This is neither as good nor as bad as it sounds. What that is is once you've translated a string and now you're translating a new document, you can tell the PO editor, show me all the strings that I've already translated that you already know so I don't have to translate them again. It's commonly misunderstood how this works. It depends. The strings must match in length. If you translate a single word and that word is used in a sentence and now you're translating a sentence that has that word in it, the translation memory will not help you there. And that's a good thing because, well, the example I found was in check. I believe it was in check. There are eight different ways to correctly pluralize or use a noun and if you automatically translate it, it won't work. It will be the wrong one every time. So yes, commonly repeated text will be in that translation memory and no, it will not, you might catch at an estimate 5 to 15% of the text with that. Which removing 5 to 15% of the translator's work is worth it. But yes, I always get people saying, oh, that translation memory, it's going to screw up because this word is translated differently depending on how it's used. Well, no, not really. But it won't do all the work for you either. So it's neither as good nor as bad as you would hope. The new translation method using PO files, this is it. All of it. This is the entire process. And of these four lines on the screen, the second line is an output line. So there are three commands. Make PO, which uses that program to extract the translatable strings from the XML file. And then you run a PO editor on that generated PO file. And then when you're done, you run make-tran and it creates the translated XML file. That's it. And you notice here what we didn't have to do. We didn't have to run the version control system at all. We didn't have to try and figure out from looking at files side by side where the translated version goes in comparison to the English version. None of that. This is an example of a PO editor. The PO edit, it's kind of the lowest common denominator. And don't worry again if you can't read this. I will show you what's going on. This is the English version of that leap seconds article we did last year, which turns out to be a pretty good example for translators because it's short. Because it's short, I should say. Because. What? It's a leap second to short. It's a leap second to short. Yes. On the left side here, we see the English version. On the right side, we see a Spanish translation of it. And you see this line highlighted here is the current one being worked on. At the bottom of the screen, we see the full version of the English string and then below it the full version of the Spanish string. And I would like to point out here, one weakness of this system, well, before I talk about the weakness, notice it's one to one. There is no searching for what your translations are. There's the thing you have to translate and then beside it, the place to enter the translation. If you have untranslated strings in here on the Spanish side, they will show as blanks. And usually, the editors are set to show those at the top as the ones that need to be translated. You aren't forced to work in commit order. You will see blank lines on the right side there for anything that hasn't been translated. It doesn't matter if there's been 100 commits or one, and it doesn't matter the order that you work on. If you want to translate one string today and save it, you can. And now I will say, talk about the weakness. On this line, you can't really see it on this screen, but it says cautions in the original. And I happen to know that was a section title, cautions about the upcoming leap second. And the section title was just a single word caution. And it's been translated, and pardon my pronunciation, it's been translated to precautiones, which I think in Spanish, Spanish would have a TH sound. I don't know. I'm an American. I'm monolingual. That word is an example of not showing context. Get text was originally written for translating programs. And in a program, you don't really have context. You have like the file menu. There's not, you know, file print. In text like this, it would really help the translator if they could see the text in the English version that happened before and after that caution so they could see the context, how that word cautions was used, because that definitely will affect how it is translated into the translated language. And most of the PO editors I've looked at do not show context, but I believe it's merely a, well, it's not a difficult issue to solve that. I think that the reason they don't show that by default is because it was and is still used to translate programs where context is not really a huge problem. And how is this done? How is this implemented? Well, there were several choices for programs to use to extract those strings, extract the translatable strings from the XML file. One of them was Debian's PO4A program. And 4A stands for for anything. And the idea there is you can take this and run this program against various file formats and it will generate a PO file, which is cool. It's been around a long time. I didn't get the feeling that development was very active on it. And then also, I knew Sean McCants of the Nome Project and he wrote this program called ITS Tool. And ITS Tool is new. It's only a couple years old. And it's very standards compliant. It's a small program written in Python. And it's active. I know the guy. I can bug him if it doesn't work. Also we use message merge and message format, which are from the Get Text Tools port or package. The interesting thing with these two is we already had a Python dependency in the documentation tool chain. So we didn't add a Python interpreter. We didn't need a Python interpreter just for this. So we really didn't add any overhead due to that. And most documentation development systems will probably have the Get Text Tools port installed already. So essentially it's tiny, the extra overhead of adding this to the doc tool chain. And so here, results. This is the important part, right? At the very end of August 2015 was when we went live with this. Since then we've had 11 new translations of articles and books into German, Dutch, Spanish, Chinese, and Korean. And I gathered that information by hand. I looked at it and it's like, okay. Is that a lot? Is that very few? I don't know. So it turned out that I had to write a non-trivial script to go through and collect this information from the repository and it's not easy to do. It turned out to be very non-trivial. Incidentally, I love showing graphs in a presentation like this. It's very PowerPoint-y. This is a graph of the number of newly translated articles and books that were added to the FreeBSD project over the years. Going from 2005 all the way to current. And you can see that at its peak there were 17 in a year. Now this can sort of distort the numbers because there was ongoing maintenance of things that had already been translated. But new translations, I thought, kind of gives you an idea of the activity level. And you can tell starting around 2009 it started to drop off. And we got a peak again at 2012, but for 2011 and 2014 there were no new translations. And sometime in this timeframe we had translation teams contact the doc project and say, we can't keep up. There have been so many changes in the documentation. We just can't keep up with them. And most of them were only trying to translate the handbook, not all of the documents and articles. I didn't say it, but I would say that was because of the old translation method. There was so much manual overhead that had to be done and so much non-value added work before you could actually get to a place where you could translate text. That was what was holding it up. Now on this particular graph, this last bar is somewhat misleading. So I've got that in a separate graph. And you can see here that for 2015, this goes to February of 2016, for 2015 we were going to have another 2014 with no new translations at all. Until the very end of August 2015 when this PO translation system went live. Incidentally the old system can still be used. The new one does not substitute it. The old one still works. All of these new translations were PO translations. We have had no new old system translations. Which is understandable. Now this is too soon to be a trend in fairness. But what I say is that this shows that now we have a hope of reversing that trend. Now we've got the tools to, so when translators come to us and say, I'm willing to volunteer and help translate your documentation into my language, we can say, hey, it's easy. The hard part is the translation. That would be the ideal thing. We do still have some challenges. And I call these challenges because when you've got a challenge, or when you've got a problem, it's just a problem. When you've got a challenge, you're challenged to come up with a solution. These are in no particular order. We have things that should not be translated. And the prime example of these are the, we have, sorry. The graph. Oh, the previous one. There. These were committed. These were new translations that were committed. If I'm not aware of a, it was Portuguese. I'm not aware of a committed Portuguese handbook then. But it's possible that I or my script missed it. Because like I say, this was sort of non-trivial to collect this information out of there. But it would be great that we had one in 2014. I mean, you know, that's, well, any is better than none. I have the, no, it would, it looked for the make files. The script looks for the make files in language subdirectories for books or articles. And the good news with this is I do have the script online. And I have a link to that at the end of this. So anyway, so the question was, why wasn't the Portuguese translation that was submitted of the handbook in, that was submitted in 2014 shown here? And the answer could be a number of things. The, I guess the answer is I don't know. But it could be due to a number of factors why it's not shown there. And probably first on the list is that my script didn't do it, didn't count it correctly. But that's good. That, I mean, that's to be encouraged. Anyway, things that should not be translated. The prime example is we have an article that contains all of the developers' PGP keys. That's all it is. There's, there's like two sentences of introduction. These are the developers' PGP keys and then 600 pages of PGP keys, which are numbers, right? We don't want those translated because the translated form of that is identical to the original. So first off, you're wasting the translators' time by showing them that. But also, we kind of want a single source for those. They should come out of one file. If they're translated, they're copied into the translated file. And now you have multiple sources for them. Some will be out of date always. And so we need a way to mark up maybe the XML and say this thing should not be translated. Ideally the translator would never even see it. So for that article, instead of seeing 600 pages of XML source, they would see the two little sentences that say these are the developer PGP keys. Now I went and searched for ways to do that and did not find any. Or rather, I found a few places that had done it, but they did it specific to their own tool chain. It didn't really work for what we used. The only way I could think of to do it was kind of a messy hack involving XML processing instructions to put our own markers around those. And then in our tool chain, maybe use temporary files and delete those from the pre-PO version. And then after the translator was done with it, add them back. It was really kind of an ugly hack. Here's our, by the way, all things I am looking for suggestions on. If you know a way to do this or even if you know a way that seems like a way to do this, but turns out to be a mistake, I am interested in that. Bad examples are good sometimes, too. Not bad examples, but examples of the way not to do it. We have another issue, which is we have a huge amount of work involved in our existing translations, the ones that were created with the old system. And we don't want to lose that. There was a lot of effort involved in creating that. How can we convert those into PO so we can reuse those? So we can preserve all that effort that went into them, but still let new translators use the PO system. There is a program called XML2PO, which is kind of an old known program. Incidentally, I say known because the known guys say known sometimes and get known other times, so I just say known. That program says that if you have the English original document and the translated document and they correspond line to line 100%, it will create a PO file out of that. That would be fine, except remember when we looked at the German translation and I said that the German translation team has their own rules for wrapping paragraphs so that the translators can read them on the screen. So the English and the German translation do not have a one to one line correspondence. I'm looking for ways around that. I look at that and say, okay, they're both XML files. If they are both current, both up to the same revision, the XML elements should correspond to one to one. Somebody who is into that whole XML thing might be able to write a program that does that. I would think it might also be an interesting problem for like a graduate student at a college somewhere because it's non-trivial and it involves language processing potentially, could be kind of neat. But I suspect that there are a lot of places when they switch from an old translation method to a new one, rather than spend the time writing a one-time conversion program, they'll just manually jump in and translate it all by hand. I suspect because it's a one-time use. Once they've switched, they won't need that again. Anyway, showing context is another thing that I talked about a little bit ago. PO editors could show context and it's not a technical problem because in the PO file with the English line, there is a comment that shows the line number it came out of on the XML file. So there's no reason the PO editor knows where to look for the text before and after it and it could show it. And along those lines, more PO editors. Right now we have three that I know of, well, two and a half. We have PO edit, we have G translator, which is a GNOME, I forget the graphic toolkit it uses. There's a KDE version whose name I forget. Context? Localize. Localize. With a K. Yes. Thank you. We have those three in ports. But there are many more out there that do different things. And the reason I think it would benefit us to have more PO editors is because I suspect that one PO editor might support things like the CJK languages, Chinese, Japanese, and Korean. It might work really well for those, while another one might be really great for, say, the European languages. And that choice of editors, particularly in the BSD community, where for 30 years we haven't been able to decide between VI and Emacs, I think that having a choice of editors like that would be good. There are also online editors like Poodle and Weblate. PCBSD has been using Poodle. And translators connect to their website, and they run it all online. There is no end user stuff to install. But I'm not sure about FreeBSD running Poodle or Weblate on their own because I suspect I'm concerned about limiting the choice of editors due to that. There is one other PO editor called Vertal, V-I-R-T-A-A-L. And that's a very highly regarded PO editor. And we have a working port of it, except it doesn't let you edit strings or see the string you're working on. It's 99% there. Mark Felder brought it the rest of the way. I believe he was the last one who worked on it and did the most work. And it just, it needs somebody who's into that. I believe it's Python GTK stuff. I think there's just something minor that needs to be done. And I will have a link to that here coming up. There are other PO editors. There are PO editors written in Java that will run on various platforms. There are even commercial web-based ones. There's one called TransFX, where it's free to use for open source projects. But I'm not sure we want to use that. Anyway, those are challenges. What about potential? Where could we go with this? And for me, the first thing I see is man pages. I want full translations of all the man pages. But ITS tool only works on XML files. But remember Debian's PO4A? 4A is for anything. They are supposed to support the man page format. I have not tested that. But wouldn't it be awesome if we got so many translators involved and it was so easy to do that we had current up-to-date translations of our man pages also? Which for me are almost more important than the documentation. You really use the documentation a lot when you're first setting something up. But the man pages are an online reference that you use all the time. For me, for instance, is it capital R or lowercase r and CP? Or does it matter? And that's what a man page is for. So for new non-English speaking people, I would really like to see the man pages translated. And if it's easy to do with PO files, why not? That would be a real selling point. The other potential of this is new translators because it's easy, if we make it easy to translate, we get new people working on docs. And new people working on docs, especially if they speak both English and another language, they tend to find errors in the English versions. Because I've said this before, people who speak English as a second language tend to speak it better than the natives because they care more is my experience. The other night, we had Bjorn Heidt, high dotting, who's probably saying his name wrong. He's a very active committer to the German project. And he submitted a review where he found errors in the English version. It's like, yeah, that's exactly, it was a wonderful thing to see. And more contributors there, those contributors find more bugs and some of them turn into committers. And in turn, they help other people. And in turn, because of that, you get new free BSD users and contributors in other areas. And so we're back to that Metcalfe's law feedback loop. So it's great. What we learned doing this, the first one is absolutely the value of cross-pollinating with other projects. Last year at BSD CAN, Ryan Lordy of the GNOME project and I were sitting in one of these conference rooms waiting for a talk to begin. And I said, oh, I would ask questions like, well, what do you do about this? And it wasn't a big, you know, it was how do you handle merging PO files, existing ones with new ones? Oh, well, you just do this. Really? That's it? And that interaction on that is just don't underestimate the value of that. We should not look at other projects as competitors. We should look at them as rich gold mines of information. We learned about commit by commit porting, which is really what the old translation process is. It's porting a document from English to another language. And that came up because the guys who were porting X11, trying to keep the X11 system up to date, posted on their mailing list and said, well, we're thinking about going file by file or commit by commit. What do people think? And I said, I'm not so sure about the benefits and problems with file by file, but here is what we've seen with commit by commit. You might have something that's critically important, but you can't commit it because there are all these other commits in line before it. You have no control over the size of work you have to do. And so a really large commit can stall the whole project while multiple people wait for that commit to be ported. And finally, with commit by commit, you might port 100 of them and find that number 101 rewrites the whole thing and wipes out all that work you've just done. So we were able to contribute on that. We found out about technical debt. When you say to somebody, oh, we do this in XML to include chapters, and they pause for a moment and say, I didn't think anybody did it that way anymore. That's kind of a warning sign that you're getting behind. And while we might be compatible with the standard, it could also mean that new stuff might not support the old way we're doing it. And to be safe, it's really best to keep up with the herd. Along those lines, UTF-8. Right now, our documentation is mostly not in UTF-8. There are a few translated languages that use it. Most do not. English does not. We need to just go ahead and convert. That needs to be done. XML techniques. The modern technique for including an external file is Xinclude. We don't use that. That did not exist when the FreeBSD documentation first got going. So we don't use it. Now the way we do it is we define like a chapter as an entity and use that entity. It works. It's standards compliant, but I have the feeling that at some point in time there will be a utility or tool we want to use that says, well, yes, we know that's part of the standard, but we don't do things that way anymore. So again, technical debt, you have to keep up with the herd or you'll find yourself in trouble at some point. And finally, I shouldn't say, and finally, acknowledgments. This was not me. This was a bunch of people who did a lot of work, Thomas Abtorp is here. A bunch of people have pushed and worked on this over the years, and it's a hard thing to solve because our translation method was part of a legacy system, and it's always hard to try and bring something new into that. And so I want to thank all these people for all the help they've done. And there's probably about another 20 that should be on there that I just didn't remember or worked on it before I even knew about it. And then at that URL is this presentation in PDF form, is the article that this presentation was based on in PDF form, is this presentation in open office form, I believe. The virtual almost working port is here. The shell script that gathers the information from the doc repository about new translations is there, and the text file and a spreadsheet from that. And there might be two or three things. Get it now while it's available. Any questions? No? Thank you.
 My name is Warren Block.  I should come over here where I can see the screen.  I am a FreeBSD Documentation Committer.  I have been using FreeBSD since before 1998 or so.  In 2011, I got a documentation commit bit.  In 2014, I joined the mysterious and misunderstood documentation engineering team.  We're going to talk about improving the FreeBSD translation tools.  Of course, the first question that comes up is, why bother?  Or maybe to put it more politely, is the effort of translating worth the result?  And I would say there's a huge portion of the world full of people who could benefit  from using FreeBSD and can't because they don't speak English even as a second or third  language.  There's a huge benefit that we can bring to them.  Secondly, I would say that translators are usually the ambassadors into a new area, region,  or country.  They are not just translators.  They use it.  They use whatever they have translated.  If you can make the job easier for translators, you can also increase the number of people  who are using your system in those areas.  And finally, you can enlarge the community.  Who knows what Metcalfe's law is?  You might not know it by that name.  Roughly paraphrased, this Bob Metcalfe, the guy who invented Ethernet.  Metcalfe's law states paraphrased, the value of a telecommunications network is proportional  to the number of nodes in that network.  In other words, the more nodes, the more valuable it is.  With the Internet, of course, if you had only two nodes on it, it would not be that  useful.  But if you have a million, it's much more useful.  And I say that FreeBSD is a telecommunications network, and the people are the nodes in  that network.  The more users, the more people discovering bugs, the more people submitting patches,  the more people adding features, enhancing documentation, and it increases the value  of FreeBSD for the very same people who are doing all that work.  So it's a feedback loop.  And that, I say, is the value of translation.  And so when we talk about the FreeBSD documentation, which is what we're talking about here, there  are really several different categories.  There are the books and articles which are marked up in DocBook XML.  That's the handbook, the very famous handbook.  But we also have several other books that are really good.  We have the Porter's Handbook, which describes how to port programs.  And it's large, and it's detailed, and it is continuously updated, and it's great.  Go read it.  We have a number of articles also that are written in DocBook XML.  Sorry, I shouldn't say written in, marked up with DocBook XML.  And of course, DocBook XML gives us the ability to render that source into a number of output  formats like HTML or PDF or EPUB or PostScript or ASCII, even plain ASCII.  And we have all those.  Secondly, we have our man pages.  10 or 20 years ago, some of the open source world kind of didn't quite see the advantage  of man pages or didn't really understand it and kind of took a wrong turn.  But we in the BSD community, this is a major feature for us.  A man page is not a tutorial.  It's meant to be a quick reference for assistive men.  You forget the exact format of a command or something.  You can look in the man page, and there it is.  And we have a huge resource in those man pages.  They are continuously updated and added to.  And those are used in the M.Doc 7 markup language.  They're mostly, there are some old ones using various horrible combinations of Roth and  Troff and all those.  I don't even know all the variations of those.  There's anyway.  And there are others.  The documentation project, part of our domain, includes the source.  If there is an error in say spelling or grammar or clarity in source files, we are allowed  to edit those.  It text files any file, really.  So that's the sweep of the free BSD documentation project.  For this particular thing, we will be concentrating on the books and articles in Doc McXML.  And before we talk about that, we will talk about a fun concept called white space.  What is white space?  I'm going to pause there while you think about that.  White space are the characters between letters that are invisible.  Spaces, tabs, carriage returns, line feeds.  There are, if I said there, there are several in HTML like non-breaking space.  There are probably 100 in UTF-8, I'm sure.  The problem for us with these is you can't see them, really.  You only see the effect.  And so when we try and explain that to people about how white space is a special thing,  we say see that there, say no, that's it.  It's invisible.  And it's difficult to explain because of that.  And the reason it's a problem for us is because translators using our old translation method  worked by commits.  And a commit just shows that a file changed.  It is very difficult to tell if it was just white space that changed or if it was the  content, the words themselves that changed.  And so translators could frequently find themselves retranslating something where the words had  not changed.  And it was, the translation is a difficult enough job that we don't want to add more  to that job for translators.  Additionally, because we have to separate changes in content and white space, it adds  a job for writers working in the English language because I can find numerous content errors  in a document.  Oh, this word is spelled wrong.  This paragraph makes no sense.  I can fix all that, but I can't change white space.  I can't rewrap the paragraph so the lines are the right length.  I can't delete or add spaces or blank lines.  I can't touch white space.  So I fix the content and I commit it.  And then I fix all the white space changes and only white space changes and commit that  with a note to translators that they can ignore this because there's no content changes in  it.  And then inevitably, when I'm doing that white space fix, I find a few more content errors.  So I fix those and commit those.  And half the time, that means I need to go back and do another white space commit.  So white space is a problem for everybody.  And this leads straight into the old translation method that we used, which is simple in concept.  It is for all intents and purposes the same as if you were porting a program and keeping  it up to date with the upstream version.  So you see that the upstream version has changed and you're working by commit.  So it's like, okay, I'll see the first commit, the change.  I will take that.  I will port that to my version of it, translate it in this case.  You have to work in the order those commits occurred.  You can't say, oh, there was a big bug fix.  I want that, but I can't do that because I've got to do these 50 commits that occurred before  it.  And you don't have any control over the size of the work you're going to do.  One commit might be a single character change.  One might be 50 pages.  You have no control over that as a translator.  And of course, we're talking about volunteers who are trying to do a difficult job already  and they come in and say, okay, I'm willing to translate for you.  And okay, here you go.  The next commit you need to work on is this one.  It's a rewrite of the whole chapter.  And you have to finish it all before you commit it and nobody else can work on that document  until that particular commit has been entered.  So it's not, translators don't have, I wouldn't say any, they have very little control of  how much work they have to do, which is not conducive to volunteer labor.  And here is an example.  And don't worry if you can't read this very well because all this is, this is the German  translation of the X11 chapter of the FreeBSD Handbook.  And here is how the German translators keep track of the latest revision in English that  they have translated.  They put a comment at the beginning of it with that commit number, which is 48082, I  believe.  And the rest of it is their translated version.  So this German version has been translated up to, it is in sync with the English version  up to 48082.  As a translator then, your next question is, well, show me the next diff I have to translate.  And to do that involves a fairly involved, oh, it's 48083 would be the next commit and  you have to search all between those.  This whole line is to find the next commit that needs to be translated.  It turns out you have to have a pretty good familiarity with subversion to do that.  So now you're not just dealing with translating, now you need to know the version control  system in a non-trivial way.  And once you find that, it spits out the commit message and then the diff.  And of course, this is an English diff.  It shows the English original and the changes.  And this particular one was not huge, but notice it says 29 more lines not shown.  So it's not trivial.  And you need, as a translator, you need to translate all of this.  You can't do part of it, you have to do all of it.  And many translators will say, well, I was going to do this quick thing and now I don't  have time to do that and so it might sit for a while.  And another question as a translator you might ask would be, well, how many commits  do I have to work through to get this thing up to the current version to where it's totally  in sync?  And again, you need to know a fairly involved command for the version control system.  And this lists the commits that have occurred to that file since that last version number  that has been translated.  However, looking at this, you can't tell.  Some of these might be white space only commits that you don't have to translate at all.  Some might be a rewrite of the entire chapter which throws out all the work you've already  done.  And again, it's not conducive to volunteers, especially people doing technically difficult  work.  At some point then finally, as a translator, you will find yourself looking at something  like this.  This is the English current version of the handbook chapter on X11 and the German translation,  which is not quite up to that level yet.  Now think for a second about the different lengths of sentences in different languages.  Some of us, like Americans, we think German words are really long.  They aren't always.  But the translation team has their own rules about how to wrap paragraphs so that their  text is easy to read for their translators.  Well, what that means is this English document and this German document do not correspond  to one-to-one at all.  There is no way you can say find the part in the German document that corresponds to  where this change occurred in the English document because that's where I have to make  my translations.  It doesn't.  This is just two editors side by side, which scroll individually.  So people come to us and they say, I would like to help translate your documentation.  And we say to them, okay, do you know Doc McXML?  No.  Do you know Subversion?  No.  Okay, here's a book, the FreeBSD Documentation Project Primer, that describes all that.  Read it and then come back when you're done.  And then we will be happy to accept the gift you're offering us of your difficult work  on translating this intense technical documentation.  And many of them turn around and say, too much for me.  And rightly so.  The interesting part of this for me is that we have existing translation teams that have  kept up for years using this method.  And that's dedication.  I can't believe they've done that.  So over the last few years, several people have worked on using GetText.  Just to review, GetText was a system that was written to write programs that didn't  have to be recompiled to work in different languages.  The programmer or possibly a separate translator would translate all the prompts and text strings  out of the program into a separate file.  And the user would set their locale on their machine.  They run the program.  It appears translated to them.  It's magic.  It doesn't sound like that particular system would work well for documentation because  documentation is lines of text and paragraphs.  But it actually turns out to work pretty well.  And many other large open source projects are using it.  And closed source for all I know.  And the way that works is that translators work on these PO files.  A program comes with you run a program that extracts translatable strings from the XML  document, in our case, the DocBook XML documents.  It extracts translatable strings from that file, generates a PO file.  The translator runs a PO editor which shows the English string and the translated string  side by side with a one-to-one correspondence.  There is no searching and there is no fighting with a version control system.  White space in these is generally not an issue.  It can come up sometimes, but let's put it this way.  It's far less of an issue because strings are sorted to be translated and white space  changes are XML separators.  Is that what you would call it?  Delimiters?  And so much of the time changes in white space are invisible because it's just ignored by  the XML.  Finally there is translation memory.  This is neither as good nor as bad as it sounds.  What that is is once you've translated a string and now you're translating a new document,  you can tell the PO editor, show me all the strings that I've already translated that  you already know so I don't have to translate them again.  It's commonly misunderstood how this works.  It depends.  The strings must match in length.  If you translate a single word and that word is used in a sentence and now you're translating  a sentence that has that word in it, the translation memory will not help you there.  And that's a good thing because, well, the example I found was in check.  I believe it was in check.  There are eight different ways to correctly pluralize or use a noun and if you automatically  translate it, it won't work.  It will be the wrong one every time.  So yes, commonly repeated text will be in that translation memory and no, it will not,  you might catch at an estimate 5 to 15% of the text with that.  Which removing 5 to 15% of the translator's work is worth it.  But yes, I always get people saying, oh, that translation memory, it's going to screw up  because this word is translated differently depending on how it's used.  Well, no, not really.  But it won't do all the work for you either.  So it's neither as good nor as bad as you would hope.  The new translation method using PO files, this is it.  All of it.  This is the entire process.  And of these four lines on the screen, the second line is an output line.  So there are three commands.  Make PO, which uses that program to extract the translatable strings from the XML file.  And then you run a PO editor on that generated PO file.  And then when you're done, you run make-tran and it creates the translated XML file.  That's it.  And you notice here what we didn't have to do.  We didn't have to run the version control system at all.  We didn't have to try and figure out from looking at files side by side where the translated  version goes in comparison to the English version.  None of that.  This is an example of a PO editor.  The PO edit, it's kind of the lowest common denominator.  And don't worry again if you can't read this.  I will show you what's going on.  This is the English version of that leap seconds article we did last year, which turns out  to be a pretty good example for translators because it's short.  Because it's short, I should say.  Because.  What?  It's a leap second to short.  It's a leap second to short.  Yes.  On the left side here, we see the English version.  On the right side, we see a Spanish translation of it.  And you see this line highlighted here is the current one being worked on.  At the bottom of the screen, we see the full version of the English string and then below  it the full version of the Spanish string.  And I would like to point out here, one weakness of this system, well, before I talk about  the weakness, notice it's one to one.  There is no searching for what your translations are.  There's the thing you have to translate and then beside it, the place to enter the translation.  If you have untranslated strings in here on the Spanish side, they will show as blanks.  And usually, the editors are set to show those at the top as the ones that need to be translated.  You aren't forced to work in commit order.  You will see blank lines on the right side there for anything that hasn't been translated.  It doesn't matter if there's been 100 commits or one, and it doesn't matter the order that  you work on.  If you want to translate one string today and save it, you can.  And now I will say, talk about the weakness.  On this line, you can't really see it on this screen, but it says cautions in the original.  And I happen to know that was a section title, cautions about the upcoming leap second.  And the section title was just a single word caution.  And it's been translated, and pardon my pronunciation, it's been translated to precautiones, which  I think in Spanish, Spanish would have a TH sound.  I don't know.  I'm an American.  I'm monolingual.  That word is an example of not showing context.  Get text was originally written for translating programs.  And in a program, you don't really have context.  You have like the file menu.  There's not, you know, file print.  In text like this, it would really help the translator if they could see the text in the  English version that happened before and after that caution so they could see the context,  how that word cautions was used, because that definitely will affect how it is translated  into the translated language.  And most of the PO editors I've looked at do not show context, but I believe it's merely  a, well, it's not a difficult issue to solve that.  I think that the reason they don't show that by default is because it was and is still  used to translate programs where context is not really a huge problem.  And how is this done?  How is this implemented?  Well, there were several choices for programs to use to extract those strings, extract the  translatable strings from the XML file.  One of them was Debian's PO4A program.  And 4A stands for for anything.  And the idea there is you can take this and run this program against various file formats  and it will generate a PO file, which is cool.  It's been around a long time.  I didn't get the feeling that development was very active on it.  And then also, I knew Sean McCants of the Nome Project and he wrote this program called  ITS Tool.  And ITS Tool is new.  It's only a couple years old.  And it's very standards compliant.  It's a small program written in Python.  And it's active.  I know the guy.  I can bug him if it doesn't work.  Also we use message merge and message format, which are from the Get Text Tools port or  package.  The interesting thing with these two is we already had a Python dependency in the documentation  tool chain.  So we didn't add a Python interpreter.  We didn't need a Python interpreter just for this.  So we really didn't add any overhead due to that.  And most documentation development systems will probably have the Get Text Tools port  installed already.  So essentially it's tiny, the extra overhead of adding this to the doc tool chain.  And so here, results.  This is the important part, right?  At the very end of August 2015 was when we went live with this.  Since then we've had 11 new translations of articles and books into German, Dutch, Spanish,  Chinese, and Korean.  And I gathered that information by hand.  I looked at it and it's like, okay.  Is that a lot?  Is that very few?  I don't know.  So it turned out that I had to write a non-trivial script to go through and collect this information  from the repository and it's not easy to do.  It turned out to be very non-trivial.  Incidentally, I love showing graphs in a presentation like this.  It's very PowerPoint-y.  This is a graph of the number of newly translated articles and books that were added to the  FreeBSD project over the years.  Going from 2005 all the way to current.  And you can see that at its peak there were 17 in a year.  Now this can sort of distort the numbers because there was ongoing maintenance of things that  had already been translated.  But new translations, I thought, kind of gives you an idea of the activity level.  And you can tell starting around 2009 it started to drop off.  And we got a peak again at 2012, but for 2011 and 2014 there were no new translations.  And sometime in this timeframe we had translation teams contact the doc project and say, we  can't keep up.  There have been so many changes in the documentation.  We just can't keep up with them.  And most of them were only trying to translate the handbook, not all of the documents and  articles.  I didn't say it, but I would say that was because of the old translation method.  There was so much manual overhead that had to be done and so much non-value added work  before you could actually get to a place where you could translate text.  That was what was holding it up.  Now on this particular graph, this last bar is somewhat misleading.  So I've got that in a separate graph.  And you can see here that for 2015, this goes to February of 2016, for 2015 we were going  to have another 2014 with no new translations at all.  Until the very end of August 2015 when this PO translation system went live.  Incidentally the old system can still be used.  The new one does not substitute it.  The old one still works.  All of these new translations were PO translations.  We have had no new old system translations.  Which is understandable.  Now this is too soon to be a trend in fairness.  But what I say is that this shows that now we have a hope of reversing that trend.  Now we've got the tools to, so when translators come to us and say, I'm willing to volunteer  and help translate your documentation into my language, we can say, hey, it's easy.  The hard part is the translation.  That would be the ideal thing.  We do still have some challenges.  And I call these challenges because when you've got a challenge, or when you've got a problem,  it's just a problem.  When you've got a challenge, you're challenged to come up with a solution.  These are in no particular order.  We have things that should not be translated.  And the prime example of these are the, we have, sorry.  The graph.  Oh, the previous one.  There.  These were committed.  These were new translations that were committed.  If I'm not aware of a, it was Portuguese.  I'm not aware of a committed Portuguese handbook then.  But it's possible that I or my script missed it.  Because like I say, this was sort of non-trivial to collect this information out of there.  But it would be great that we had one in 2014.  I mean, you know, that's, well, any is better than none.  I have the, no, it would, it looked for the make files.  The script looks for the make files in language subdirectories for books or articles.  And the good news with this is I do have the script online.  And I have a link to that at the end of this.  So anyway, so the question was, why wasn't the Portuguese translation that was submitted  of the handbook in, that was submitted in 2014 shown here?  And the answer could be a number of things.  The, I guess the answer is I don't know.  But it could be due to a number of factors why it's not shown there.  And probably first on the list is that my script didn't do it, didn't count it correctly.  But that's good.  That, I mean, that's to be encouraged.  Anyway, things that should not be translated.  The prime example is we have an article that contains all of the developers' PGP keys.  That's all it is.  There's, there's like two sentences of introduction.  These are the developers' PGP keys and then 600 pages of PGP keys, which are numbers,  right?  We don't want those translated because the translated form of that is identical to the  original.  So first off, you're wasting the translators' time by showing them that.  But also, we kind of want a single source for those.  They should come out of one file.  If they're translated, they're copied into the translated file.  And now you have multiple sources for them.  Some will be out of date always.  And so we need a way to mark up maybe the XML and say this thing should not be translated.  Ideally the translator would never even see it.  So for that article, instead of seeing 600 pages of XML source, they would see the two  little sentences that say these are the developer PGP keys.  Now I went and searched for ways to do that and did not find any.  Or rather, I found a few places that had done it, but they did it specific to their own  tool chain.  It didn't really work for what we used.  The only way I could think of to do it was kind of a messy hack involving XML processing  instructions to put our own markers around those.  And then in our tool chain, maybe use temporary files and delete those from the pre-PO version.  And then after the translator was done with it, add them back.  It was really kind of an ugly hack.  Here's our, by the way, all things I am looking for suggestions on.  If you know a way to do this or even if you know a way that seems like a way to do this,  but turns out to be a mistake, I am interested in that.  Bad examples are good sometimes, too.  Not bad examples, but examples of the way not to do it.  We have another issue, which is we have a huge amount of work involved in our existing  translations, the ones that were created with the old system.  And we don't want to lose that.  There was a lot of effort involved in creating that.  How can we convert those into PO so we can reuse those?  So we can preserve all that effort that went into them, but still let new translators  use the PO system.  There is a program called XML2PO, which is kind of an old known program.  Incidentally, I say known because the known guys say known sometimes and get known other  times, so I just say known.  That program says that if you have the English original document and the translated document  and they correspond line to line 100%, it will create a PO file out of that.  That would be fine, except remember when we looked at the German translation and I said  that the German translation team has their own rules for wrapping paragraphs so that  the translators can read them on the screen.  So the English and the German translation do not have a one to one line correspondence.  I'm looking for ways around that.  I look at that and say, okay, they're both XML files.  If they are both current, both up to the same revision, the XML elements should correspond  to one to one.  Somebody who is into that whole XML thing might be able to write a program that does  that.  I would think it might also be an interesting problem for like a graduate student at a college  somewhere because it's non-trivial and it involves language processing potentially,  could be kind of neat.  But I suspect that there are a lot of places when they switch from an old translation method  to a new one, rather than spend the time writing a one-time conversion program, they'll just  manually jump in and translate it all by hand.  I suspect because it's a one-time use.  Once they've switched, they won't need that again.  Anyway, showing context is another thing that I talked about a little bit ago.  PO editors could show context and it's not a technical problem because in the PO file  with the English line, there is a comment that shows the line number it came out of  on the XML file.  So there's no reason the PO editor knows where to look for the text before and after  it and it could show it.  And along those lines, more PO editors.  Right now we have three that I know of, well, two and a half.  We have PO edit, we have G translator, which is a GNOME, I forget the graphic toolkit it  uses.  There's a KDE version whose name I forget.  Context?  Localize.  Localize.  With a K. Yes.  Thank you.  We have those three in ports.  But there are many more out there that do different things.  And the reason I think it would benefit us to have more PO editors is because I suspect  that one PO editor might support things like the CJK languages, Chinese, Japanese, and Korean.  It might work really well for those, while another one might be really great for, say,  the European languages.  And that choice of editors, particularly in the BSD community, where for 30 years we  haven't been able to decide between VI and Emacs, I think that having a choice of editors  like that would be good.  There are also online editors like Poodle and Weblate.  PCBSD has been using Poodle.  And translators connect to their website, and they run it all online.  There is no end user stuff to install.  But I'm not sure about FreeBSD running Poodle or Weblate on their own because I suspect  I'm concerned about limiting the choice of editors due to that.  There is one other PO editor called Vertal, V-I-R-T-A-A-L.  And that's a very highly regarded PO editor.  And we have a working port of it, except it doesn't let you edit strings or see the string  you're working on.  It's 99% there.  Mark Felder brought it the rest of the way.  I believe he was the last one who worked on it and did the most work.  And it just, it needs somebody who's into that.  I believe it's Python GTK stuff.  I think there's just something minor that needs to be done.  And I will have a link to that here coming up.  There are other PO editors.  There are PO editors written in Java that will run on various platforms.  There are even commercial web-based ones.  There's one called TransFX, where it's free to use for open source projects.  But I'm not sure we want to use that.  Anyway, those are challenges.  What about potential?  Where could we go with this?  And for me, the first thing I see is man pages.  I want full translations of all the man pages.  But ITS tool only works on XML files.  But remember Debian's PO4A?  4A is for anything.  They are supposed to support the man page format.  I have not tested that.  But wouldn't it be awesome if we got so many translators involved and it was so easy to  do that we had current up-to-date translations of our man pages also?  Which for me are almost more important than the documentation.  You really use the documentation a lot when you're first setting something up.  But the man pages are an online reference that you use all the time.  For me, for instance, is it capital R or lowercase r and CP?  Or does it matter?  And that's what a man page is for.  So for new non-English speaking people, I would really like to see the man pages translated.  And if it's easy to do with PO files, why not?  That would be a real selling point.  The other potential of this is new translators because it's easy, if we make it easy to translate,  we get new people working on docs.  And new people working on docs, especially if they speak both English and another language,  they tend to find errors in the English versions.  Because I've said this before, people who speak English as a second language tend to  speak it better than the natives because they care more is my experience.  The other night, we had Bjorn Heidt, high dotting, who's probably saying his name wrong.  He's a very active committer to the German project.  And he submitted a review where he found errors in the English version.  It's like, yeah, that's exactly, it was a wonderful thing to see.  And more contributors there, those contributors find more bugs and some of them turn into  committers.  And in turn, they help other people.  And in turn, because of that, you get new free BSD users and contributors in other areas.  And so we're back to that Metcalfe's law feedback loop.  So it's great.  What we learned doing this, the first one is absolutely the value of cross-pollinating  with other projects.  Last year at BSD CAN, Ryan Lordy of the GNOME project and I were sitting in one of these  conference rooms waiting for a talk to begin.  And I said, oh, I would ask questions like, well, what do you do about this?  And it wasn't a big, you know, it was how do you handle merging PO files, existing ones  with new ones?  Oh, well, you just do this.  Really?  That's it?  And that interaction on that is just don't underestimate the value of that.  We should not look at other projects as competitors.  We should look at them as rich gold mines of information.  We learned about commit by commit porting, which is really what the old translation process  is.  It's porting a document from English to another language.  And that came up because the guys who were porting X11, trying to keep the X11 system  up to date, posted on their mailing list and said, well, we're thinking about going file  by file or commit by commit.  What do people think?  And I said, I'm not so sure about the benefits and problems with file by file, but here is  what we've seen with commit by commit.  You might have something that's critically important, but you can't commit it because  there are all these other commits in line before it.  You have no control over the size of work you have to do.  And so a really large commit can stall the whole project while multiple people wait for  that commit to be ported.  And finally, with commit by commit, you might port 100 of them and find that number 101  rewrites the whole thing and wipes out all that work you've just done.  So we were able to contribute on that.  We found out about technical debt.  When you say to somebody, oh, we do this in XML to include chapters, and they pause for  a moment and say, I didn't think anybody did it that way anymore.  That's kind of a warning sign that you're getting behind.  And while we might be compatible with the standard, it could also mean that new stuff  might not support the old way we're doing it.  And to be safe, it's really best to keep up with the herd.  Along those lines, UTF-8.  Right now, our documentation is mostly not in UTF-8.  There are a few translated languages that use it.  Most do not.  English does not.  We need to just go ahead and convert.  That needs to be done.  XML techniques.  The modern technique for including an external file is Xinclude.  We don't use that.  That did not exist when the FreeBSD documentation first got going.  So we don't use it.  Now the way we do it is we define like a chapter as an entity and use that entity.  It works.  It's standards compliant, but I have the feeling that at some point in time there will be a  utility or tool we want to use that says, well, yes, we know that's part of the standard,  but we don't do things that way anymore.  So again, technical debt, you have to keep up with the herd or you'll find yourself in  trouble at some point.  And finally, I shouldn't say, and finally, acknowledgments.  This was not me.  This was a bunch of people who did a lot of work, Thomas Abtorp is here.  A bunch of people have pushed and worked on this over the years, and it's a hard thing  to solve because our translation method was part of a legacy system, and it's always hard  to try and bring something new into that.  And so I want to thank all these people for all the help they've done.  And there's probably about another 20 that should be on there that I just didn't remember  or worked on it before I even knew about it.  And then at that URL is this presentation in PDF form, is the article that this presentation  was based on in PDF form, is this presentation in open office form, I believe.  The virtual almost working port is here.  The shell script that gathers the information from the doc repository about new translations  is there, and the text file and a spreadsheet from that.  And there might be two or three things.  Get it now while it's available.  Any questions?  No?  Thank you.
improving_the_freebsd_Translation_Tools_GoPa2PdALPI_0_47.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_48_131.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_132_179.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_180_251.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_252_311.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_312_383.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_384_443.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_444_587.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_588_743.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_744_851.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_852_935.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_936_3731.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_3732_4427.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_4428_5051.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_5052_5435.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_5436_5543.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_5544_5771.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_5772_6095.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_6096_6323.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_6324_7223.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_7224_11279.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_11280_13331.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_13332_13883.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_13884_14843.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_14844_16595.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_16596_17747.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_17748_20759.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_20760_26123.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_26124_26483.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_26484_26963.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_26964_27083.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_27084_27179.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_27180_27527.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_27528_32159.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_32160_34631.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_34632_35411.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_35412_38195.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_38196_39935.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_39936_40715.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_40716_40799.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_40800_42827.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_42828_43607.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_43608_44375.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_44376_50975.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_50976_51227.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_51228_52643.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_52644_52919.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_52920_53195.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_53196_54167.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_54168_54443.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_54444_57959.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_57960_60479.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_60480_60875.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_60876_61223.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_61224_61991.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_61992_62471.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_62472_63575.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_63576_64547.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_64548_65819.png improving_the_freebsd_Translation_Tools_GoPa2PdALPI_65820_65836.png