<commit_before><?xml version="1.0" standalone="no"?>
<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V1.1//EN"
    "http://cvs.apache.org/viewcvs.cgi/*checkout*/xml-forrest/src/resources/schema/dtd/document-v11.dtd">
<document>
  <header>
    <title>FOP: Hyphenation</title>
  </header>
  <body>
    <section id="std">
      <title>Standard Hyphenation Support</title>
      <p>The following table summarizes FOP's standard hyphenation support.
Please note that the "view" links reflect current CVS, and may be different than the contents of released code. See <link href="#patterns">Hyphenation Patterns</link> for a brief explanation of the contents of these files.</p>
      <table>
        <tr>
          <th>language_COUNTRY code</th>
          <th>Description</th>
          <th>View Patterns (maintenance branch CVS)</th>
        </tr>
        <tr>
          <td>en</td>
          <td>English</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/en.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>es</td>
          <td>Spanish</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/es.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>fi</td>
          <td>Finnish</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/fi.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>hu</td>
          <td>Hungarian</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/hu.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>it</td>
          <td>Italian</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/it.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>pl</td>
          <td>Polish</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/pl.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>pt</td>
          <td>Portuguese</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/pt.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>ru</td>
          <td>Russian</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/ru.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
      </table>
    </section>
    <section id="custom">
    <title>Custom Hyphenation Support</title>
    <section id="custom-intro">
      <title>Introduction</title>
      <p>FOP uses an XML-based TeX-like hyphenation pattern scheme.
However, because of <link href="#license-issues">licensing issues</link>, there are currently some significant holes in FOP's hyphenation support.
The information in this section is intended to help you work around these limitations, if possible, add support for other languages, or enhance FOP's support of current languages.</p>
      <note>If you have access to hyphenation patterns that are licensed in an Apache-compatible way, or if you have made improvements to an existing FOP hyphenation pattern, or if you have created one from scratch, please consider contributing these to FOP so that they can benefit other FOP users as well. Please inquire on the <link href="maillist.html#fop-user">FOP User mailing list</link>.</note>
    </section>
    <section id="custom-license-issues">
      <title>License Issues</title>
      <p>Many of the hyphenation files distributed with TeX and its offspring are licenced under the <fork href="http://www.latex-project.org/lppl.html">LaTeX Project Public License (LPPL)</fork>, which prevents them from being distributed with Apache software.
Although Apache FOP cannot redistribute hyphenation pattern files that do not conform with its license scheme, that does not necessarily prevent users from using such hyphenation patterns with FOP.
However, it does place on the user the responsibility for determining whether the user can rightly use such hyphenation patterns under the hyphenation pattern license.</p>
      <warning>The user is responsible to settle license issues for hyphenation pattern files that are obtained from non-Apache sources.</warning>
    </section>
    <section id="custom-sources">
      <title>Sources of Custom Hyphenation Pattern Files</title>
      <p>The most important source of hyphenation pattern files is the <fork href="http://www.ctan.org/tex-archive/language/hyphenation/">CTAN TeX Archive</fork>.</p>
    </section>
    <section id="custom-install">
      <title>Installing Custom Hyphenation Patterns</title>
      <p>To install custom a custom hyphenation pattern for use with FOP:</p>
      <ol>
        <li>Convert the TeX hyphenation pattern file to the FOP format. The FOP format is an xml file conforming to the DTD found at <code>{fop-dir}/src/hyph/hyphenation.dtd</code>.</li>
        <li>Name this new file following this schema: <code>languageCode_countryCode.xml</code>.
The country code is optional, and should be used only if needed. For example:
          <ul>
            <li><code>en_US.xml</code> would be the file name for an American English hyphenation pattern.</li>
            <li><code>it.xml</code> would be the file name for an Italian hyphenation pattern.</li>
          </ul>
The language and country codes must match the XSL-FO input, which follows <link href="http://www.ics.uci.edu/pub/ietf/http/related/iso639.txt">ISO 639</link> (languages) and
<link href="http://www.ics.uci.edu/pub/ietf/http/related/iso3166.txt">ISO 3166</link> (countries).
NOTE: The ISO 639/ISO 3166 convention is that language names are written in lower case, while country codes are written in upper case.</li>
        <li>There are two ways to make the FOP-compatible hyphenation pattern file accessible to FOP:
          <ul>
            <li>Place the FOP-compatible hyphenation pattern file into the directory {fop-dir}/src/hyph and rebuild FOP. The file will be picked up and added to fop.jar.</li>
            <li>Put the file into a directory of your choice and configure FOP to look for custom patterns in this directory, by setting the <link href="configuration.html#hyphenation-dir">&lt;hyphenation-dir&gt; configuration option</link>.</li>
          </ul>
        </li>
      </ol>
    </section>
  </section>
  <section id="patterns">
    <title>Hyphenation Patterns</title>
    <p>If you would like to build your own hyphenation pattern files, or modify existing ones, this section will help you understand how to do so. Even when creating a pattern file from scratch, it may be beneficial to start with an existing file and modify it. See <link href="#std">Standard Hyphenation Support</link> or the source distribution (src/hyph) for examples. Here is a brief explanation of the contents of FOP's hyphenation patterns:</p>
    <ul>
      <li>The root of the pattern file is the &lt;hyphenation-info> element.</li>
      <li>&lt;hyphen-char> is self-explanatory: its attribute "value" contains the default character to be used for hyphenating this language. For English, this is the hyphen "-".</li>
      <li>&lt;hyphen-min> contains two attributes:
        <ul><commit_msg>add caveat about accuracy of the patterns documentation<commit_after><?xml version="1.0" standalone="no"?>
<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V1.1//EN"
    "http://cvs.apache.org/viewcvs.cgi/*checkout*/xml-forrest/src/resources/schema/dtd/document-v11.dtd">
<document>
  <header>
    <title>FOP: Hyphenation</title>
  </header>
  <body>
    <section id="std">
      <title>Standard Hyphenation Support</title>
      <p>The following table summarizes FOP's standard hyphenation support.
Please note that the "view" links reflect current CVS, and may be different than the contents of released code. See <link href="#patterns">Hyphenation Patterns</link> for a brief explanation of the contents of these files.</p>
      <table>
        <tr>
          <th>language_COUNTRY code</th>
          <th>Description</th>
          <th>View Patterns (maintenance branch CVS)</th>
        </tr>
        <tr>
          <td>en</td>
          <td>English</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/en.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>es</td>
          <td>Spanish</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/es.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>fi</td>
          <td>Finnish</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/fi.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>hu</td>
          <td>Hungarian</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/hu.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>it</td>
          <td>Italian</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/it.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>pl</td>
          <td>Polish</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/pl.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>pt</td>
          <td>Portuguese</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/pt.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
        <tr>
          <td>ru</td>
          <td>Russian</td>
          <td><jump href="http://cvs.apache.org/viewcvs.cgi/xml-fop/src/hyph/ru.xml?rev=1.1.2&amp;content-type=text/vnd.viewcvs-markup">view</jump></td>
        </tr>
      </table>
    </section>
    <section id="custom">
    <title>Custom Hyphenation Support</title>
    <section id="custom-intro">
      <title>Introduction</title>
      <p>FOP uses an XML-based TeX-like hyphenation pattern scheme.
However, because of <link href="#license-issues">licensing issues</link>, there are currently some significant holes in FOP's hyphenation support.
The information in this section is intended to help you work around these limitations, if possible, add support for other languages, or enhance FOP's support of current languages.</p>
      <note>If you have access to hyphenation patterns that are licensed in an Apache-compatible way, or if you have made improvements to an existing FOP hyphenation pattern, or if you have created one from scratch, please consider contributing these to FOP so that they can benefit other FOP users as well. Please inquire on the <link href="maillist.html#fop-user">FOP User mailing list</link>.</note>
    </section>
    <section id="custom-license-issues">
      <title>License Issues</title>
      <p>Many of the hyphenation files distributed with TeX and its offspring are licenced under the <fork href="http://www.latex-project.org/lppl.html">LaTeX Project Public License (LPPL)</fork>, which prevents them from being distributed with Apache software.
Although Apache FOP cannot redistribute hyphenation pattern files that do not conform with its license scheme, that does not necessarily prevent users from using such hyphenation patterns with FOP.
However, it does place on the user the responsibility for determining whether the user can rightly use such hyphenation patterns under the hyphenation pattern license.</p>
      <warning>The user is responsible to settle license issues for hyphenation pattern files that are obtained from non-Apache sources.</warning>
    </section>
    <section id="custom-sources">
      <title>Sources of Custom Hyphenation Pattern Files</title>
      <p>The most important source of hyphenation pattern files is the <fork href="http://www.ctan.org/tex-archive/language/hyphenation/">CTAN TeX Archive</fork>.</p>
    </section>
    <section id="custom-install">
      <title>Installing Custom Hyphenation Patterns</title>
      <p>To install custom a custom hyphenation pattern for use with FOP:</p>
      <ol>
        <li>Convert the TeX hyphenation pattern file to the FOP format. The FOP format is an xml file conforming to the DTD found at <code>{fop-dir}/src/hyph/hyphenation.dtd</code>.</li>
        <li>Name this new file following this schema: <code>languageCode_countryCode.xml</code>.
The country code is optional, and should be used only if needed. For example:
          <ul>
            <li><code>en_US.xml</code> would be the file name for an American English hyphenation pattern.</li>
            <li><code>it.xml</code> would be the file name for an Italian hyphenation pattern.</li>
          </ul>
The language and country codes must match the XSL-FO input, which follows <link href="http://www.ics.uci.edu/pub/ietf/http/related/iso639.txt">ISO 639</link> (languages) and
<link href="http://www.ics.uci.edu/pub/ietf/http/related/iso3166.txt">ISO 3166</link> (countries).
NOTE: The ISO 639/ISO 3166 convention is that language names are written in lower case, while country codes are written in upper case.</li>
        <li>There are two ways to make the FOP-compatible hyphenation pattern file accessible to FOP:
          <ul>
            <li>Place the FOP-compatible hyphenation pattern file into the directory {fop-dir}/src/hyph and rebuild FOP. The file will be picked up and added to fop.jar.</li>
            <li>Put the file into a directory of your choice and configure FOP to look for custom patterns in this directory, by setting the <link href="configuration.html#hyphenation-dir">&lt;hyphenation-dir&gt; configuration option</link>.</li>
          </ul>
        </li>
      </ol>
    </section>
  </section>
  <section id="patterns">
    <title>Hyphenation Patterns</title>
    <p>If you would like to build your own hyphenation pattern files, or modify existing ones, this section will help you understand how to do so. Even when creating a pattern file from scratch, it may be beneficial to start with an existing file and modify it. See <link href="#std">Standard Hyphenation Support</link> or the source distribution (src/hyph) for examples. Here is a brief explanation of the contents of FOP's hyphenation patterns:</p>
    <warning>The remaining content of this section should be considered "draft" quality. It was drafted from theoretical literature, and has not been tested against actual FOP behavior. It may contain errors or omissions. Do not rely on these instructions without testing everything stated here. If you use these instructions, please provide feedback on the <link href="maillist.html#fop-user">FOP User mailing list</link>, either confirming their accuracy, or raising specific problems that we can address.</warning>
    <ul>
      <li>The root of the pattern file is the &lt;hyphenation-info> element.</li>
      <li>&lt;hyphen-char> is self-explanatory: its attribute "value" contains the default character to be used for hyphenating this language. For English, this is the hyphen "-".</li>
      <li>&lt;hyphen-min> contains two attributes:
        <ul>
{'fraction_non_alphanumeric': 0.12625289128758674, 'fraction_numerical': 0.0065535851966075555, 'mean_word_length': 3.239989103786434, 'pattern_counts': {'":': 0, '<': 395, '<?xml version=': 2, '>': 401, 'https://': 0, 'lorem ipsum': 0, 'www.': 8, 'xml': 44}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3825747', 'n_tokens_mistral': 4944, 'n_tokens_neox': 4741, 'n_words': 1458}