<commit_before>.. -*- mode: rst; encoding: utf-8 -*-

=====================================
Internationalization and Localization
=====================================

Genshi provides comprehensive supporting infrastructure for internationalizing
and localizing templates. That includes functionality for extracting
localizable strings from templates, as well as a template filter and special
directives that can apply translations to templates as they get rendered.

This support is based on `gettext`_ message catalogs and the `gettext Python 
module`_. The extraction process can be used from the API level, or through
the front-ends implemented by the `Babel`_ project, for which Genshi provides
a plugin.

.. _`gettext`: http://www.gnu.org/software/gettext/
.. _`gettext python module`: http://docs.python.org/lib/module-gettext.html
.. _`babel`: http://babel.edgewall.org/


.. contents:: Contents
   :depth: 2
.. sectnum::


Basics
======

The simplest way to internationalize and translate templates would be to wrap
all localizable strings in a ``gettext()`` function call (which is often
aliased to ``_()`` for brevity). In that case, no extra template filter is
required.

.. code-block:: genshi

  <p>${_("Hello, world!")}</p>

However, this approach results in significant “character noise” in templates,
making them harder to read and preview.

The ``genshi.filters.Translator`` filter allows you to get rid of the 
explicit `gettext`_ function calls, so you can (often) just continue to write:

.. code-block:: genshi

  <p>Hello, world!</p>

This text will still be extracted and translated as if you had wrapped it in a
``_()`` call.

.. note:: For parameterized or pluralizable messages, you need to use the
          special `template directives`_ described below, or use the
          corresponding ``gettext`` function in embedded Python expressions.

You can control which tags should be ignored by this process; for example, it
doesn't really make sense to translate the content of the HTML 
``<script></script>`` element. Both ``<script>`` and ``<style>`` are excluded
by default.

Attribute values can also be automatically translated. The default is to 
consider the attributes ``abbr``, ``alt``, ``label``, ``prompt``, ``standby``,
``summary``, and ``title``, which is a list that makes sense for HTML
documents.  Of course, you can tell the translator to use a different set of
attribute names, or none at all.

----------------
Language Tagging
----------------

You can control automatic translation in your templates using the ``xml:lang``
attribute. If the value of that attribute is a literal string, the contents and
attributes of the element will be ignored:

.. code-block:: genshi

  <p xml:lang="en">Hello, world!</p>

On the other hand, if the value of the ``xml:lang`` attribute contains a Python
expression, the element contents and attributes are still considered for 
automatic translation:

.. code-block:: genshi

  <html xml:lang="$locale">
    ...
  </html>


.. _`template directives`:

Template Directives
===================

Sometimes localizable strings in templates may contain dynamic parameters, or
they may depend on the numeric value of some variable to choose a proper
plural form. Sometimes the strings contain embedded markup, such as tags for
emphasis or hyperlinks, and you don't want to rely on the people doing the
translations to know the syntax and escaping rules of HTML and XML.

In those cases the simple text extraction and translation process described
above is not sufficient. You could just use ``gettext`` API functions in
embedded Python expressions for parameters and pluralization, but that does
not help when messages contain embedded markup. Genshi provides special
template directives for internationalization that attempt to provide a
comprehensive solution for this problem space.

To enable these directives, you'll need to register them with the templates
they are used in. You can do this by adding them manually via the
``Template.add_directives(namespace, factory)`` (where ``namespace`` would be
“http://genshi.edgewall.org/i18n” and ``factory`` would be an instance of the
``Translator`` class). Or you can just call the ``Translator.setup(template)``
class method, which both registers the directives and adds the translation
filter.

After the directives have been registered with the template engine on the
Python side of your application, you need to declare the corresponding
directive namespace in all markup templates that use them. For example:

.. code-block:: genshi

  <html xmlns:py="http://genshi.edgewall.org/"
        xmlns:i18n="http://genshi.edgewall.org/i18n/">
    …
  </html>

These directives only make sense in the context of `markup templates`_. For
`text templates`_, you can just use the corresponding ``gettext`` API calls as needed.

.. note:: The internationalization directives are still somewhat experimental
          and have some known issues. However, the attribute language they
          implement should be stable and is not subject to change
          substantially in future versions.

.. _`markup templates`: xml-templates.html
.. _`text templates`: text-templates.html

--------
Messages
--------
<commit_msg>Pull up r1147 to trunk.<commit_after>.. -*- mode: rst; encoding: utf-8 -*-

=====================================
Internationalization and Localization
=====================================

Genshi provides comprehensive supporting infrastructure for internationalizing
and localizing templates. That includes functionality for extracting
localizable strings from templates, as well as a template filter and special
directives that can apply translations to templates as they get rendered.

This support is based on `gettext`_ message catalogs and the `gettext Python 
module`_. The extraction process can be used from the API level, or through
the front-ends implemented by the `Babel`_ project, for which Genshi provides
a plugin.

.. _`gettext`: http://www.gnu.org/software/gettext/
.. _`gettext python module`: http://docs.python.org/lib/module-gettext.html
.. _`babel`: http://babel.edgewall.org/


.. contents:: Contents
   :depth: 2
.. sectnum::


Basics
======

The simplest way to internationalize and translate templates would be to wrap
all localizable strings in a ``gettext()`` function call (which is often
aliased to ``_()`` for brevity). In that case, no extra template filter is
required.

.. code-block:: genshi

  <p>${_("Hello, world!")}</p>

However, this approach results in significant “character noise” in templates,
making them harder to read and preview.

The ``genshi.filters.Translator`` filter allows you to get rid of the 
explicit `gettext`_ function calls, so you can (often) just continue to write:

.. code-block:: genshi

  <p>Hello, world!</p>

This text will still be extracted and translated as if you had wrapped it in a
``_()`` call.

.. note:: For parameterized or pluralizable messages, you need to use the
          special `template directives`_ described below, or use the
          corresponding ``gettext`` function in embedded Python expressions.

You can control which tags should be ignored by this process; for example, it
doesn't really make sense to translate the content of the HTML 
``<script></script>`` element. Both ``<script>`` and ``<style>`` are excluded
by default.

Attribute values can also be automatically translated. The default is to 
consider the attributes ``abbr``, ``alt``, ``label``, ``prompt``, ``standby``,
``summary``, and ``title``, which is a list that makes sense for HTML
documents.  Of course, you can tell the translator to use a different set of
attribute names, or none at all.

----------------
Language Tagging
----------------

You can control automatic translation in your templates using the ``xml:lang``
attribute. If the value of that attribute is a literal string, the contents and
attributes of the element will be ignored:

.. code-block:: genshi

  <p xml:lang="en">Hello, world!</p>

On the other hand, if the value of the ``xml:lang`` attribute contains a Python
expression, the element contents and attributes are still considered for 
automatic translation:

.. code-block:: genshi

  <html xml:lang="$locale">
    ...
  </html>


.. _`template directives`:

Template Directives
===================

Sometimes localizable strings in templates may contain dynamic parameters, or
they may depend on the numeric value of some variable to choose a proper
plural form. Sometimes the strings contain embedded markup, such as tags for
emphasis or hyperlinks, and you don't want to rely on the people doing the
translations to know the syntax and escaping rules of HTML and XML.

In those cases the simple text extraction and translation process described
above is not sufficient. You could just use ``gettext`` API functions in
embedded Python expressions for parameters and pluralization, but that does
not help when messages contain embedded markup. Genshi provides special
template directives for internationalization that attempt to provide a
comprehensive solution for this problem space.

To enable these directives, you'll need to register them with the templates
they are used in. You can do this by adding them manually via the
``Template.add_directives(namespace, factory)`` (where ``namespace`` would be
“http://genshi.edgewall.org/i18n” and ``factory`` would be an instance of the
``Translator`` class). Or you can just call the ``Translator.setup(template)``
class method, which both registers the directives and adds the translation
filter.

After the directives have been registered with the template engine on the
Python side of your application, you need to declare the corresponding
directive namespace in all markup templates that use them. For example:

.. code-block:: genshi

  <html xmlns:py="http://genshi.edgewall.org/"
        xmlns:i18n="http://genshi.edgewall.org/i18n">
    …
  </html>

These directives only make sense in the context of `markup templates`_. For
`text templates`_, you can just use the corresponding ``gettext`` API calls as needed.

.. note:: The internationalization directives are still somewhat experimental
          and have some known issues. However, the attribute language they
          implement should be stable and is not subject to change
          substantially in future versions.

.. _`markup templates`: xml-templates.html
.. _`text templates`: text-templates.html

--------
Messages
--------

{'fraction_non_alphanumeric': 0.10563447421107827, 'fraction_numerical': 0.0019067594622938315, 'mean_word_length': 4.237144283574638, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 14}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1487860', 'n_tokens_mistral': 2703, 'n_tokens_neox': 2570, 'n_words': 1346}