<commit_before>.. This Source Code Form is subject to the terms of the Mozilla Public
.. License, v. 2.0. If a copy of the MPL was not distributed with this
.. file, You can obtain one at http://mozilla.org/MPL/2.0/.

.. _l10n:

============
Localization
============

The site is fully localizable. Localization files are not shipped with the code
distribution, but are available in separate GitHub repositories. The proper repos
can be cloned and kept up-to-date using the ``l10n_update`` management command:

.. code-block:: console

    $ ./manage.py l10n_update

If you don't already have a ``locale`` directory, this command will clone the
git repo containing the .lang translation files (either the dev or prod files
depending on your ``DEV`` setting). If the folder is already present, it will
update the repository to the latest version. It do the same thing for the
repository for the .ftl translation files in ``git-repos/www-l10n`` directory.

Fluent
======

Bedrock's Localization (l10n) system is based on `Project Fluent`_. This is a
departure from the legacy system (see below) that relied on a gettext work flow
of string extraction from template and code files, in that it relies on developers
directly editing the default language (English in our case) Fluent files and using
the string IDs created there in their templates and views.

The default files for the Fluent system live in the ``l10n`` directory in the root
of the bedrock project. This directory houses directories for each locale the developers
directly implement (mostly simplified English "en", and "en-US"). The simplified English
files are the default fallback for every string ID on the site and should be strings that
are plain and easy to understand English, as free from colloquialisms as possible. The
translators are able to easily understand the meaning of the string, and can then add their
own local flair to the ideas.

.. note::

    We have some :ref:`fluent tools <fluent>` to aid in the transition from the legacy system.

.. _Project Fluent: https://projectfluent.org/

.ftl files
----------

When adding translatable strings to the site you start by putting them all into an .ftl
file in the ``l10n/en/`` directory with a path that matches or is somehow meaningful
for the expected location of the template or view in which they'll be used. For example,
strings for the ``mozorg/mission.html`` template would go into the ``l10n/en/mozorg/mission.ftl``
file. Locales are activated for a particular .ftl file, not template or URL, so you should use
a unique file for most URLs, unless they're meant to be translated and activated for new locales
simultaneously.

You can have shared .ftl files that you can load into any template render, but only the first
.ftl file in the list of the ones for a page render will determine whether the page is active
for a locale.

Activation of a locale happens automatically once certain rules are met. A developer can mark
some string IDs as being "Required", which means that the file won't be activated for a locale
until that locale has translated all of those required strings. The other rule is a percentage
completion rule: a certain percentage (configurable) of the strings IDs in the "en" file must
be translated in the file for a locale before it will be marked as active. We'll get into how
exactly this works later.

Translating with .ftl files
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The `Fluent file syntax`_ is well document on the Fluent Project's site. We use "double hash" or
"group" comments to indicate strings required for activation. A group comment only ends when
another group comment starts however, so you should either group your required strings at the
bottom of a file, or also have a "not required" group comment. Here's an example:

.. code-block:: fluent

    ### File for example.html

    ## Required
    example-page-title = The Page Title
    example-page-desc = This page is a test.

    ##
    example-footer = This string isn't as important


Any group comment (a comment that starts with "##") that starts with "Required" (case does not
matter) will start a required strings block, and any other group comment will end it.

Once you have your strings in your .ftl file you can place them in your template. We'll use the
above .ftl file for a simple Jinja template example:

.. code-block:: jinja

    <!doctype html>
    <html>
    <head>
        <title>{{ ftl('example-page-title') }}</title>
    </head>
    <body>
        <h1>{{ ftl('example-page-title') }}</h1>
        <p>{{ ftl('example-page-desc') }}</p>
        <footer>
            <p>{{ ftl('example-footer') }}</p>
        </footer>
    </body>
    </html>

.. _Fluent file syntax: https://projectfluent.org/fluent/guide/

FTL String IDs
~~~~~~~~~~~~~~

Our convention for string ID creation is the following:

1. String IDs should be all lower-case alphanumeric characters.
2. Words should be separated with hyphens.
3. IDs should be prefixed with the name of the template file (e.g. ``firefox-new-skyline`` for ``firefox-new-skyline.html``)
4. If you need to create a new string for the same place on a page and to transition to it as it is translated, you can
   add a version suffix to the string ID: e.g. ``firefox-new-skyline-main-page-title-v2``.
5. The ID should be as descriptive as possible to make sense to the developer, but could be anything as long as it adheres
   to the rules above.

The ``ftl`` helper function
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``ftl()`` function takes a string ID and returns the string in the current language,
or simplified english if the string isn't translated. If you'd like to use a different
string ID in the case that the primary one isn't translated you can specify that like this:

.. code-block:: python

    ftl('primary-string-id', fallback='fallback-string-id')

When a fallback is specified it will be used only if the primary isn't translated in the current
locale. English locales (e.g. en-US, en-GB) will never use the fallback and will print the simplified
english version of the primary string if not overridden in the more specific locale.

You can also pass in replacement variables into the ``ftl()`` function for use with `fluent variables`_.
If you had a variable in your fluent file like this:

.. code-block:: fluent

    welcome = Welcome, { $user }!

You could use that in a template like this:

.. code-block:: jinja<commit_msg>Add Fluent documentation on using brand names<commit_after>.. This Source Code Form is subject to the terms of the Mozilla Public
.. License, v. 2.0. If a copy of the MPL was not distributed with this
.. file, You can obtain one at http://mozilla.org/MPL/2.0/.

.. _l10n:

============
Localization
============

The site is fully localizable. Localization files are not shipped with the code
distribution, but are available in separate GitHub repositories. The proper repos
can be cloned and kept up-to-date using the ``l10n_update`` management command:

.. code-block:: console

    $ ./manage.py l10n_update

If you don't already have a ``locale`` directory, this command will clone the
git repo containing the .lang translation files (either the dev or prod files
depending on your ``DEV`` setting). If the folder is already present, it will
update the repository to the latest version. It do the same thing for the
repository for the .ftl translation files in ``git-repos/www-l10n`` directory.

Fluent
======

Bedrock's Localization (l10n) system is based on `Project Fluent`_. This is a
departure from the legacy system (see below) that relied on a gettext work flow
of string extraction from template and code files, in that it relies on developers
directly editing the default language (English in our case) Fluent files and using
the string IDs created there in their templates and views.

The default files for the Fluent system live in the ``l10n`` directory in the root
of the bedrock project. This directory houses directories for each locale the developers
directly implement (mostly simplified English "en", and "en-US"). The simplified English
files are the default fallback for every string ID on the site and should be strings that
are plain and easy to understand English, as free from colloquialisms as possible. The
translators are able to easily understand the meaning of the string, and can then add their
own local flair to the ideas.

.. note::

    We have some :ref:`fluent tools <fluent>` to aid in the transition from the legacy system.

.. _Project Fluent: https://projectfluent.org/

.ftl files
----------

When adding translatable strings to the site you start by putting them all into an .ftl
file in the ``l10n/en/`` directory with a path that matches or is somehow meaningful
for the expected location of the template or view in which they'll be used. For example,
strings for the ``mozorg/mission.html`` template would go into the ``l10n/en/mozorg/mission.ftl``
file. Locales are activated for a particular .ftl file, not template or URL, so you should use
a unique file for most URLs, unless they're meant to be translated and activated for new locales
simultaneously.

You can have shared .ftl files that you can load into any template render, but only the first
.ftl file in the list of the ones for a page render will determine whether the page is active
for a locale.

Activation of a locale happens automatically once certain rules are met. A developer can mark
some string IDs as being "Required", which means that the file won't be activated for a locale
until that locale has translated all of those required strings. The other rule is a percentage
completion rule: a certain percentage (configurable) of the strings IDs in the "en" file must
be translated in the file for a locale before it will be marked as active. We'll get into how
exactly this works later.

Translating with .ftl files
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The `Fluent file syntax`_ is well document on the Fluent Project's site. We use "double hash" or
"group" comments to indicate strings required for activation. A group comment only ends when
another group comment starts however, so you should either group your required strings at the
bottom of a file, or also have a "not required" group comment. Here's an example:

.. code-block:: fluent

    ### File for example.html

    ## Required
    example-page-title = The Page Title
    example-page-desc = This page is a test.

    ##
    example-footer = This string isn't as important


Any group comment (a comment that starts with "##") that starts with "Required" (case does not
matter) will start a required strings block, and any other group comment will end it.

Once you have your strings in your .ftl file you can place them in your template. We'll use the
above .ftl file for a simple Jinja template example:

.. code-block:: jinja

    <!doctype html>
    <html>
    <head>
        <title>{{ ftl('example-page-title') }}</title>
    </head>
    <body>
        <h1>{{ ftl('example-page-title') }}</h1>
        <p>{{ ftl('example-page-desc') }}</p>
        <footer>
            <p>{{ ftl('example-footer') }}</p>
        </footer>
    </body>
    </html>

.. _Fluent file syntax: https://projectfluent.org/fluent/guide/

FTL String IDs
~~~~~~~~~~~~~~

Our convention for string ID creation is the following:

1. String IDs should be all lower-case alphanumeric characters.
2. Words should be separated with hyphens.
3. IDs should be prefixed with the name of the template file (e.g. ``firefox-new-skyline`` for ``firefox-new-skyline.html``)
4. If you need to create a new string for the same place on a page and to transition to it as it is translated, you can
   add a version suffix to the string ID: e.g. ``firefox-new-skyline-main-page-title-v2``.
5. The ID should be as descriptive as possible to make sense to the developer, but could be anything as long as it adheres
   to the rules above.

Using brand names
~~~~~~~~~~~~~~~~~

Common brand names are stored in a global `brands.ftl`_ file, in the form of `terms`_. Terms are useful for
keeping brand names separated from the rest of the translations, so that they can be managed in a consistent
way across all translated files, and also updated easily in a global context. In general the brand names in
this file remain in English and should not be translated, however locales still have the choice and control
to make adjustments should it suit their particular language.

.. code-block:: text

    -brand-name = Firefox

    about = About { -brand-name }.
    update-successful = { -brand-name } has been updated.

.. important::

    When adding a new term to ``brands.ftl``, the new term should also be manually added to the
    `mozilla-l10n/www-l10n`_ repo for *all locales*. The reason for this is that if a term does not exist
    for a particular locale, then it does not fall back to English like a regular string does. Instead,
    the term variable name is displayed on the page.

.. _brands.ftl: https://github.com/mozilla/bedrock/blob/master/l10n/en/brands.ftl
.. _terms: https://projectfluent.org/fluent/guide/terms.html
.. _mozilla-l10n/www-l10n: https://github.com/mozilla-l10n/www-l10n

The ``ftl`` helper function
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``ftl()`` function takes a string ID and returns the string in the current language,
or simplified english if the string isn't translated. If you'd like to use a different
string ID in the case that the primary one isn't translated you can specify that like this:

.. code-block:: python

    ftl('primary-string-id', fallback='fallback-string-id')

When a fallback is specified it will be used only if the primary isn't translated in the current
locale. English locales (e.g. en-US, en-GB) will never use the fallback and will print the simplified
english version of the primary string if not overridden in the more specific locale.

You can also pass in replacement variables into the ``ftl()`` function for use with `fluent variables`_.
If you had a variable in your fluent file like this:

.. code-block:: fluent

    welcome = Welcome, { $user }!

You could use that in a template like this:

.. code-block:: jinja
{'fraction_non_alphanumeric': 0.08497240696193575, 'fraction_numerical': 0.004952596575633225, 'mean_word_length': 3.802922188243289, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 7, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1490971', 'n_tokens_mistral': 3885, 'n_tokens_neox': 3693, 'n_words': 2092}