<commit_before>//- 💫 DOCS > USAGE > ADDING LANGUAGES

include ../../_includes/_mixins

p
    |  Adding full support for a language touches many different parts of the
    |  spaCy library. This guide explains how to fit everything together, and
    |  points you to the specific workflows for each component. Obviously,
    |  there are lots of ways you can organise your code when you implement
    |  your own #[+api("language") #[code Language]] class. This guide will
    |  focus on how it's done within spaCy. For full language support, we'll
    |  need to:

+list("numbers")
    +item
        |  Create a #[strong #[code Language] subclass] and
        |  #[a(href="#language-subclass") implement it].

    +item
        |  Define custom #[strong language data], like a
        |  #[a(href="#stop-words") stop list], #[a(href="#tag-map") tag map]
        |  and #[a(href="#tokenizer-exceptions") tokenizer exceptions].

    +item
        |  #[strong Build the vocabulary] including
        |  #[a(href="#word-frequencies") word frequencies],
        |  #[a(href="#brown-clusters") Brown clusters] and
        |  #[a(href="#word-vectors") word vectors].

    +item
        |  #[strong Set up] a #[a(href="#model-directory") model direcory] and #[strong train] the #[a(href="#train-tagger-parser") tagger and parser].

p
    |  For some languages, you may also want to develop a solution for
    |  lemmatization and morphological analysis.

+h(2, "language-subclass") Creating a #[code Language] subclass

p
    |  Language-specific code and resources should be organised into a
    |  subpackage of spaCy, named according to the language's
    |  #[+a("https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes") ISO code].
    |  For instance, code and resources specific to Spanish are placed into a
    |  directory #[code spacy/lang/es], which can be imported as
    |  #[code spacy.lang.es].

p
    |  To get started, you can use our
    |  #[+src(gh("spacy-dev-resources", "templates/new_language")) templates]
    |  for the most important files. Here's what the class template looks like:

+code("__init__.py (excerpt)").
    # import language-specific data
    from .stop_words import STOP_WORDS
    from .tokenizer_exceptions import TOKENIZER_EXCEPTIONS
    from .lex_attrs import LEX_ATTRS

    from ..tokenizer_exceptions import BASE_EXCEPTIONS
    from ...language import Language
    from ...attrs import LANG
    from ...util import update_exc

    class Xxxxx(Language):
        lang = 'xx' # language ISO code

        # override defaults
        class Defaults(Language.Defaults):
            lex_attr_getters = dict(Language.Defaults.lex_attr_getters)
            lex_attr_getters[LANG] = lambda text: 'xx' # language ISO code

            # optional: replace flags with custom functions, e.g. like_num()
            lex_attr_getters.update(LEX_ATTRS)

            # merge base exceptions and custom tokenizer exceptions
            tokenizer_exceptions = update_exc(BASE_EXCEPTIONS, TOKENIZER_EXCEPTIONS)
            stop_words = set(STOP_WORDS)

    # set default export – this allows the language class to be lazy-loaded
    __all__ = ['Xxxxx']

+aside("Why lazy-loading?")
    |  Some languages contain large volumes of custom data, like lemmatizer
    |  loopup tables, or complex regular expression that are expensive to
    |  compute. As of spaCy v2.0, #[code Language] classes are not imported on
    |  initialisation and are only loaded when you import them directly, or load
    |  a model that requires a language to be loaded. To lazy-load languages in
    |  your application, you can use the #[code util.load_lang_class()] helper
    |  function with the two-letter language code as its argument.

+h(2, "language-data") Adding language data

p
    |  Every language is full of exceptions and special cases, especially
    |  amongst the most common words. Some of these exceptions are shared
    |  between multiple languages, while others are entirely idiosyncratic.
    |  spaCy makes it easy to deal with these exceptions on a case-by-case
    |  basis, by defining simple rules and exceptions. The exceptions data is
    |  defined in Python the
    |  #[+src(gh("spacy-dev-resources", "templates/new_language")) language data],
    |  so that Python functions can be used to help you generalise and combine
    |  the data as you require.

+infobox("For languages with non-latin characters")
    |  In order for the tokenizer to split suffixes, prefixes and infixes, spaCy
    |  needs to know the language's character set. If the language you're adding
    |  uses non-latin characters, you might need to add the required character
    |  classes to the global
    |  #[+src(gh("spacy", "spacy/lang/punctuation.py")) punctuation.py].
    |  spaCy uses the #[+a("https://pypi.python.org/pypi/regex/") #[code regex] library]
    |  to keep this simple and readable. If the language requires very specific
    |  punctuation rules, you should consider overwriting the default regular
    |  expressions with your own in the language's #[code Defaults].

p
    |  Here's an overview of the individual components that can be included
    |  in the language data. For more details on them, see the sections below.

+table(["File name", "Variables", "Description"])
    +row
        +cell #[+src(gh()) stop_words.py]
        +cell #[code STOP_WORDS] (set)
        +cell
            |  List of most common words. Matching tokens will return #[code True]
            |  for #[code is_stop].

    +row
        +cell #[+src(gh()) tokenizer_exceptions.py]
        +cell #[code TOKENIZER_EXCEPTIONS] (dict), #[code TOKEN_MATCH] (regex)
        +cell
            |  Special-case rules for the tokenizer, for example, contractions
            |  and abbreviations containing punctuation.

    +row
        +cell #[+src(gh()) punctuation.py]
        +cell
            |  #[code TOKENIZER_PREFIXES], #[code TOKENIZER_SUFFIXES],
            |  #[code TOKENIZER_INFIXES] (dicts)
        +cell Regular expressions for splitting tokens, e.g. on punctuation.

    +row
        +cell #[+src(gh()) lex_attrs.py]
        +cell #[code LEX_ATTRS] (dict)
        +cell
            |  Functions for setting lexical attributes on tokens, e.g.
            |  #[code is_punct] or #[code like_num].

    +row
        +cell #[+src(gh()) tag_map.py]
        +cell #[code TAG_MAP] (dict)
        +cell
            |  Dictionary mapping strings in your tag set to
            |  #[+a("http://universaldependencies.org/u/pos/all.html") Universal Dependencies]
            |  tags.

    +row
        +cell #[+src(gh()) morph_rules.py]
        +cell #[code MORPH_RULES] (dict)
        +cell

    +row
        +cell #[+src(gh()) lemmatizer.py]
        +cell #[code LOOKUP] (dict)
        +cell
            |  Lookup-based lemmatization table. If more lemmatizer data is
            |  available, it should live in #[code /lemmatizer/lookup.py].

    +row
        +cell /lemmatizer
        +cell #[code LEMMA_RULES], #[code LEMMA_INDEX], #[code LEMMA_EXC] (dicts)
        +cell Lemmatization rules, keyed by part of speech.

+h(3, "stop-words") Stop words
<commit_msg>Update adding languages docs<commit_after>//- 💫 DOCS > USAGE > ADDING LANGUAGES

include ../../_includes/_mixins

p
    |  Adding full support for a language touches many different parts of the
    |  spaCy library. This guide explains how to fit everything together, and
    |  points you to the specific workflows for each component. Obviously,
    |  there are lots of ways you can organise your code when you implement
    |  your own #[+api("language") #[code Language]] class. This guide will
    |  focus on how it's done within spaCy. For full language support, we'll
    |  need to:

+list("numbers")
    +item Create a #[strong #[code Language] subclass].
    +item
        |  Define custom #[strong language data], like a
        |  #[a(href="#stop-words") stop list] and
        |  #[a(href="#tokenizer-exceptions") tokenizer exceptions].

    +item
        |  #[strong Build the vocabulary] including
        |  #[a(href="#word-frequencies") word frequencies],
        |  #[a(href="#brown-clusters") Brown clusters] and
        |  #[a(href="#word-vectors") word vectors].

    +item
        |  #[strong Set up] a #[a(href="#model-directory") model direcory] and
        |  #[strong train] the #[a(href="#train-tagger-parser") tagger and parser].

p
    |  For some languages, you may also want to develop a solution for
    |  lemmatization and morphological analysis.

+h(2, "language-subclass") Creating a #[code Language] subclass

p
    |  Language-specific code and resources should be organised into a
    |  subpackage of spaCy, named according to the language's
    |  #[+a("https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes") ISO code].
    |  For instance, code and resources specific to Spanish are placed into a
    |  directory #[code spacy/lang/es], which can be imported as
    |  #[code spacy.lang.es].

p
    |  To get started, you can use our
    |  #[+src(gh("spacy-dev-resources", "templates/new_language")) templates]
    |  for the most important files. Here's what the class template looks like:

+code("__init__.py (excerpt)").
    # import language-specific data
    from .stop_words import STOP_WORDS
    from .tokenizer_exceptions import TOKENIZER_EXCEPTIONS
    from .lex_attrs import LEX_ATTRS

    from ..tokenizer_exceptions import BASE_EXCEPTIONS
    from ...language import Language
    from ...attrs import LANG
    from ...util import update_exc

    class Xxxxx(Language):
        lang = 'xx' # language ISO code

        # override defaults
        class Defaults(Language.Defaults):
            lex_attr_getters = dict(Language.Defaults.lex_attr_getters)
            lex_attr_getters[LANG] = lambda text: 'xx' # language ISO code

            # optional: replace flags with custom functions, e.g. like_num()
            lex_attr_getters.update(LEX_ATTRS)

            # merge base exceptions and custom tokenizer exceptions
            tokenizer_exceptions = update_exc(BASE_EXCEPTIONS, TOKENIZER_EXCEPTIONS)
            stop_words = set(STOP_WORDS)

    # set default export – this allows the language class to be lazy-loaded
    __all__ = ['Xxxxx']

+aside("Why lazy-loading?")
    |  Some languages contain large volumes of custom data, like lemmatizer
    |  loopup tables, or complex regular expression that are expensive to
    |  compute. As of spaCy v2.0, #[code Language] classes are not imported on
    |  initialisation and are only loaded when you import them directly, or load
    |  a model that requires a language to be loaded. To lazy-load languages in
    |  your application, you can use the #[code util.load_lang_class()] helper
    |  function with the two-letter language code as its argument.

+h(2, "language-data") Adding language data

p
    |  Every language is full of exceptions and special cases, especially
    |  amongst the most common words. Some of these exceptions are shared
    |  between multiple languages, while others are entirely idiosyncratic.
    |  spaCy makes it easy to deal with these exceptions on a case-by-case
    |  basis, by defining simple rules and exceptions. The exceptions data is
    |  defined in Python the
    |  #[+src(gh("spacy-dev-resources", "templates/new_language")) language data],
    |  so that Python functions can be used to help you generalise and combine
    |  the data as you require.

p
    |  Here's an overview of the individual components that can be included
    |  in the language data. For more details on them, see the sections below.

+image
    include ../../assets/img/docs/language_data.svg

+table(["File name", "Variables", "Description"])
    +row
        +cell #[+src(gh()) stop_words.py]
        +cell #[code STOP_WORDS] (set)
        +cell
            |  List of most common words. Matching tokens will return #[code True]
            |  for #[code is_stop].

    +row
        +cell #[+src(gh()) tokenizer_exceptions.py]
        +cell #[code TOKENIZER_EXCEPTIONS] (dict), #[code TOKEN_MATCH] (regex)
        +cell
            |  Special-case rules for the tokenizer, for example, contractions
            |  and abbreviations containing punctuation.

    +row
        +cell #[+src(gh()) punctuation.py]
        +cell
            |  #[code TOKENIZER_PREFIXES], #[code TOKENIZER_SUFFIXES],
            |  #[code TOKENIZER_INFIXES] (dicts)
        +cell Regular expressions for splitting tokens, e.g. on punctuation.

    +row
        +cell #[+src(gh()) lex_attrs.py]
        +cell #[code LEX_ATTRS] (dict)
        +cell
            |  Functions for setting lexical attributes on tokens, e.g.
            |  #[code is_punct] or #[code like_num].

    +row
        +cell #[+src(gh()) tag_map.py]
        +cell #[code TAG_MAP] (dict)
        +cell
            |  Dictionary mapping strings in your tag set to
            |  #[+a("http://universaldependencies.org/u/pos/all.html") Universal Dependencies]
            |  tags.

    +row
        +cell #[+src(gh()) morph_rules.py]
        +cell #[code MORPH_RULES] (dict)
        +cell

    +row
        +cell #[+src(gh()) lemmatizer.py]
        +cell #[code LOOKUP] (dict)
        +cell
            |  Lookup-based lemmatization table. If more lemmatizer data is
            |  available, it should live in #[code /lemmatizer/lookup.py].

    +row
        +cell /lemmatizer
        +cell #[code LEMMA_RULES], #[code LEMMA_INDEX], #[code LEMMA_EXC] (dicts)
        +cell Lemmatization rules, keyed by part of speech.

+infobox("For languages with non-latin characters")
    |  In order for the tokenizer to split suffixes, prefixes and infixes, spaCy
    |  needs to know the language's character set. If the language you're adding
    |  uses non-latin characters, you might need to add the required character
    |  classes to the global
    |  #[+src(gh("spacy", "spacy/lang/punctuation.py")) punctuation.py].
    |  spaCy uses the #[+a("https://pypi.python.org/pypi/regex/") #[code regex] library]
    |  to keep this simple and readable. If the language requires very specific
    |  punctuation rules, you should consider overwriting the default regular
    |  expressions with your own in the language's #[code Defaults].

+h(3, "stop-words") Stop words

{'fraction_non_alphanumeric': 0.10030013261673763, 'fraction_numerical': 0.0012563690933203044, 'mean_word_length': 2.784469096671949, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6947571', 'n_tokens_mistral': 4308, 'n_tokens_neox': 4234, 'n_words': 1517}