<commit_before>//- 💫 DOCS > USAGE > PROCESSING PIPELINES > PIPELINES

p
    |  spaCy makes it very easy to create your own pipelines consisting of
    |  reusable components – this includes spaCy's default tagger,
    |  parser and entity regcognizer, but also your own custom processing
    |  functions. A pipeline component can be added to an already existing
    |  #[code nlp] object, specified when initialising a #[code Language] class,
    |  or defined within a
    |  #[+a("/usage/training#saving-loading") model package].

p
    |  When you load a model, spaCy first consults the model's
    |  #[+a("/usage/training#saving-loading") #[code meta.json]]. The
    |  meta typically includes the model details, the ID of a language class,
    |  and an optional list of pipeline components. spaCy then does the
    |  following:

+aside-code("meta.json (excerpt)", "json").
    {
        "name": "example_model",
        "lang": "en"
        "description": "Example model for spaCy",
        "pipeline": ["tagger", "parser"]
    }

+list("numbers")
    +item
        |  Load the #[strong language class and data] for the given ID via
        |  #[+api("top-level#util.get_lang_class") #[code get_lang_class]] and initialise
        |  it. The #[code Language] class contains the shared vocabulary,
        |  tokenization rules and the language-specific annotation scheme.
    +item
        |  Iterate over the #[strong pipeline names] and create each component
        |  using #[+api("language#create_pipe") #[code create_pipe]], which
        |  looks them up in #[code Language.factories].
    +item
        |  Add each pipeline component to the pipeline in order, using
        |  #[+api("language#add_pipe") #[code add_pipe]].
    +item
        |  Make the #[strong model data] available to the #[code Language] class
        |  by calling #[+api("language#from_disk") #[code from_disk]] with the
        |  path to the model data ditectory.

p
    |  So when you call this...

+code.
    nlp = spacy.load('en')

p
    | ... the model tells spaCy to use the language #[code "en"] and the
    |  pipeline #[code.u-break ["tagger", "parser", "ner"]]. spaCy will then
    |  initialise #[code spacy.lang.en.English], and create each pipeline
    |  component and add it to the processing pipeline. It'll then load in the
    |  model's data from its data ditectory and return the modified
    |  #[code Language] class for you to use as the #[code nlp] object.

p
    |  Fundamentally, a #[+a("/models") spaCy model] consists of three
    |  components: #[strong the weights], i.e. binary data loaded in from a<commit_msg>Fix typo ditectory -> directory<commit_after>//- 💫 DOCS > USAGE > PROCESSING PIPELINES > PIPELINES

p
    |  spaCy makes it very easy to create your own pipelines consisting of
    |  reusable components – this includes spaCy's default tagger,
    |  parser and entity regcognizer, but also your own custom processing
    |  functions. A pipeline component can be added to an already existing
    |  #[code nlp] object, specified when initialising a #[code Language] class,
    |  or defined within a
    |  #[+a("/usage/training#saving-loading") model package].

p
    |  When you load a model, spaCy first consults the model's
    |  #[+a("/usage/training#saving-loading") #[code meta.json]]. The
    |  meta typically includes the model details, the ID of a language class,
    |  and an optional list of pipeline components. spaCy then does the
    |  following:

+aside-code("meta.json (excerpt)", "json").
    {
        "name": "example_model",
        "lang": "en"
        "description": "Example model for spaCy",
        "pipeline": ["tagger", "parser"]
    }

+list("numbers")
    +item
        |  Load the #[strong language class and data] for the given ID via
        |  #[+api("top-level#util.get_lang_class") #[code get_lang_class]] and initialise
        |  it. The #[code Language] class contains the shared vocabulary,
        |  tokenization rules and the language-specific annotation scheme.
    +item
        |  Iterate over the #[strong pipeline names] and create each component
        |  using #[+api("language#create_pipe") #[code create_pipe]], which
        |  looks them up in #[code Language.factories].
    +item
        |  Add each pipeline component to the pipeline in order, using
        |  #[+api("language#add_pipe") #[code add_pipe]].
    +item
        |  Make the #[strong model data] available to the #[code Language] class
        |  by calling #[+api("language#from_disk") #[code from_disk]] with the
        |  path to the model data directory.

p
    |  So when you call this...

+code.
    nlp = spacy.load('en')

p
    | ... the model tells spaCy to use the language #[code "en"] and the
    |  pipeline #[code.u-break ["tagger", "parser", "ner"]]. spaCy will then
    |  initialise #[code spacy.lang.en.English], and create each pipeline
    |  component and add it to the processing pipeline. It'll then load in the
    |  model's data from its data directory and return the modified
    |  #[code Language] class for you to use as the #[code nlp] object.

p
    |  Fundamentally, a #[+a("/models") spaCy model] consists of three
    |  components: #[strong the weights], i.e. binary data loaded in from a
{'fraction_non_alphanumeric': 0.1101438304314913, 'fraction_numerical': 0.0, 'mean_word_length': 2.914814814814815, 'pattern_counts': {'":': 8, '<': 3, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1686842', 'n_tokens_mistral': 1578, 'n_tokens_neox': 1575, 'n_words': 618}