<commit_before>---
title: Embeddings, Transformers and Transfer Learning
teaser: Using transformer embeddings like BERT in spaCy
menu:
  - ['Embedding Layers', 'embedding-layers']
  - ['Transformers', 'transformers']
  - ['Static Vectors', 'static-vectors']
  - ['Pretraining', 'pretraining']
next: /usage/training
---

<!-- TODO: intro, short explanation of embeddings/transformers, Tok2Vec and Transformer components, point user to processing pipelines docs for more general info that user should know first -->

If you're looking for details on using word vectors and semantic similarity,
check out the
[linguistic features docs](/usage/linguistic-features#vectors-similarity).

<Accordion title="What’s the difference between word vectors and language models?" id="vectors-vs-language-models">

The key difference between [word vectors](#word-vectors) and contextual language
models such as [transformers](#transformers) is that word vectors model
**lexical types**, rather than _tokens_. If you have a list of terms with no
context around them, a transformer model like BERT can't really help you. BERT
is designed to understand language **in context**, which isn't what you have. A
word vectors table will be a much better fit for your task. However, if you do
have words in context — whole sentences or paragraphs of running text — word
vectors will only provide a very rough approximation of what the text is about.

Word vectors are also very computationally efficient, as they map a word to a
vector with a single indexing operation. Word vectors are therefore useful as a
way to **improve the accuracy** of neural network models, especially models that
are small or have received little or no pretraining. In spaCy, word vector
tables are only used as **static features**. spaCy does not backpropagate
gradients to the pretrained word vectors table. The static vectors table is
usually used in combination with a smaller table of learned task-specific
embeddings.

</Accordion>

<Accordion title="When should I add word vectors to my model?">

Word vectors are not compatible with most [transformer models](#transformers),
but if you're training another type of NLP network, it's almost always worth
adding word vectors to your model. As well as improving your final accuracy,
word vectors often make experiments more consistent, as the accuracy you reach
will be less sensitive to how the network is randomly initialized. High variance
due to random chance can slow down your progress significantly, as you need to
run many experiments to filter the signal from the noise.

Word vector features need to be enabled prior to training, and the same word
vectors table will need to be available at runtime as well. You cannot add word
vector features once the model has already been trained, and you usually cannot
replace one word vectors table with another without causing a significant loss
of performance.

</Accordion>

## Shared embedding layers {#embedding-layers}

<!-- TODO: write -->

![Pipeline components using a shared embedding component vs. independent embedding layers](../images/tok2vec.svg)

| Shared                                                                                      | Independent                                                             |
| ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| ✅ **smaller:** models only need to include a single copy of the embeddings                 | ❌ **larger:** models need to include the embeddings for each component |
| ✅ **faster:**                                                                              | ❌ **slower:**                                                          |
| ❌ **less composable:** all components require the same embedding component in the pipeline | ✅ **modular:** components can be moved and swapped freely              |

![Pipeline components listening to shared embedding component](../images/tok2vec-listener.svg)

<!-- TODO: explain the listener concept, how it works etc. -->

## Using transformer models {#transformers}

Transformers are a family of neural network architectures that compute **dense,
context-sensitive representations** for the tokens in your documents. Downstream
models in your pipeline can then use these representations as input features to
**improve their predictions**. You can connect multiple components to a single
transformer model, with any or all of those components giving feedback to the
transformer to fine-tune it to your tasks. spaCy's transformer support
interoperates with [PyTorch](https://pytorch.org) and the
[HuggingFace `transformers`](https://huggingface.co/transformers/) library,
giving you access to thousands of pretrained models for your pipelines. There
are many [great guides](http://jalammar.github.io/illustrated-transformer/) to
transformer models, but for practical purposes, you can simply think of them as
a drop-in replacement that let you achieve **higher accuracy** in exchange for
**higher training and runtime costs**.

### Setup and installation {#transformers-installation}

> #### System requirements
>
> We recommend an NVIDIA **GPU** with at least **10GB of memory** in order to
> work with transformer models. Make sure your GPU drivers are up to date and
> you have **CUDA v9+** installed.

> The exact requirements will depend on the transformer model. Training a
> transformer-based model without a GPU will be too slow for most practical
> purposes.
>
> Provisioning a new machine will require about **5GB** of data to be
> downloaded: 3GB CUDA runtime, 800MB PyTorch, 400MB CuPy, 500MB weights, 200MB<commit_msg>Update embeddings-transformers.md<commit_after>---
title: Embeddings, Transformers and Transfer Learning
teaser: Using transformer embeddings like BERT in spaCy
menu:
  - ['Embedding Layers', 'embedding-layers']
  - ['Transformers', 'transformers']
  - ['Static Vectors', 'static-vectors']
  - ['Pretraining', 'pretraining']
next: /usage/training
---

spaCy supports a number of transfer and multi-task learning workflows that can
often help improve your pipeline's efficiency or accuracy. Transfer learning 
refers to techniques such as word vector tables and language model pretraining.
These techniques can be used to import knowledge from raw text into your
pipeline, so that your models are able to generalize better from your
annotated examples.

You can convert word vectors from popular tools like FastText and Gensim, or
you can load in any pretrained transformer model if you install our
`spacy-transformers` integration. You can also do your own language model pretraining
via the `spacy pretrain` command. You can even share your transformer or other
contextual embedding model across multiple components, which can make long
pipelines several times more efficient.

In order to use transfer learning, you'll need to have at least a few annotated
examples for all of the classes you're trying to predict. If you don't, you
could try using a "one-shot learning" approach using 
[vectors and similarity](/usage/linguistic-features#vectors-similarity).

<Accordion title="What’s the difference between word vectors and language models?" id="vectors-vs-language-models">

The key difference between [word vectors](#word-vectors) and contextual language
models such as [transformers](#transformers) is that word vectors model
**lexical types**, rather than _tokens_. If you have a list of terms with no
context around them, a transformer model like BERT can't really help you. BERT
is designed to understand language **in context**, which isn't what you have. A
word vectors table will be a much better fit for your task. However, if you do
have words in context — whole sentences or paragraphs of running text — word
vectors will only provide a very rough approximation of what the text is about.

Word vectors are also very computationally efficient, as they map a word to a
vector with a single indexing operation. Word vectors are therefore useful as a
way to **improve the accuracy** of neural network models, especially models that
are small or have received little or no pretraining. In spaCy, word vector
tables are only used as **static features**. spaCy does not backpropagate
gradients to the pretrained word vectors table. The static vectors table is
usually used in combination with a smaller table of learned task-specific
embeddings.

</Accordion>

<Accordion title="When should I add word vectors to my model?">

Word vectors are not compatible with most [transformer models](#transformers),
but if you're training another type of NLP network, it's almost always worth
adding word vectors to your model. As well as improving your final accuracy,
word vectors often make experiments more consistent, as the accuracy you reach
will be less sensitive to how the network is randomly initialized. High variance
due to random chance can slow down your progress significantly, as you need to
run many experiments to filter the signal from the noise.

Word vector features need to be enabled prior to training, and the same word
vectors table will need to be available at runtime as well. You cannot add word
vector features once the model has already been trained, and you usually cannot
replace one word vectors table with another without causing a significant loss
of performance.

</Accordion>

## Shared embedding layers {#embedding-layers}

You can share a single token-to-vector embedding model between multiple
components using the `Tok2Vec` component. Other components in
your pipeline can "connect" to the `Tok2Vec` component by including a _listener layer_
within their model. At the beginning of training, the `Tok2Vec` component will
grab a reference to the relevant listener layers in the rest of your pipeline.
Then, when the `Tok2Vec` component processes a batch of documents, it will pass
forward its predictions to the listeners, allowing the listeners to reuse the
predictions when they are eventually called. A similar mechanism is used to
pass gradients from the listeners back to the `Tok2Vec` model. The
`Transformer` component and `TransformerListener` layer do the same thing for
transformer models, making it easy to share a single transformer model across
your whole pipeline.

Training a single transformer or other embedding layer for use with multiple
components is termed _multi-task learning_. Multi-task learning is sometimes
less consistent, and the results are generally harder to reason about (as there's
more going on). You'll usually want to compare your accuracy against a single-task
approach to understand whether the weight-sharing is impacting your accuracy,
and whether you can address the problem by adjusting the hyper-parameters. We
are not currently aware of any foolproof recipe.

The main disadvantage of sharing weights between components is flexibility.
If your components are independent, you can train pipelines separately and
merge them together much more easily. Shared weights also make it more
difficult to resume training of only part of your pipeline. If you train only
part of your pipeline, you risk hurting the accuracy of the other components,
as you'll be changing the shared embedding layer those components are relying
on. <!-- TODO: Once rehearsal is tested, mention it here. -->


![Pipeline components using a shared embedding component vs. independent embedding layers](../images/tok2vec.svg)

| Shared                                                                                      | Independent                                                             |
| ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| ✅ **smaller:** models only need to include a single copy of the embeddings                 | ❌ **larger:** models need to include the embeddings for each component |
| ✅ **faster:** embed the documents once for your whole pipeline                             | ❌ **slower:** rerun the embedding for each component                   |
| ❌ **less composable:** all components require the same embedding component in the pipeline | ✅ **modular:** components can be moved and swapped freely              |
| ?? **accuracy:** weight sharing may increase or decrease accuracy, depending on your task and data, but usually the impact is small                                   |

![Pipeline components listening to shared embedding component](../images/tok2vec-listener.svg)


## Using transformer models {#transformers}

Transformers are a family of neural network architectures that compute **dense,
context-sensitive representations** for the tokens in your documents. Downstream
models in your pipeline can then use these representations as input features to
**improve their predictions**. You can connect multiple components to a single
transformer model, with any or all of those components giving feedback to the
transformer to fine-tune it to your tasks. spaCy's transformer support
interoperates with [PyTorch](https://pytorch.org) and the
[HuggingFace `transformers`](https://huggingface.co/transformers/) library,
giving you access to thousands of pretrained models for your pipelines. There
are many [great guides](http://jalammar.github.io/illustrated-transformer/) to
transformer models, but for practical purposes, you can simply think of them as
a drop-in replacement that let you achieve **higher accuracy** in exchange for
**higher training and runtime costs**.

### Setup and installation {#transformers-installation}

> #### System requirements
>
> We recommend an NVIDIA **GPU** with at least **10GB of memory** in order to
> work with transformer models. Make sure your GPU drivers are up to date and
> you have **CUDA v9+** installed.

> The exact requirements will depend on the transformer model. Training a
> transformer-based model without a GPU will be too slow for most practical
> purposes.
>
> Provisioning a new machine will require about **5GB** of data to be
> downloaded: 3GB CUDA runtime, 800MB PyTorch, 400MB CuPy, 500MB weights, 200MB
{'fraction_non_alphanumeric': 0.08102975311247099, 'fraction_numerical': 0.0030948864036013223, 'mean_word_length': 3.548304542546385, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 37, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5316559', 'n_tokens_mistral': 3286, 'n_tokens_neox': 3099, 'n_words': 1877}