<commit_before>---
title: Transformers
teaser: Using transformer models like BERT in spaCy
menu:
  - ['Installation', 'install']
  - ['Runtime Usage', 'runtime']
  - ['Training Usage', 'training']
next: /usage/training
---

## Installation {#install hidden="true"}

spaCy v3.0 lets you use almost **any statistical model** to power your pipeline.
You can use models implemented in a variety of
[frameworks](https://thinc.ai/docs/usage-frameworks), including TensorFlow,
PyTorch and MXNet. To keep things sane, spaCy expects models from these
frameworks to be wrapped with a common interface, using our machine learning
library [Thinc](https://thinc.ai). A transformer model is just a statistical
model, so the
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) package
actually has very little work to do: it just has to provide a few functions that
do the required plumbing. It also provides a pipeline component,
[`Transformer`](/api/transformer), that lets you do multi-task learning and lets
you save the transformer outputs for later use.

To use transformers with spaCy, you need the
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) package
installed. It takes care of all the setup behind the scenes, and makes sure the
transformer pipeline component is available to spaCy.

```bash
$ pip install spacy-transformers
```

## Runtime usage {#runtime}

Transformer models can be used as **drop-in replacements** for other types of
neural networks, so your spaCy pipeline can include them in a way that's
completely invisible to the user. Users will download, load and use the model in
the standard way, like any other spaCy pipeline. Instead of using the
transformers as subnetworks directly, you can also use them via the
[`Transformer`](/api/transformer) pipeline component.

![The processing pipeline with the transformer component](../images/pipeline_transformer.svg)

The `Transformer` component sets the
[`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute,
which lets you access the transformers outputs at runtime.

```bash
$ python -m spacy download en_core_trf_lg
```

```python
### Example
import spacy

nlp = spacy.load("en_core_trf_lg")
for doc in nlp.pipe(["some text", "some other text"]):
    tokvecs = doc._.trf_data.tensors[-1]
```

You can also customize how the [`Transformer`](/api/transformer) component sets
annotations onto the [`Doc`](/api/doc), by customizing the `annotation_setter`.
This callback will be called with the raw input and output data for the whole
batch, along with the batch of `Doc` objects, allowing you to implement whatever
you need. The annotation setter is called with a batch of [`Doc`](/api/doc)
objects and a [`FullTransformerBatch`](/api/transformer#fulltransformerbatch)
containing the transformers data for the batch.

```python
def custom_annotation_setter(docs, trf_data):
    # TODO:
    ...

nlp = spacy.load("en_core_trf_lg")
nlp.get_pipe("transformer").annotation_setter = custom_annotation_setter
doc = nlp("This is a text")
print()  # TODO:
```
<commit_msg>Update transformer docs intro. Also write system requirements<commit_after>---
title: Transformers
teaser: Using transformer models like BERT in spaCy
menu:
  - ['Installation', 'install']
  - ['Runtime Usage', 'runtime']
  - ['Training Usage', 'training']
next: /usage/training
---

## Installation {#install hidden="true"}

Transformers are a family of neural network architectures that compute dense,
context-sensitive representations for the tokens in your documents. Downstream
models in your pipeline can then use these representations as input features to
improve their predictions. You can connect multiple components to a single
transformer model, with any or all of those components giving feedback to the
transformer to fine-tune it to your tasks. spaCy's transformer support
interoperates with PyTorch and the [Huggingface transformers](https://huggingface.co/transformers/)
library, giving you access to thousands of pretrained models for your pipelines.
There are many [great guides](http://jalammar.github.io/illustrated-transformer/)
to transformer models, but for practical purposes, you can simply think of them
as a drop-in replacement that let you achieve higher accuracy in exchange for
higher training and runtime costs.

## System requirements

We recommend an NVIDIA GPU with at least 10GB of memory in order to work with
transformer models. The exact requirements will depend on the transformer you
model you choose and whether you're training the pipeline or simply running it.
Training a transformer-based model without a GPU will be too slow for most
practical purposes. A GPU will usually also achieve better
price-for-performance when processing large batches of documents. The only
context where a GPU might not be worthwhile is if you're serving the model in
a context where documents are received individually, rather than in batches. In
this context, CPU may be more cost effective.

You'll also need to make sure your GPU drivers are up-to-date and v9+ of the
CUDA runtime is installed. Unfortunately, there's little we can do to help with
this part: the steps will vary depending on your device, operating system and
the version of CUDA you're targetting (you'll want to use one that's well
supported by cupy, PyTorch and Tensorflow).

Once you have CUDA installed, you'll need to install two pip packages, `cupy`
and `spacy-transformers`. The `cupy` library is just like `numpy`, but for GPU.
The best way to install it is to choose a wheel that matches the version of CUDA
you're using. For instance, if you're using CUDA 10.2, you would run
`pip install cupy-cuda102`. Finally, if you've installed CUDA in a non-standard
location, you'll need to set the `CUDA_PATH` environment variable to the base
of your CUDA installation. See the cupy documentation for more details.
download a few extra dependencies. 

If provisioning a fresh environment, you'll generally have to download about
5GB of data in total: 3GB for CUDA, about 400MB for the CuPy wheel, 800MB for
PyTorch (required by `spacy-transformers`), 500MB for the transformer weights,
and about 200MB in various other binaries.

In summary, let's say you're using CUDA 10.2, and you've installed it in
`/opt/nvidia/cuda`:

```
export CUDA_PATH="/opt/nvidia/cuda"
pip install cupy-cuda102
pip install spacy-transformers
```

## Runtime usage {#runtime}

Transformer models can be used as **drop-in replacements** for other types of
neural networks, so your spaCy pipeline can include them in a way that's
completely invisible to the user. Users will download, load and use the model in
the standard way, like any other spaCy pipeline. Instead of using the
transformers as subnetworks directly, you can also use them via the
[`Transformer`](/api/transformer) pipeline component.

![The processing pipeline with the transformer component](../images/pipeline_transformer.svg)

The `Transformer` component sets the
[`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute,
which lets you access the transformers outputs at runtime.

```bash
$ python -m spacy download en_core_trf_lg
```

```python
### Example
import spacy
from thinc.api import use_pytorch_for_gpu_memory, require_gpu

# Use the GPU, with memory allocations directed via PyTorch.
# This prevents out-of-memory errors that would otherwise occur from competing
# memory pools.
use_pytorch_for_gpu_memory()
require_gpu(0)

nlp = spacy.load("en_core_trf_lg")
for doc in nlp.pipe(["some text", "some other text"]):
    tokvecs = doc._.trf_data.tensors[-1]
```

You can also customize how the [`Transformer`](/api/transformer) component sets
annotations onto the [`Doc`](/api/doc), by customizing the `annotation_setter`.
This callback will be called with the raw input and output data for the whole
batch, along with the batch of `Doc` objects, allowing you to implement whatever
you need. The annotation setter is called with a batch of [`Doc`](/api/doc)
objects and a [`FullTransformerBatch`](/api/transformer#fulltransformerbatch)
containing the transformers data for the batch.

```python
def custom_annotation_setter(docs, trf_data):
    # TODO:
    ...

nlp = spacy.load("en_core_trf_lg")
nlp.get_pipe("transformer").annotation_setter = custom_annotation_setter
doc = nlp("This is a text")
print()  # TODO:
```

{'fraction_non_alphanumeric': 0.08117464486092874, 'fraction_numerical': 0.004058732243046437, 'mean_word_length': 5.031677465802736, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2958117', 'n_tokens_mistral': 2414, 'n_tokens_neox': 2258, 'n_words': 1071}