<commit_before># Big Bird: Transformers for Longer Sequences

Not an official Google product.

# What is BigBird?
BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.

As a consequence of the capability to handle longer context,
BigBird drastically improves performance on various NLP tasks such as question answering and summarization.

More details and comparisons can be found in our [presentation](https://docs.google.com/presentation/d/1FdMNqG2b8XYc89_v7-_2sba7Iz6YAlXXWuMxUbrKFK0/preview).


# Citation
If you find this useful, please cite our [NeurIPS 2020 paper](https://papers.nips.cc/paper/2020/hash/c8512d142a2d849725f31a9a7a361ab9-Abstract.html):
```
@article{zaheer2020bigbird,
  title={Big bird: Transformers for longer sequences},
  author={<NAME>, <NAME> and Ainslie, Joshua and Alberti, <NAME> Ontanon, Santiago and Pham, <NAME> <NAME> and Wang, Qifan and <NAME> and others},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}
```


# Code

The most important directory is `core`.
There are three main files in `core`.

*   [attention.py](bigbird/core/attention.py):
    Contains BigBird linear attention mechanism
*   [encoder.py](bigbird/core/encoder.py):
    Contains the main long sequence encoder stack
*   [modeling.py](bigbird/core/modeling):
    Contains packaged BERT and seq2seq transformer models with BigBird attention


### Colab/IPython Notebook

A quick fine-tuning demonstration for text classification is provided in
[imdb.ipynb](bigbird/classifier/imdb.ipynb)


### Create GCP Instance
Please create a project first and create an instance in a zone which has quota as follows

```bash
gcloud compute instances create \
  bigbird \
  --zone=europe-west4-a \
  --machine-type=n1-standard-16 \
  --boot-disk-size=50GB \
  --image-project=ml-images \
  --image-family=tf-2-3-1 \
  --maintenance-policy TERMINATE \
  --restart-on-failure \
  --scopes=cloud-platform

gcloud compute tpus create \
  bigbird \
  --zone=europe-west4-a \
  --accelerator-type=v3-32 \
  --version=2.3.1<commit_msg>fix link of modeling.py<commit_after># Big Bird: Transformers for Longer Sequences

Not an official Google product.

# What is BigBird?
BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.

As a consequence of the capability to handle longer context,
BigBird drastically improves performance on various NLP tasks such as question answering and summarization.

More details and comparisons can be found in our [presentation](https://docs.google.com/presentation/d/1FdMNqG2b8XYc89_v7-_2sba7Iz6YAlXXWuMxUbrKFK0/preview).


# Citation
If you find this useful, please cite our [NeurIPS 2020 paper](https://papers.nips.cc/paper/2020/hash/c8512d142a2d849725f31a9a7a361ab9-Abstract.html):
```
@article{zaheer2020bigbird,
  title={Big bird: Transformers for longer sequences},
  author={<NAME> and <NAME> and <NAME>, <NAME>, <NAME> and <NAME> and <NAME> and others},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}
```


# Code

The most important directory is `core`.
There are three main files in `core`.

*   [attention.py](bigbird/core/attention.py):
    Contains BigBird linear attention mechanism
*   [encoder.py](bigbird/core/encoder.py):
    Contains the main long sequence encoder stack
*   [modeling.py](bigbird/core/modeling.py):
    Contains packaged BERT and seq2seq transformer models with BigBird attention


### Colab/IPython Notebook

A quick fine-tuning demonstration for text classification is provided in
[imdb.ipynb](bigbird/classifier/imdb.ipynb)


### Create GCP Instance
Please create a project first and create an instance in a zone which has quota as follows

```bash
gcloud compute instances create \
  bigbird \
  --zone=europe-west4-a \
  --machine-type=n1-standard-16 \
  --boot-disk-size=50GB \
  --image-project=ml-images \
  --image-family=tf-2-3-1 \
  --maintenance-policy TERMINATE \
  --restart-on-failure \
  --scopes=cloud-platform

gcloud compute tpus create \
  bigbird \
  --zone=europe-west4-a \
  --accelerator-type=v3-32 \
  --version=2.3.1
{'fraction_non_alphanumeric': 0.09685979655019902, 'fraction_numerical': 0.029632905793896505, 'mean_word_length': 4.30868544600939, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3449756', 'n_tokens_mistral': 1522, 'n_tokens_neox': 1390, 'n_words': 456}