<commit_before># Efficiently scaling transformer inference
This repo includes

* Benchmarks to replicate the results in the paper
* A complete implementation of text generation with a transformer using the techniques in the paper

## To replicate the head-to-head benchmarks from the paper at 540B scale

* Ensure you are running on 64 TPUv4 chips, smaller numbers would be better suited for smaller models

```python
python3 run_benchmark.py
```

This generates the latency and MFU numbers for the PALM and MT-NLG implementations in the following plot from the paper. The FastertTransformer baseline numbers are drawn from NVIDIA's repo.

<p align="center">
<img src="./assets/fasterformer_comparison_v2.pdf" alt="fastertransformer_comparison" width="600"/>
</p>


## To generate text

```python
python3 run_generation.py --model 540b --quantized False
```

The current weight paths only load internal PaLM weights, which are unavailable externally. Using this externally will require modification of the checkpoint paths and transformer layer def to suit your own models.<commit_msg>- Adding arxiv link<commit_after># Scaling Transformer Inference Efficiency
This repo includes

* Benchmarks to replicate the results in the paper - [Scaling Transformer Inference Efficiency](http://arxiv.org/abs/2211.05102)
* A complete implementation of text generation with a transformer using the techniques in the paper

## To replicate the head-to-head benchmarks from the paper at 540B scale

* Ensure you are running on 64 TPUv4 chips, smaller numbers would be better suited for smaller models

```python
python3 run_benchmark.py
```

This generates the latency and MFU numbers for the PALM and MT-NLG implementations in the following plot from the paper. The FastertTransformer baseline numbers are drawn from NVIDIA's repo.

<p align="center">
<img src="./assets/fasterformer_comparison_v2.pdf" alt="fastertransformer_comparison" width="600"/>
</p>


## To generate text

```python
python3 run_generation.py --model 540b --quantized False
```

The current weight paths only load internal PaLM weights, which are unavailable externally. Using this externally will require modification of the checkpoint paths and transformer layer def to suit your own models.
{'fraction_non_alphanumeric': 0.06169551708832668, 'fraction_numerical': 0.017310252996005325, 'mean_word_length': 5.141689373297003, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4258078', 'n_tokens_mistral': 630, 'n_tokens_neox': 549, 'n_words': 286}