jacobfulano commited on
Commit
4ce1645
1 Parent(s): aadf20a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -15,7 +15,7 @@ Hugging Face's [bert-base-uncased](https://huggingface.co/bert-base-uncased).
15
 
16
  __This model was trained with [ALiBi](https://arxiv.org/abs/2108.12409) on a sequence length of 1024 tokens.__
17
 
18
- ALiBi allows a model trained with a sequence length n to extrapolate to sequence lengths >2n. For more details, see [Train Short, Test Long: Attention with Linear
19
  Biases Enables Input Length Extrapolation (Press et al. 2022)](https://arxiv.org/abs/2108.12409)
20
 
21
  It is part of the family of MosaicBERT-Base models:
 
15
 
16
  __This model was trained with [ALiBi](https://arxiv.org/abs/2108.12409) on a sequence length of 1024 tokens.__
17
 
18
+ ALiBi allows a model trained with a sequence length n to easily extrapolate to sequence lengths >2n during finetuning. For more details, see [Train Short, Test Long: Attention with Linear
19
  Biases Enables Input Length Extrapolation (Press et al. 2022)](https://arxiv.org/abs/2108.12409)
20
 
21
  It is part of the family of MosaicBERT-Base models: