
# nlp-waseda/bigbird-base-japanese

## Model description

This is a Japanese BigBird base model pretrained on Japanese Wikipedia, the Japanese portion of CC-100, and the Japanese portion of OSCAR.

## How to use

You can use this model for masked language modeling as follows:
```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("nlp-waseda/bigbird-base-japanese")
model = AutoModelForMaskedLM.from_pretrained("nlp-waseda/bigbird-base-japanese")

sentence = '[MASK] 大学 で 自然 言語 処理 を 学ぶ 。' # input should be segmented into words by Juman++ in advance
encoding = tokenizer(sentence, return_tensors='pt')
...
```

You can fine-tune this model on downstream tasks.

## Tokenization

The input text should be segmented into words by [Juman++](https://github.com/ku-nlp/jumanpp) in advance. Juman++ 2.0.0-rc3 was used for pretraining. Each word is tokenized into tokens by [sentencepiece](https://github.com/google/sentencepiece).

## Vocabulary

The vocabulary consists of 32000 tokens including words ([JumanDIC](https://github.com/ku-nlp/JumanDIC)) and subwords induced by the unigram language model of [sentencepiece](https://github.com/google/sentencepiece).

## Training procedure

This model was trained on Japanese Wikipedia (as of 20221101), the Japanese portion of CC-100, and the and the Japanese portion of OSCAR. It took two weeks using 16 NVIDIA A100 GPUs using [transformers](https://github.com/huggingface/transformers) and [DeepSpeed](https://github.com/microsoft/DeepSpeed).

The following hyperparameters were used during pretraining:
- learning_rate: 1e-4
- per_device_train_batch_size: 6
- gradient_accumulation_steps: 2
- total_train_batch_size: 192
- max_seq_length: 4096
- training_steps: 600000
- warmup_steps: 6000
- bf16: true
- deepspeed: [ds_config.json](https://huggingface.co/nlp-waseda/bigbird-base-japanese/blob/main/ds_config.json)

## Performance on JGLUE

We fine-tuned the following models and evaluated them on the dev set of JGLUE.
We tuned learning rate and training epochs for each model and task following [the JGLUE paper](https://www.jstage.jst.go.jp/article/jnlp/30/1/30_63/_pdf/-char/ja).

For the tasks other than MARC-ja, the maximum length is short, so the attention_type was set to "original_full", and fine-tuning was performed. For MARC-ja, both "block_sparse" and "original_full" were used.

| Model                         | MARC-ja/acc | JSTS/pearson | JSTS/spearman | JNLI/acc | JSQuAD/EM | JSQuAD/F1 | JComQA/acc |
|-------------------------------|--------------|---------------|----------|-----------|-----------|------------|------------|
| Waseda RoBERTa base           |  0.965       | 0.913        | 0.876         | 0.905    | 0.853     | 0.916     | 0.853      |
| Waseda RoBERTa large (seq512) | 0.969       | 0.925        | 0.890         | 0.928    | 0.910     | 0.955     | 0.900      |
| BigBird base (original_full)  |0.959       | 0.888        | 0.846         | 0.896    | 0.884     | 0.933     | 0.787      |
| BigBird base (block_sparse) |0.959       | -        | -         | -    | -    | -     | -      |

## Acknowledgments

This work was supported by AI Bridging Cloud Infrastructure (ABCI) through the "Construction of a Japanese Large-Scale General-Purpose Language Model that Handles Long Sequences" at the 3rd ABCI Grand Challenge 2022.

{"language": "ja", "license": "cc-by-sa-4.0", "datasets": ["wikipedia", "cc100", "oscar"], "mask_token": "[MASK]", "widget": [{"text": "[MASK] \u5927\u5b66 \u3067 \u81ea\u7136 \u8a00\u8a9e \u51e6\u7406 \u3092 \u5b66\u3076 \u3002"}]}
transformers pytorch big_bird fill-mask ja dataset:wikipedia dataset:cc100 dataset:oscar license:cc-by-sa-4.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #big_bird #fill-mask #ja #dataset-wikipedia #dataset-cc100 #dataset-oscar #license-cc-by-sa-4.0 #autotrain_compatible #endpoints_compatible #region-us 

nlp-waseda/bigbird-base-japanese
================================


Model description
-----------------


This is a Japanese BigBird base model pretrained on Japanese Wikipedia, the Japanese portion of CC-100, and the Japanese portion of OSCAR.


How to use
----------


You can use this model for masked language modeling as follows:


You can fine-tune this model on downstream tasks.


Tokenization
------------


The input text should be segmented into words by Juman++ in advance. Juman++ 2.0.0-rc3 was used for pretraining. Each word is tokenized into tokens by sentencepiece.


Vocabulary
----------


The vocabulary consists of 32000 tokens including words (JumanDIC) and subwords induced by the unigram language model of sentencepiece.


Training procedure
------------------


This model was trained on Japanese Wikipedia (as of 20221101), the Japanese portion of CC-100, and the and the Japanese portion of OSCAR. It took two weeks using 16 NVIDIA A100 GPUs using transformers and DeepSpeed.


The following hyperparameters were used during pretraining:


* learning\_rate: 1e-4
* per\_device\_train\_batch\_size: 6
* gradient\_accumulation\_steps: 2
* total\_train\_batch\_size: 192
* max\_seq\_length: 4096
* training\_steps: 600000
* warmup\_steps: 6000
* bf16: true
* deepspeed: ds\_config.json


Performance on JGLUE
--------------------


We fine-tuned the following models and evaluated them on the dev set of JGLUE.
We tuned learning rate and training epochs for each model and task following the JGLUE paper.


For the tasks other than MARC-ja, the maximum length is short, so the attention\_type was set to "original\_full", and fine-tuning was performed. For MARC-ja, both "block\_sparse" and "original\_full" were used.



Acknowledgments
---------------


This work was supported by AI Bridging Cloud Infrastructure (ABCI) through the "Construction of a Japanese Large-Scale General-Purpose Language Model that Handles Long Sequences" at the 3rd ABCI Grand Challenge 2022.



TAGS
#transformers #pytorch #big_bird #fill-mask #ja #dataset-wikipedia #dataset-cc100 #dataset-oscar #license-cc-by-sa-4.0 #autotrain_compatible #endpoints_compatible #region-us 
