
 
BertJapanese
	
Overview
	
The BERT models trained on Japanese text.
There are models with two different tokenization methods:
Tokenize with MeCab and WordPiece. This requires some extra dependencies, fugashi which is a wrapper around MeCab.
Tokenize into characters.
To use MecabTokenizer, you should pip install transformers["ja"] (or pip install -e .["ja"] if you install
from source) to install dependencies.
See details on cl-tohoku repository.
Example of using a model with MeCab and WordPiece tokenization:
	Copied
 import torch
 from transformers import AutoModel, AutoTokenizer
 bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese")
 tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese")
 ## Input Japanese Text
 line = "吾輩は猫である。"
 inputs = tokenizer(line, return_tensors="pt")
 print(tokenizer.decode(inputs["input_ids"][0]))
[CLS] 吾輩 は 猫 で ある 。 [SEP]
 outputs = bertjapanese(**inputs)
Example of using a model with Character tokenization:
	Copied
 bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese-char")
 tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-char")
 ## Input Japanese Text
 line = "吾輩は猫である。"
 inputs = tokenizer(line, return_tensors="pt")
 print(tokenizer.decode(inputs["input_ids"][0]))
[CLS] 吾 輩 は 猫 で あ る 。 [SEP]
 outputs = bertjapanese(**inputs)
Tips:
This implementation is the same as BERT, except for tokenization method. Refer to the documentation of BERT for more usage examples.
This model was contributed by cl-tohoku.
BertJapaneseTokenizer
	
class transformers.BertJapaneseTokenizer
<
source
>
(
vocab_file
spm_file = None
do_lower_case = False
do_word_tokenize = True
do_subword_tokenize = True
word_tokenizer_type = 'basic'
subword_tokenizer_type = 'wordpiece'
never_split = None
unk_token = '[UNK]'
sep_token = '[SEP]'
pad_token = '[PAD]'
cls_token = '[CLS]'
mask_token = '[MASK]'
mecab_kwargs = None
sudachi_kwargs = None
jumanpp_kwargs = None
**kwargs
)
Parameters 
vocab_file (str) —
Path to a one-wordpiece-per-line vocabulary file.
spm_file (str, optional) —
Path to SentencePiece file (generally has a .spm or .model
extension) that contains the vocabulary.
do_lower_case (bool, optional, defaults to True) —
Whether to lower case the input. Only has an effect when do_basic_tokenize=True.
do_word_tokenize (bool, optional, defaults to True) —
Whether to do word tokenization.
do_subword_tokenize (bool, optional, defaults to True) —
Whether to do subword tokenization.
word_tokenizer_type (str, optional, defaults to "basic") —
Type of word tokenizer. Choose from [“basic”, “mecab”, “sudachi”, “jumanpp”].
subword_tokenizer_type (str, optional, defaults to "wordpiece") —
Type of subword tokenizer. Choose from [“wordpiece”, “character”, “sentencepiece”,].
mecab_kwargs (dict, optional) —
Dictionary passed to the MecabTokenizer constructor.
sudachi_kwargs (dict, optional) —
Dictionary passed to the SudachiTokenizer constructor.
jumanpp_kwargs (dict, optional) —
Dictionary passed to the JumanppTokenizer constructor.
Construct a BERT tokenizer for Japanese text.
This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer
to: this superclass for more information regarding those methods.
build_inputs_with_special_tokens
<
source
>
(
token_ids_0: typing.List[int]
token_ids_1: typing.Optional[typing.List[int]] = None
)
→
List[int]
Parameters 
token_ids_0 (List[int]) —
List of IDs to which the special tokens will be added.
token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.
Returns
List[int]
List of input IDs with the appropriate special tokens.
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A BERT sequence has the following format:
single sequence: [CLS] X [SEP]
pair of sequences: [CLS] A [SEP] B [SEP]
convert_tokens_to_string
<
source
>
(
tokens
)
Converts a sequence of tokens (string) in a single string.
create_token_type_ids_from_sequences
<
source
>
(
token_ids_0: typing.List[int]
token_ids_1: typing.Optional[typing.List[int]] = None
)
→
List[int]
Parameters 
token_ids_0 (List[int]) —
List of IDs.
token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.
Returns
List[int]
List of token type IDs according to the given sequence(s).
Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence
pair mask has the following format:
	Copied
0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |
If token_ids_1 is None, this method only returns the first portion of the mask (0s).
get_special_tokens_mask
<
source
>
(
token_ids_0: typing.List[int]
token_ids_1: typing.Optional[typing.List[int]] = None
already_has_special_tokens: bool = False
)
→
List[int]
Parameters 
token_ids_0 (List[int]) —
List of IDs.
token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.
already_has_special_tokens (bool, optional, defaults to False) —
Whether or not the token list is already formatted with special tokens for the model.
Returns
List[int]
A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer prepare_for_model method.
←BertGeneration
Bertweet→
BertJapanese
Overview
BertJapaneseTokenizer
