What tokenizer is best?

#2
by turian - opened

What tokenizer is best to use with flair/pos-english?

It was not clear to me from the model card what tokenizer was used during training. And the default FLAIR tokenizer is relatively simple.

flair org

The tagger was trained on Ontonotes, so it reuses the tokenization from this dataset.

Sign up or log in to comment