Add tokenizer.json
Browse files- README.md +2 -0
- tokenizer.json +0 -0
README.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
| 1 |
This repository doesn't contain a model, but only a tokenizer that can be used with the
|
| 2 |
`tokenizers` library.
|
| 3 |
|
|
|
|
|
|
|
| 4 |
```python
|
| 5 |
from tokenizers import Tokenizer
|
| 6 |
tokenizer = Tokenizer.from_pretrained("anthony/tokenizers-test")
|
|
|
|
| 1 |
This repository doesn't contain a model, but only a tokenizer that can be used with the
|
| 2 |
`tokenizers` library.
|
| 3 |
|
| 4 |
+
This tokenizer is just a copy of `bert-base-uncased`.
|
| 5 |
+
|
| 6 |
```python
|
| 7 |
from tokenizers import Tokenizer
|
| 8 |
tokenizer = Tokenizer.from_pretrained("anthony/tokenizers-test")
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|