israel commited on
Commit
fe24670
1 Parent(s): e54be65

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +10 -0
README.md CHANGED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ ```
2
+ from transformers import AutoTokenizer
3
+
4
+ tokenizer = AutoTokenizer.from_pretrained("israel/AmhWordPieceTokenizer")
5
+
6
+ ecoding = tokenizer.encode("ኮሌጁ ቢያስተምርም ወደስራ የሚመድባቸው መንግስት ነው abcs")
7
+
8
+ encoding.tokens
9
+
10
+ ```