hajime9652 commited on
Commit
fca1a82
1 Parent(s): 3fd5fe6

Update Readme

Browse files
Files changed (1) hide show
  1. README.md +62 -0
README.md CHANGED
@@ -1,2 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  See this documents https://qiita.com/mkt3/items/4d0ae36f3f212aee8002
2
  published by https://github.com/mkt3
 
1
+ ---
2
+ language:
3
+ - ja
4
+ thumbnail:
5
+ tags:
6
+ - xlnet
7
+ - lm-head
8
+ - causal-lm
9
+ license:
10
+ datasets:
11
+ metrics:
12
+ ---
13
+
14
+ # XLNet-japanese
15
+
16
+ ## Model description
17
+ This model require Mecab and senetencepiece with XLNetTokenizer.
18
+ See details https://qiita.com/mkt3/items/4d0ae36f3f212aee8002
19
+
20
+ ## Intended uses & limitations
21
+
22
+ #### How to use
23
+
24
+ ```python
25
+ import MeCab
26
+ import subprocess
27
+
28
+ from transformers import (
29
+ pipeline,
30
+ XLNetLMHeadModel,
31
+ XLNetTokenizer
32
+ )
33
+
34
+ class XLNet():
35
+ def __init__(self):
36
+ cmd = 'echo `mecab-config --dicdir`"/mecab-ipadic-neologd"'
37
+ path = (subprocess.Popen(cmd, stdout=subprocess.PIPE,
38
+ shell=True).communicate()[0]).decode('utf-8')
39
+ self.m = MeCab.Tagger(f"-Owakati -d {path}")
40
+
41
+ self.gen_model = XLNetLMHeadModel.from_pretrained("hajime9652/xlnet-japanese")
42
+ self.gen_tokenizer = XLNetTokenizer.from_pretrained("hajime9652/xlnet-japanese")
43
+
44
+ def generate(self, prompt="福岡のご飯は美味しい。コンパクトで暮らしやすい街。"):
45
+ prompt = self.m.parse(prompt)
46
+ inputs = self.gen_tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
47
+ prompt_length = len(self.gen_tokenizer.decode(inputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))
48
+ outputs = self.gen_model.generate(inputs, max_length=200, do_sample=True, top_p=0.95, top_k=60)
49
+ generated = prompt + self.gen_tokenizer.decode(outputs[0])[prompt_length:]
50
+ return generated
51
+ ```
52
+
53
+ #### Limitations and bias
54
+
55
+ ## Training data
56
+
57
+
58
+ ## Training procedure
59
+
60
+ ## Eval results
61
+
62
+ ###
63
  See this documents https://qiita.com/mkt3/items/4d0ae36f3f212aee8002
64
  published by https://github.com/mkt3