<jupyter_start><jupyter_text>[Sudachiベースの学習済みWord2Vecモデルを使う](https://ohke.hateblo.jp/entry/2019/06/01/120000)  [chiVe: Japanese Word Embedding with Sudachi & NWJC](https://github.com/WorksApplications/chiVe)<jupyter_code>!wget https://object-storage.tyo2.conoha.io/v1/nc_2520839e1f9641b08211a5c85243124a/chive/chive-1.0-mc5-20190314.tar.gz
!tar -zxvf ./chive-1.0-mc5-20190314.tar.gz
!pip install gensim
# モデルのロード
from gensim.models import KeyedVectors
from gensim.test.utils import datapath

nwjc_model = KeyedVectors.load_word2vec_format(
    datapath('/content/chive-1.0-mc5-20190314.txt'),
    binary=False
)

# 語数, 次元数
print(len(nwjc_model.vocab), nwjc_model.vector_size)
print(nwjc_model.most_similar('平成', topn=5))
print(nwjc_model.most_similar(positive=['兄弟', '女'], negative=['男'], topn=5))<jupyter_output>/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.
  if np.issubdtype(vec.dtype, np.int):<jupyter_text>[Pythonで形態素解析器Sudachiを使う (SudachiPy)](https://ohke.hateblo.jp/entry/2019/03/09/101500)  [SudachiPy](https://github.com/WorksApplications/SudachiPy)<jupyter_code>!pip install SudachiPy
!pip install https://object-storage.tyo2.conoha.io/v1/nc_2520839e1f9641b08211a5c85243124a/sudachi/SudachiDict_core-20200127.tar.gz
!sudachipy tokenize -h
from sudachipy import tokenizer
from sudachipy import dictionary

tokenizer_obj = dictionary.Dictionary().create()
# Multi-granular tokenization
# using `system_core.dic` or `system_full.dic` version 20190781
# you may not be able to replicate this particular example due to dictionary you use

mode = tokenizer.Tokenizer.SplitMode.C
[m.surface() for m in tokenizer_obj.tokenize("国家公務員", mode)]
# => ['国家公務員']
mode = tokenizer.Tokenizer.SplitMode.B
[m.surface() for m in tokenizer_obj.tokenize("国家公務員", mode)]
# => ['国家', '公務員']
mode = tokenizer.Tokenizer.SplitMode.A
[m.surface() for m in tokenizer_obj.tokenize("国家公務員", mode)]
# => ['国家', '公務', '員']
# Morpheme information

m = tokenizer_obj.tokenize("食べ", mode)[0]

m.surface() # => '食べ'
m.dictionary_form() # => '食べる'
m.reading_form() # => 'タベ'
m.part_of_speech() # => ['動詞', '一般', '*', '*', '下一段-バ行', '連用形-一般']
# Normalization

tokenizer_obj.tokenize("附属", mode)[0].normalized_form()
# => '付属'
tokenizer_obj.tokenize("SUMMER", mode)[0].normalized_form()
# => 'サマー'
tokenizer_obj.tokenize("シュミレーション", mode)[0].normalized_form()
# => 'シミュレーション'
from sudachipy import tokenizer

tokenizer_obj = dictionary.Dictionary().create()
print(type(tokenizer_obj))
# <class 'sudachipy.tokenizer.Tokenizer'>

text = '友人・我孫子とスカイツリーでスパゲティを食った。'

mode = tokenizer.Tokenizer.SplitMode.C
tokens = tokenizer_obj.tokenize(text, tokenizer.Tokenizer.SplitMode.C)
print(type(tokens))
# <class 'sudachipy.morphemelist.MorphemeList'>

for t in tokens:
    print(t.surface(), t.part_of_speech(), t.reading_form(), t.normalized_form())<jupyter_output><class 'sudachipy.tokenizer.Tokenizer'>
<class 'sudachipy.morphemelist.MorphemeList'>
友人 ['名詞', '普通名詞', '一般', '*', '*', '*'] ユウジン 友人
・ ['補助記号', '一般', '*', '*', '*', '*'] ・ ・
我孫子 ['名詞', '固有名詞', '地名', '一般', '*', '*'] アビコ 我孫子
と ['助詞', '格助詞', '*', '*', '*', '*'] ト と
スカイツリー ['名詞', '固有名詞', '一般', '*', '*', '*'] スカイツリー スカイツリー
で ['助詞', '格助詞', '*', '*', '*', '*'] デ で
スパゲティ ['名詞', '普通名詞', '一般', '*', '*', '*'] スパゲティ スパゲッティ
を ['助詞', '格助詞', '*', '*', '*', '*'] ヲ を
食っ ['動詞', '一般', '*', '*', '五段-ワア行', '連用形-促音便'] クッ 食う
た ['助動詞', '*', '*', '*', '助動詞-タ', '終止形-一般'] タ た
。 ['補助記号', '句点', '*', '*', '*', '*'] 。 。
{'fraction_non_alphanumeric': 0.20591487009397458, 'fraction_numerical': 0.04090657822001106, 'mean_word_length': 5.932950191570881, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 21, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '231015', 'license': 'Apache-2.0', 'n_tokens_mistral': 1823, 'n_tokens_neox': 1714, 'n_words': 256, 'path': 'chiVe.ipynb', 'repo_name': 'lvisdd/nlp_tutorial', 'size': '3618'}