<jupyter_start><jupyter_text>CRFを用いた固有表現認識このノートブックでは、[sklearn-crfsuite](https://sklearn-crfsuite.readthedocs.io/en/latest/)を用いて、固有表現認識のモデルを構築します。データセットとしては　CoNLLを使用します。  準備  パッケージのインストール<jupyter_code>!pip install sklearn-crfsuite==0.3.6 seqeval==1.2.2 eli5==0.11.0 scikit-learn==0.23.2<jupyter_output>Requirement already satisfied: sklearn-crfsuite==0.3.6 in /usr/local/lib/python3.7/dist-packages (0.3.6)
Requirement already satisfied: seqeval==1.2.2 in /usr/local/lib/python3.7/dist-packages (1.2.2)
Requirement already satisfied: eli5==0.11.0 in /usr/local/lib/python3.7/dist-packages (0.11.0)
Requirement already satisfied: scikit-learn==0.23.2 in /usr/local/lib/python3.7/dist-packages (0.23.2)
Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from sklearn-crfsuite==0.3.6) (0.8.9)
Requirement already satisfied: tqdm>=2.0 in /usr/local/lib/python3.7/dist-packages (from sklearn-crfsuite==0.3.6) (4.62.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sklearn-crfsuite==0.3.6) (1.15.0)
Requirement already satisfied: python-crfsuite>=0.8.3 in /usr/local/lib/python3.7/dist-packages (from sklearn-crfsuite==0.3.6) (0.9.7)
Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from seqeval==1.2[...]<jupyter_text>インポート<jupyter_code>import eli5
import scipy
from seqeval.metrics import classification_report, f1_score
from sklearn_crfsuite import CRF
from sklearn.metrics import make_scorer
from sklearn.model_selection import RandomizedSearchCV<jupyter_output><empty_output><jupyter_text>データの読み込みまずはCoNLLのデータセットをアップロードします。ノートブックと同じ階層にDataフォルダがあり、その下に`conll2003/en`フォルダがあるので、学習・検証・テスト用データセットをアップロードしましょう。Colabでない場合は、データセットを読み込むときに正しいパスを指定します。<jupyter_code>from google.colab import files
uploaded = files.upload()
!head train.txt<jupyter_output>-DOCSTART- -X- -X- O

EU NNP B-NP B-ORG
rejects VBZ B-VP O
German JJ B-NP B-MISC
call NN I-NP O
to TO B-VP O
boycott VB I-VP O
British JJ B-NP B-MISC
lamb NN I-NP O<jupyter_text>次に、データセットを読み込みます。<jupyter_code>def load_conll(file_path):
    sents = []
    sent = []
    with open(file_path, encoding='utf-8') as f:
      for line in f:
          line = line.strip()
          if line.startswith('-DOCSTART'):
            continue
          if line:
              word, pos, _, tag = line.split()
              sent.append((word, pos, tag))
          else:
              if len(sent) == 0:
                continue
              sents.append(sent)
              sent = []
    return sents
train_sents = load_conll('train.txt')
valid_sents = load_conll('valid.txt')
test_sents = load_conll('test.txt')
train_sents[0]
print(len(train_sents))
print(len(valid_sents))
print(len(test_sents))<jupyter_output>14041
3250
3453<jupyter_text>前処理データを読み込み終えたので、特徴を定義します。今回は前後2単語に関して、以下の特徴を使います。- 小文字化した単語- 大文字だけからなるか- 単語の先頭の文字は大文字か- 数字か- 品詞<jupyter_code>def word2features(sent, i):
    word = sent[i][0]
    postag = sent[i][1]

    features = {
        'bias': 1.0,
        'word.lower()': word.lower(),
        'word.isupper()': word.isupper(),
        'word.istitle()': word.istitle(),
        'word.isdigit()': word.isdigit(),
        'postag': postag,
    }
    if i > 0:
        word1 = sent[i-1][0]
        postag1 = sent[i-1][1]
        features.update({
            '-1:word.lower()': word1.lower(),
            '-1:word.isupper()': word1.isupper(),
            '-1:word.istitle()': word1.istitle(),
            '-1:postag': postag1,
        })
    else:
        features['BOS'] = True

    if i > 1:
        word2 = sent[i-2][0]
        postag2 = sent[i-2][1]
        features.update({
            '-2:word.lower()': word2.lower(),
            '-2:word.isupper()': word2.isupper(),
            '-2:word.istitle()': word2.istitle(),
            '-2:postag': postag2,
        })
    else:
        features['-2:BOS'] = True

    if i < len(sent)-1:
        word1 = sent[i+1][0]
        postag1 = sent[i+1][1]
        features.update({
            '+1:word.lower()': word1.lower(),
            '+1:word.isupper()': word1.isupper(),
            '+1:word.istitle()': word1.istitle(),
            '+1:postag': postag1,
        })
    else:
        features['EOS'] = True

    if i < len(sent) - 2:
        word2 = sent[i+2][0]
        postag2 = sent[i+2][1]
        features.update({
            '+2:word.lower()': word2.lower(),
            '+2:word.isupper()': word2.isupper(),
            '+2:word.istitle()': word2.istitle(),
            '+2:postag': postag2,
        })
    else:
        features['+2:EOS'] = True

    return features

def sent2features(sent):
    return [word2features(sent, i) for i in range(len(sent))]

def sent2labels(sent):
    return [label for token, postag, label in sent]

def sent2tokens(sent):
    return [token for token, postag, label in sent]
sent2features(train_sents[0])[0]
X_train = [sent2features(s) for s in train_sents]
y_train = [sent2labels(s) for s in train_sents]

X_valid = [sent2features(s) for s in valid_sents]
y_valid = [sent2labels(s) for s in valid_sents]

X_test = [sent2features(s) for s in test_sents]
y_test = [sent2labels(s) for s in test_sents]<jupyter_output><empty_output><jupyter_text>モデルの学習<jupyter_code>%%time
model = CRF(
    algorithm='lbfgs',
    max_iterations=100,
    all_possible_transitions=False
)
model.fit(X_train + X_valid, y_train + y_valid)<jupyter_output>CPU times: user 46.8 s, sys: 240 ms, total: 47 s
Wall time: 46.9 s<jupyter_text>モデルの評価<jupyter_code>y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred, digits=4))<jupyter_output>precision    recall  f1-score   support

         LOC     0.8385    0.7842    0.8104      1668
        MISC     0.7602    0.6638    0.7087       702
         ORG     0.7256    0.7038    0.7145      1661
         PER     0.8041    0.8683    0.8350      1617

   micro avg     0.7861    0.7697    0.7778      5648
   macro avg     0.7821    0.7550    0.7672      5648
weighted avg     0.7857    0.7697    0.7766      5648<jupyter_text>重みの検査[eli5](https://eli5.readthedocs.io/en/latest/index.html)の`show_weights`を使って、状態と繊維の特徴に関する重みを見てみましょう。<jupyter_code>eli5.show_weights(model, top=30)<jupyter_output>/usr/local/lib/python3.7/dist-packages/sklearn/base.py:213: FutureWarning: From version 0.24, get_params will raise an AttributeError if a parameter cannot be retrieved as an instance attribute. Previously it would return None.
  FutureWarning)<jupyter_text>遷移の行列を見ると、モデルはB-XのあとにI-Xが来ることを学習していることがわかります。また、ある種の遷移はほとんど起こらないことも学習しているようです。たとえば、B-LOCのあとにI-ORGが来ることはほとんどないといったことです。  ハイパーパラメータの最適化性能を向上させるに、ハイパーパラメータの最適化をしてみましょう。ここでは、ランダムサーチと3分割交差検定を使用して、正則化パラメータの探索をしてみます。<jupyter_code>%%time
model = CRF(
    algorithm='lbfgs',
    max_iterations=100,
    all_possible_transitions=True
)

params_space = {
    'c1': scipy.stats.expon(scale=0.5),
    'c2': scipy.stats.expon(scale=0.05),
}

f1_scorer = make_scorer(f1_score)

rs = RandomizedSearchCV(
    model,
    params_space,
    cv=3,
    verbose=1,
    n_jobs=-1,
    n_iter=30,
    scoring=f1_scorer
)
rs.fit(X_train + X_valid, y_train + y_valid)
print('best params:', rs.best_params_)
print('best CV score:', rs.best_score_)
print('model size: {:0.2f}M'.format(rs.best_estimator_.size_ / 1000000))<jupyter_output>best params: {'c1': 0.028100798861946483, 'c2': 0.002451948365671623}
best CV score: 0.8532850512553268
model size: 1.61M<jupyter_text>テストデータを使って、最適なパラメータのモデルを評価してみましょう。<jupyter_code>crf = rs.best_estimator_
y_pred = crf.predict(X_test)
print(classification_report(y_test, y_pred, digits=4))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1392857142857143, 'fraction_numerical': 0.059788359788359786, 'mean_word_length': 3.38827626233314, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '328625', 'license': 'MIT', 'n_tokens_mistral': 3522, 'n_tokens_neox': 3265, 'n_words': 508, 'path': 'ch05/02_NERTraining.ipynb', 'repo_name': 'oreilly-japan/practical-nlp-ja', 'size': '7560'}