<jupyter_start><jupyter_text>Finetuning of the pretrained Japanese BERT modelFinetune the pretrained model to solve multi-class classification problems.  This notebook requires the following objects:- trained sentencepiece model (model and vocab files)- pretraiend Japanese BERT modelDataset is livedoor ニュースコーパス in https://www.rondhuit.com/download.html.  We make test:dev:train = 2:2:6 datasets. Results:- Full training data  - BERT with SentencePiece    ```                    precision    recall  f1-score   support    dokujo-tsushin       0.98      0.94      0.96       178      it-life-hack       0.96      0.97      0.96       172     kaden-channel       0.99      0.98      0.99       176    livedoor-homme       0.98      0.88      0.93        95       movie-enter       0.96      0.99      0.98       158            peachy       0.94      0.98      0.96       174              smax       0.98      0.99      0.99       167      sports-watch       0.98      1.00      0.99       190        topic-news       0.99      0.98      0.98       163         micro avg       0.97      0.97      0.97      1473         macro avg       0.97      0.97      0.97      1473      weighted avg       0.97      0.97      0.97      1473    ```  - sklearn GradientBoostingClassifier with MeCab    ```                      precision    recall  f1-score   support    dokujo-tsushin       0.89      0.86      0.88       178      it-life-hack       0.91      0.90      0.91       172     kaden-channel       0.90      0.94      0.92       176    livedoor-homme       0.79      0.74      0.76        95       movie-enter       0.93      0.96      0.95       158            peachy       0.87      0.92      0.89       174              smax       0.99      1.00      1.00       167      sports-watch       0.93      0.98      0.96       190        topic-news       0.96      0.86      0.91       163         micro avg       0.92      0.92      0.92      1473         macro avg       0.91      0.91      0.91      1473      weighted avg       0.92      0.92      0.91      1473    ```- Small training data (1/5 of full training data)  - BERT with SentencePiece    ```                    precision    recall  f1-score   support    dokujo-tsushin       0.97      0.87      0.92       178      it-life-hack       0.86      0.86      0.86       172     kaden-channel       0.95      0.94      0.95       176    livedoor-homme       0.82      0.82      0.82        95       movie-enter       0.97      0.99      0.98       158            peachy       0.89      0.95      0.92       174              smax       0.94      0.96      0.95       167      sports-watch       0.97      0.97      0.97       190        topic-news       0.94      0.94      0.94       163         micro avg       0.93      0.93      0.93      1473         macro avg       0.92      0.92      0.92      1473      weighted avg       0.93      0.93      0.93      1473    ```  - sklearn GradientBoostingClassifier with MeCab    ```                    precision    recall  f1-score   support    dokujo-tsushin       0.82      0.71      0.76       178      it-life-hack       0.86      0.88      0.87       172     kaden-channel       0.91      0.87      0.89       176    livedoor-homme       0.67      0.63      0.65        95       movie-enter       0.87      0.95      0.91       158            peachy       0.70      0.78      0.73       174              smax       1.00      1.00      1.00       167      sports-watch       0.87      0.95      0.91       190        topic-news       0.92      0.82      0.87       163         micro avg       0.85      0.85      0.85      1473         macro avg       0.85      0.84      0.84      1473      weighted avg       0.86      0.85      0.85      1473    ```<jupyter_code>import configparser
import glob
import os
import pandas as pd
import subprocess
import sys
import tarfile 
from urllib.request import urlretrieve

CURDIR = os.getcwd()
CONFIGPATH = os.path.join(CURDIR, os.pardir, 'config.ini')
config = configparser.ConfigParser()
config.read(CONFIGPATH)<jupyter_output><empty_output><jupyter_text>Data preparingYou need execute the following cells just once.<jupyter_code>FILEURL = config['FINETUNING-DATA']['FILEURL']
FILEPATH = config['FINETUNING-DATA']['FILEPATH']
EXTRACTDIR = config['FINETUNING-DATA']['TEXTDIR']<jupyter_output><empty_output><jupyter_text>Download and unzip data.<jupyter_code>%%time

urlretrieve(FILEURL, FILEPATH)

mode = "r:gz"
tar = tarfile.open(FILEPATH, mode) 
tar.extractall(EXTRACTDIR) 
tar.close()<jupyter_output><empty_output><jupyter_text>Data preprocessing.<jupyter_code>def extract_txt(filename):
    with open(filename) as text_file:
        # 0: URL, 1: timestamp
        text = text_file.readlines()[2:]
        text = [sentence.strip() for sentence in text]
        text = list(filter(lambda line: line != '', text))
        return ''.join(text)
categories = [ 
    name for name 
    in os.listdir( os.path.join(EXTRACTDIR, "text") ) 
    if os.path.isdir( os.path.join(EXTRACTDIR, "text", name) ) ]

categories = sorted(categories)
categories
table = str.maketrans({
    '\n': '',
    '\t': '　',
    '\r': '',
})
%%time

all_text = []
all_label = []

for cat in categories:
    files = glob.glob(os.path.join(EXTRACTDIR, "text", cat, "{}*.txt".format(cat)))
    files = sorted(files)
    body = [ extract_txt(elem).translate(table) for elem in files ]
    label = [cat] * len(body)
    
    all_text.extend(body)
    all_label.extend(label)
df = pd.DataFrame({'text' : all_text, 'label' : all_label})
df.head()
df = df.sample(frac=1, random_state=23).reset_index(drop=True)
df.head()<jupyter_output><empty_output><jupyter_text>Save data as tsv files.  test:dev:train = 2:2:6. To check the usability of finetuning, we also prepare sampled training data (1/5 of full training data).<jupyter_code>df[:len(df) // 5].to_csv( os.path.join(EXTRACTDIR, "test.tsv"), sep='\t', index=False)
df[len(df) // 5:len(df)*2 // 5].to_csv( os.path.join(EXTRACTDIR, "dev.tsv"), sep='\t', index=False)
df[len(df)*2 // 5:].to_csv( os.path.join(EXTRACTDIR, "train.tsv"), sep='\t', index=False)

### 1/5 of full training data.
# df[:len(df) // 5].to_csv( os.path.join(EXTRACTDIR, "test.tsv"), sep='\t', index=False)
# df[len(df) // 5:len(df)*2 // 5].to_csv( os.path.join(EXTRACTDIR, "dev.tsv"), sep='\t', index=False)
# df[len(df)*2 // 5:].sample(frac=0.2, random_state=23).to_csv( os.path.join(EXTRACTDIR, "train.tsv"), sep='\t', index=False)<jupyter_output><empty_output><jupyter_text>Finetune pre-trained modelIt will take a lot of hours to execute the following cells on CPU environment.  You can also use colab to recieve the power of TPU. You need to uplode the created data onto your GCS bucket.[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1zZH2GWe0U-7GjJ2w2duodFfEUptvHjcx)<jupyter_code>PRETRAINED_MODEL_PATH = '../model/model.ckpt-1400000'
FINETUNE_OUTPUT_DIR = '../model/livedoor_output'
%%time
# It will take many hours on CPU environment.

!python3 ../src/run_classifier.py \
  --task_name=livedoor \
  --do_train=true \
  --do_eval=true \
  --data_dir=../data/livedoor \
  --model_file=../model/wiki-ja.model \
  --vocab_file=../model/wiki-ja.vocab \
  --init_checkpoint={PRETRAINED_MODEL_PATH} \
  --max_seq_length=512 \
  --train_batch_size=4 \
  --learning_rate=2e-5 \
  --num_train_epochs=10 \
  --output_dir={FINETUNE_OUTPUT_DIR}<jupyter_output><empty_output><jupyter_text>Predict using the finetuned modelLet's predict test data using the finetuned model.<jupyter_code>%%time
# It will take many hours on CPU environment.

!python3 ../src/run_classifier.py \
  --task_name=livedoor \
  --do_predict=true \
  --data_dir=../data/livedoor \
  --model_file=../model/wiki-ja.model \
  --vocab_file=../model/wiki-ja.vocab \
  --output_dir={FINETUNE_OUTPUT_DIR}
import sys
sys.path.append("../src")

from run_classifier import LivedoorProcessor
processor = LivedoorProcessor()
label_list = processor.get_labels()
result = pd.read_csv(FINETUNE_OUTPUT_DIR+"/test_results.tsv", sep='\t', header=None)
result.head()<jupyter_output><empty_output><jupyter_text>Read test data set and add prediction results.<jupyter_code>import pandas as pd
test_df = pd.read_csv("../data/livedoor/test.tsv", sep='\t')
test_df['predict'] = [ label_list[idx] for idx in result.idxmax(axis=1) ]
test_df.head()
sum( test_df['label'] == test_df['predict'] ) / len(test_df)<jupyter_output><empty_output><jupyter_text>A littel more detailed check using `sklearn.metrics`.<jupyter_code>!pip install scikit-learn
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
print(classification_report(test_df['label'], test_df['predict']))
print(confusion_matrix(test_df['label'], test_df['predict']))<jupyter_output><empty_output><jupyter_text>Simple baseline model.<jupyter_code>import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
train_df = pd.read_csv("../data/livedoor/train.tsv", sep='\t')
dev_df = pd.read_csv("../data/livedoor/dev.tsv", sep='\t')
test_df = pd.read_csv("../data/livedoor/test.tsv", sep='\t')
!apt-get install -q -y mecab libmecab-dev mecab-ipadic mecab-ipadic-utf8
!pip install mecab-python3==0.7
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.ensemble import GradientBoostingClassifier
import MeCab
m = MeCab.Tagger("-Owakati")
train_dev_df = pd.concat([train_df, dev_df])
train_dev_xs = train_dev_df['text'].apply(lambda x: m.parse(x))
train_dev_ys = train_dev_df['label']

test_xs = test_df['text'].apply(lambda x: m.parse(x))
test_ys = test_df['label']
vectorizer = TfidfVectorizer(max_features=750)
train_dev_xs_ = vectorizer.fit_transform(train_dev_xs)
test_xs_ = vectorizer.transform(test_xs)<jupyter_output><empty_output><jupyter_text>The following set up is not exactly identical to that of BERT because inside Classifier it uses `train_test_split` with shuffle.  In addition, parameters are not well tuned, however, we think it's enough to check the power of BERT.<jupyter_code>%%time

model = GradientBoostingClassifier(n_estimators=200,
                                   validation_fraction=len(dev_df)/len(train_df),
                                   n_iter_no_change=5,
                                   tol=0.01,
                                   random_state=23)

### 1/5 of full training data.
# model = GradientBoostingClassifier(n_estimators=200,
#                                    validation_fraction=len(dev_df)/len(train_df),
#                                    n_iter_no_change=5,
#                                    tol=0.01,
#                                    random_state=23)

model.fit(train_dev_xs_, train_dev_ys)
print(classification_report(test_ys, model.predict(test_xs_)))
print(confusion_matrix(test_ys, model.predict(test_xs_)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1165644171779141, 'fraction_numerical': 0.06162439337057046, 'mean_word_length': 2.2342315664791235, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 58, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '428014', 'license': 'Apache-2.0', 'n_tokens_mistral': 4039, 'n_tokens_neox': 3706, 'n_words': 816, 'path': 'notebook/finetune-to-livedoor-corpus.ipynb', 'repo_name': 'miyamonz/bert-japanese', 'size': '10921'}