<jupyter_start><jupyter_text>`wongnai-corpus` Classification Benchmark We provide two benchmarks for 5-star multi-class classification of [wongnai-corpus](https://github.com/wongnai/wongnai-corpus): [fastText](https://github.com/facebookresearch/fastText) and [ULMFit](https://github.com/cstorm125/thai2fit). In both cases, we first finetune the embeddings using all data. The benchmark numbers are based on the test set. Performance metric is the micro-averaged F1 by the test set of [Wongnai Challenge](https://www.kaggle.com/c/wongnai-challenge-review-rating-prediction/data).| model     | micro_f1_public | micro_f1_private | |-----------|-----------------|------------------|| **ULMFit** | **0.59313**          | **0.60322**           || fastText | 0.5145          | 0.5109           || LinearSVC | 0.5022          | 0.4976           || Kaggle Score | 0.59139          | 0.58139          || [BERT](https://github.com/ThAIKeras/bert) | 0.56612 | 0.57057 |<jupyter_code># #uncomment if you are running from google colab
# !pip install sklearn_crfsuite
# !pip install https://github.com/PyThaiNLP/pythainlp/archive/dev.zip
# !pip install fastai==1.0.45
# !wget https://github.com/wongnai/wongnai-corpus/raw/master/review/review_dataset.zip; unzip review_dataset.zip
# !mkdir wongnai_data; ls
import pandas as pd
import numpy as np
from ast import literal_eval
from tqdm import tqdm_notebook
from collections import Counter
import re

#viz
import matplotlib.pyplot as plt
import seaborn as sns

from fastai.text import *
from fastai.callbacks import CSVLogger

from pythainlp import word_tokenize

ft_data = 'ft_data/'<jupyter_output><empty_output><jupyter_text>Oversampling We oversampled class `1` and `2` for 11 and 3 times respectively in order to balance the classes.<jupyter_code>train_df = pd.read_csv('w_review_train.csv',sep=';',header=None).drop_duplicates()
train_df.columns = ['review','rating']
test_df = pd.read_csv('test_file.csv',sep=';')
test_df['rating'] = 0
all_df = pd.concat([pd.DataFrame(test_df['review']),\
                   pd.DataFrame(train_df['review'])]).reset_index(drop=True)
train_df.rating.value_counts() / train_df.shape[0]
two_df = pd.concat([train_df[train_df.rating==2].copy() for i in range(2)]).reset_index(drop=True)
one_df = pd.concat([train_df[train_df.rating==1].copy() for i in range(10)]).reset_index(drop=True)
train_bal = pd.concat([train_df,one_df,two_df]).reset_index(drop=True)
train_bal.rating.value_counts() / train_bal.shape[0]<jupyter_output><empty_output><jupyter_text>[fastText](https://github.com/facebookresearch/fastText) Model We used embeddings pretrained on [Thai Wikipedia Dump](https://github.com/facebookresearch/fastText/blob/master/docs/pretrained-vectors.md) and finetuned them using all of `wisesight-sentiment` using skipgram model. After that, we do a 5-class classification.| model    | micro_f1_public | micro_f1_private ||----------|-----------------|------------------|| fastText | 0.5145          | 0.5109           |<jupyter_code>df_txts = ['train','train_bal','test']
dfs = [train_df,train_bal,test_df]

for i in range(3):
    df = dfs[i]
    ft_lines = []
    for _,row in df.iterrows():
        ft_lab = f'__label__{row["rating"]}'
        ft_text = replace_newline(f'{row["review"]}')
        ft_line = f'{ft_lab} {ft_text}'
        ft_lines.append(ft_line)

    doc = '\n'.join(ft_lines)
    with open(f'{ft_data}{df_txts[i]}.txt','w') as f:
        f.write(doc)
    f.close()
#for fasttext embedding finetuning
ft_lines = []
for _,row in all_df.iterrows():
    ft_lab = '__label__0'
    ft_text = replace_newline(f'{row["review"]}')
    ft_line = f'{ft_lab} {ft_text}'
    ft_lines.append(ft_line)

doc = '\n'.join(ft_lines)
with open(f'{ft_data}df_all.txt','w') as f:
    f.write(doc)
f.close()
#finetune with all data
!/home/charin/fastText-0.1.0/fasttext skipgram \
-pretrainedVectors 'model/wiki.th.vec' -dim 300 \
-input ft_data/df_all.txt -output 'model/finetuned'
#train classifier
!/home/charin/fastText-0.1.0/fasttext supervised \
-input 'ft_data/train_bal.txt' -output 'model/wongnai_bal' \
-pretrainedVectors 'model/finetuned.vec' -epoch 5 -dim 300 -wordNgrams 2 
#get prediction
preds = !/home/charin/fastText-0.1.0/fasttext predict 'model/wongnai_bal.bin' 'ft_data/test.txt'
pred_lab = np.array([int(i.split('__')[-1]) for i in preds])
submit_df = pd.DataFrame({'reviewID':test_df.reviewID,
                          'rating':pred_lab})
submit_df.head()
submit_df.to_csv('submit_fastttext_bal.csv',index=False)<jupyter_output><empty_output><jupyter_text>LinearSVC Model Code for LinearSVC is provided by [@lukkiddd](https://github.com/lukkiddd).| model     | micro_f1_public | micro_f1_private | |-----------|-----------------|------------------|| LinearSVC | 0.5022          | 0.4976           |<jupyter_code>X_train, y_train = train_bal['review'], train_bal['rating']
X_test = test_df['review']
import string
def process_text(text):
    nopunc = [char for char in text if char not in string.punctuation]
    nopunc = ''.join(nopunc)
    return [word for word in word_tokenize(nopunc, engine='ulmfit') if word and not re.search(pattern=r"\s+", string=word)]
def split_text(text):
    return text.split()

train_splits = []
test_splits = []
for i in tqdm_notebook(range(train_bal.shape[0])):
    train_splits.append(' '.join(process_text(train_bal['review'][i])))
for i in tqdm_notebook(range(test_df.shape[0])):
    test_splits.append(' '.join(process_text(test_df['review'][i])))
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.svm import LinearSVC

text_clf = Pipeline([
    ('vect', CountVectorizer(tokenizer=split_text, ngram_range=(1,2))),
    ('tfidf', TfidfTransformer()),
    ('clf', LinearSVC()),
])

text_clf.fit(X_train, y_train)
pred_lab = text_clf.predict(X_test)
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown='ignore')
pred_lab = text_clf.predict(X_test)
submit_df = pd.DataFrame({'reviewID':test_df.reviewID,
                          'rating':pred_lab})
submit_df.head()
submit_df.to_csv('submit_linearsvc.csv',index=False)<jupyter_output><empty_output><jupyter_text>[ULMFit](https://github.com/cstorm125/thai2fit) Model | model     | micro_f1_public | micro_f1_private | |-----------|-----------------|------------------|| ULMFit | 0.59590          | 0.59731           |<jupyter_code># #uncomment if you are running from google colab
# !pip install sklearn_crfsuite
# !pip install https://github.com/PyThaiNLP/pythainlp/archive/dev.zip
# !pip install fastai==1.0.45
# !wget https://github.com/wongnai/wongnai-corpus/raw/master/review/review_dataset.zip; unzip review_dataset.zip
# !mkdir wongnai_data; ls
import pandas as pd
import numpy as np
from ast import literal_eval
from tqdm import tqdm_notebook
from collections import Counter
import re

#viz
import matplotlib.pyplot as plt
import seaborn as sns

from fastai.text import *
from fastai.callbacks import CSVLogger

from pythainlp.ulmfit import *

model_path = 'wongnai_data/'
#process data
train_df = pd.read_csv('w_review_train.csv',sep=';',header=None).drop_duplicates()
train_df.columns = ['review','rating']
test_df = pd.read_csv('test_file.csv',sep=';')
test_df['rating'] = 0
all_df = pd.concat([pd.DataFrame(test_df['review']),\
                   pd.DataFrame(train_df['review'])]).reset_index(drop=True)
two_df = pd.concat([train_df[train_df.rating==2].copy() for i in range(2)]).reset_index(drop=True)
one_df = pd.concat([train_df[train_df.rating==1].copy() for i in range(10)]).reset_index(drop=True)
train_bal = pd.concat([train_df,one_df,two_df]).reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Finetune Language Model<jupyter_code># tt = Tokenizer(tok_func = ThaiTokenizer, lang = 'th', pre_rules = pre_rules_th, post_rules=post_rules_th)
# processor = [TokenizeProcessor(tokenizer=tt, chunksize=10000, mark_fields=False),
#             NumericalizeProcessor(vocab=None, max_vocab=60000, min_freq=3)]

# data_lm = (TextList.from_df(all_df, model_path, cols=['review'], processor=processor)
#     .random_split_by_pct(valid_pct = 0.01, seed = 1412)
#     .label_for_lm()
#     .databunch(bs=64))
# data_lm.sanity_check()
# data_lm.save('wongnai_lm.pkl')
data_lm = load_data(model_path,'wongnai_lm.pkl')
data_lm.sanity_check()
len(data_lm.train_ds), len(data_lm.valid_ds)
data_lm.show_batch(5)
next(iter(data_lm.train_dl))
config = dict(emb_sz=400, n_hid=1550, n_layers=4, pad_token=1, qrnn=False, tie_weights=True, out_bias=True,
             output_p=0.25, hidden_p=0.1, input_p=0.2, embed_p=0.02, weight_p=0.15)
trn_args = dict(drop_mult=0.9, clip=0.12, alpha=2, beta=1)

learn = language_model_learner(data_lm, AWD_LSTM, config=config, pretrained=False, **trn_args)

#load pretrained models
learn.load_pretrained(**_THWIKI_LSTM)
learn.predict('สวัสดีครับพี่น้องเสื้อ', 50, temperature=0.5)
learn.lr_find()
learn.recorder.plot()
len(learn.data.vocab.itos)
#train frozen
print('training frozen')
learn.freeze_to(-1)
learn.fit_one_cycle(1, 1e-3, moms=(0.8, 0.7))
#train unfrozen
print('training unfrozen')
learn.unfreeze()
learn.fit_one_cycle(10, 1e-4, moms=(0.8, 0.7))
learn.save('wongnai_lm')
learn.save_encoder('wongnai_enc')<jupyter_output><empty_output><jupyter_text>Classification<jupyter_code>test_df.head()
tt = Tokenizer(tok_func = ThaiTokenizer, lang = 'th', pre_rules = pre_rules_th, post_rules=post_rules_th)
processor = [TokenizeProcessor(tokenizer=tt, chunksize=10000, mark_fields=False),
            NumericalizeProcessor(vocab=data_lm.vocab, max_vocab=60000, min_freq=3)]

data_cls = (TextList.from_df(train_bal, model_path, cols=['review'], processor=processor)
    .random_split_by_pct(valid_pct = 0.01, seed = 1412)
    .label_from_df('rating')
    .add_test(TextList.from_df(test_df, model_path, cols=['review'], processor=processor))
    .databunch(bs=32)
    )

data_cls.sanity_check()
data_cls.save('wongnai_cls.pkl')
#make sure we got the right number of vocab
len(data_cls.vocab.itos), len(data_lm.vocab.itos)
data_cls.show_batch(5)
config = dict(emb_sz=400, n_hid=1550, n_layers=4, pad_token=1, qrnn=False,
             output_p=0.25, hidden_p=0.1, input_p=0.2, embed_p=0.02, weight_p=0.15)
trn_args = dict(bptt=70, drop_mult=0.5, alpha=2, beta=1)

learn = text_classifier_learner(data_cls, AWD_LSTM, config=config, pretrained=False, **trn_args)
learn.opt_func = partial(optim.Adam, betas=(0.7, 0.99))

#load pretrained finetuned model
learn.load_encoder('wongnai_enc')
learn.lr_find()
learn.recorder.plot()
# #train unfrozen
# learn.freeze_to(-1)
# learn.fit_one_cycle(1, 2e-2, moms=(0.8, 0.7))
# #gradual unfreezing
# learn.freeze_to(-2)
# learn.fit_one_cycle(1, slice(1e-2 / (2.6 ** 4), 1e-2), moms=(0.8, 0.7))
# learn.freeze_to(-3)
# learn.fit_one_cycle(1, slice(5e-3 / (2.6 ** 4), 5e-3), moms=(0.8, 0.7))
# learn.unfreeze()
# learn.fit_one_cycle(1, slice(1e-3 / (2.6 ** 4), 1e-3), moms=(0.8, 0.7))<jupyter_output><empty_output><jupyter_text>```epoch     train_loss  valid_loss  accuracy1         1.187845    1.158394    0.472803Total time: 08:20epoch     train_loss  valid_loss  accuracy1         0.889035    0.828990    0.629707Total time: 08:39epoch     train_loss  valid_loss  accuracy1         0.760357    0.751162    0.656904Total time: 11:40epoch     train_loss  valid_loss  accuracy1         0.628719    0.721673    0.669456Total time: 18:06```  Submission<jupyter_code>config = dict(emb_sz=400, n_hid=1550, n_layers=4, pad_token=1, qrnn=False,
             output_p=0.25, hidden_p=0.1, input_p=0.2, embed_p=0.02, weight_p=0.15)
trn_args = dict(bptt=70, drop_mult=0.5, alpha=2, beta=1)

learn = text_classifier_learner(data_cls, AWD_LSTM, config=config, pretrained=False, **trn_args)
learn.opt_func = partial(optim.Adam, betas=(0.7, 0.99))

learn.load('wongnai_cls');
probs,y= learn.get_preds(DatasetType.Test, ordered=True)
preds = np.argmax(probs.numpy(),1) + 1
Counter(preds)
submit_df = pd.DataFrame({'reviewID': test_df.reviewID,'rating':preds})
submit_df.head()
submit_df.to_csv('submit_ulmfit.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15999008837862394, 'fraction_numerical': 0.038572726521846866, 'mean_word_length': 4.6952022577610535, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 13, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '82717', 'license': 'Apache-2.0', 'n_tokens_mistral': 4884, 'n_tokens_neox': 4594, 'n_words': 800, 'path': 'notebooks/wongnai_classification.ipynb', 'repo_name': 'Dobita21/pythainlp', 'size': '12107'}