<jupyter_start><jupyter_text>Thai2Vec Text Feature Extraction Using ULMFitThis notebook demonstrates how to use `thai2vec` to convert Thai texts into vectors (roll credits!), after which said vectors can be used for various machine learning tasks such as classification, clustering, translation, question answering and so on. The idea is to train a language model that "understands" the texts then extract certain vectors that the model "thinks" represents the texts we want. We use 113,962 product reviews scraped from an ecommerce website as our sample dataset. Our workflow is as follows:* Perform 80/20 train-validation split* Minimal text cleaning and tokenization using `newmm` engine of `pyThaiNLP`* Get embeddings of ecommerce dataset from train set* Load pretrained Thai Wikipedia embeddings; for those embeddings which exist only in ecommerce dataset, we use the average of Wikipedia embeddings instead* Train language model* Cut out the encoder and extract vectors from there.  Imports<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline

import re
import html
import dill as pickle
from IPython.display import Image
from IPython.core.display import HTML 
from collections import Counter
from sklearn.model_selection import train_test_split

from fastai.text import *
from pythainlp.tokenize import word_tokenize
from utils import *

DATA_PATH='/home/ubuntu/Projects/new2vec/ec_data/'
RAW_PATH = f'{DATA_PATH}raw/'
MODEL_PATH = f'{DATA_PATH}models/'

raw_files = !ls {RAW_PATH}<jupyter_output><empty_output><jupyter_text>Raw Texts to CSVs<jupyter_code># dfs = []
# j = 0
# for i in range(len(raw_files)):
#     raw_file = raw_files[i]
#     star = i+1
#     with open(f'{RAW_PATH}{raw_file}') as f:
#         lines = f.readlines()
#         for line in lines:
#             df = pd.DataFrame({'target':star,'text':line},index=[j])
#             dfs.append(df)
#             j+=1
# df_raw = pd.concat(dfs)
#df_raw.to_csv(f'{DATA_PATH}raw.csv',index=False)
df_raw = pd.read_csv(f'{DATA_PATH}raw.csv')
df_raw.head()
df_raw.shape, Counter(df_raw['target'])<jupyter_output><empty_output><jupyter_text>Data Preparation We use the format of `(label, text)` for our train and validation csvs. We regroup the labels a little after some eyeballing since there are a lot of overlapping in meaning for the labels:* `good`: 4-5 stars* `neutral`: 3 star* `bad`: 1-2 starsWe use 80/20 train-validation split.<jupyter_code>df_raw.query('target==5').head(10)
df_raw.query('target==4').head(10)
df_raw.query('target==3').head(10)
df_raw.query('target==2').head(10)
df_raw.query('target==1').head(10)
def star_to_rating(star):
    if star > 3:
        rating = 2
    elif star < 3:
        rating = 0
    else:
        rating = 1
    return(rating)

df_raw.iloc[:,0] = df_raw.iloc[:,0].apply(star_to_rating)
df_raw.head()
Counter(df_raw['target'])
# #train/validation split
# df_trn, df_val = train_test_split(df_raw, test_size = 0.2, random_state=1412)

# df_trn.to_csv(f'{DATA_PATH}train.csv', index=False)
# df_val.to_csv(f'{DATA_PATH}valid.csv', index=False)
df_trn = pd.read_csv(f'{DATA_PATH}train.csv')
df_val = pd.read_csv(f'{DATA_PATH}valid.csv')
df_trn.shape, df_val.shape
df_trn.head()<jupyter_output><empty_output><jupyter_text>Language Modeling  Text Processing We first determine the vocab for the reviews, then train a language model based on our training set. We perform the following minimal text processing:* The token `xbos` is used to note start of a text since we will be chaining them together for the language model training. * `pyThaiNLP`'s `newmm` word tokenizer is used to tokenize the texts.* `tkrep` is used to replace repetitive characters such as `อร่อยมากกกกกก` becoming `อรอ่ยมาtkrep6ก`<jupyter_code>max_vocab = 60000
min_freq = 2

df_trn = pd.read_csv(f'{DATA_PATH}train.csv',chunksize=30000)
df_val = pd.read_csv(f'{DATA_PATH}valid.csv',chunksize=30000)

trn_lm,trn_tok,trn_labels,itos_cls,stoi_cls,freq = numericalizer(df_trn)
val_lm,val_tok,val_labels,itos_cls,stoi_cls,freq = numericalizer(df_val,itos_cls)
# np.save(f'{MODEL_PATH}trn_tok.npy', trn_tok)
# np.save(f'{MODEL_PATH}val_tok.npy', val_tok)
# np.save(f'{MODEL_PATH}trn_lm.npy', trn_lm)
# np.save(f'{MODEL_PATH}val_lm.npy', val_lm)
# pickle.dump(itos_cls, open(f'{MODEL_PATH}itos_cls.pkl', 'wb'))
trn_lm = np.load(f'{MODEL_PATH}trn_lm.npy')
val_lm = np.load(f'{MODEL_PATH}val_lm.npy')
itos_cls = pickle.load(open(f'{MODEL_PATH}itos_cls.pkl', 'rb'))
vocab_size=len(itos_cls)
trn_lm.shape,val_lm.shape,vocab_size
trn_lm[:2]<jupyter_output><empty_output><jupyter_text>Load Pretrained Language Model Instead of starting from random weights, we import the language model pretrained on Wikipedia (see `pretrained_wiki.ipynb`). For words that appear only in the Wongnai dataset but not Wikipedia, we start with the average of all embeddings instead. Max vocab size is set at 60,000 and minimum frequency of 2 consistent with the pretrained model. We ended up with 19,998 embeddings for all reviews.<jupyter_code>em_sz = 300
vocab_size = len(itos_cls)
wgts = torch.load(f'{MODEL_PATH}thwiki_model2.h5', map_location=lambda storage, loc: storage)
itos_pre = pickle.load(open(f'{MODEL_PATH}itos_pre.pkl','rb'))
stoi_pre = collections.defaultdict(lambda:-1, {v:k for k,v in enumerate(itos_pre)})
#pretrained weights
wgts = merge_wgts(em_sz, wgts, itos_pre, itos_cls)<jupyter_output><empty_output><jupyter_text>Train Language Model<jupyter_code>#model parameters
em_sz,nh,nl = 300,1150,3
wd=1e-7
bptt=70
bs=64
opt_fn = partial(optim.Adam, betas=(0.8, 0.99))
weight_factor = 0.7
drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15])*weight_factor
#data loader
trn_dl = LanguageModelLoader(np.concatenate(trn_lm), bs, bptt)
val_dl = LanguageModelLoader(np.concatenate(val_lm), bs, bptt)
md = LanguageModelData(path=DATA_PATH, pad_idx=1, n_tok=vocab_size, 
                       trn_dl=trn_dl, val_dl=val_dl, bs=bs, bptt=bptt)
#model fitter
learner= md.get_model(opt_fn, em_sz, nh, nl, 
    dropouti=drops[0], dropout=drops[1], wdrop=drops[2], dropoute=drops[3], dropouth=drops[4])
learner.metrics = [accuracy]
#load the saved models + new embeddings
learner.model.load_state_dict(wgts)
#find optimal learning rate
learner.lr_find2(start_lr = 1e-6, end_lr=0.1)
learner.sched.plot()
lr=0.03/10
lr
#train to warmup
learner.fit(lr, 1, wds=wd, use_clr=(10,3), cycle_len=1)
#use_clr (peak as ratio of lr, iterations to grow and descend e.g. 10 is 1/10 grow and 9/10 descend)
learner.unfreeze()
learner.fit(lr, 1, wds=wd, use_clr=(8,10), cycle_len=10)
learner.sched.plot_loss()
learner.save('ec_lm')
learner.save_encoder('ec_enc')<jupyter_output><empty_output><jupyter_text>Text Feature Extraction Note that we are converting text to vectors without teaching the model any label. Thus, similar sentence structures describing negative and positive sentiments will have "similar" vectors, as opposed to two different sentence structures describing the same sentiment.<jupyter_code>from utils import *
#load, adjust batch size and set model to eval mode
learner.load('ec_lm')
m = learner.model
m[0].bs=1
m.eval()

#sample sentence
ss = 'สวัสดีครับพี่น้องเสื้อ'
v = document_vector(ss,m,stoi_cls)
v.shape,v[:10]
from numpy import dot
from numpy.linalg import norm

def cosine_sim(a,b):
    return(dot(a, b)/(norm(a)*norm(b)))

v1 = document_vector('สินค้าดีมาก',m,stoi_cls)
v2 = document_vector('สินค้าแย่มาก',m,stoi_cls)
v3 = document_vector('ใช้ดีมากมาย',m,stoi_cls)
v4 = document_vector('สินค้าใช้ดี',m,stoi_cls)

print('Same structure; different sentiments: ',cosine_sim(v1,v2))
print('Different structures; same sentiment: ',cosine_sim(v1,v3))
print('Same structure; same sentiment: ',cosine_sim(v1,v4))
print('Different structures; different sentiments: ',cosine_sim(v3,v2))<jupyter_output>Same structure; different sentiments:  0.75125444
Different structures; same sentiment:  0.08146612
Same structure; same sentiment:  0.6695422
Different structures; different sentiments:  0.052667953<jupyter_text>Clustering In cases where we do not know any labels to our data and want to group them solely on their semantic meanings, we can use the features extracted to perform clustering algorithms such as K-means.<jupyter_code>df_val = pd.read_csv(f'{DATA_PATH}valid.csv')
Counter(df_val['target'])
#get all vectors from validation texts
vec_val = []
for i in range(df_val.shape[0]):
    vec_val.append(document_vector(df_val.iloc[i,1],m,stoi_cls).reshape(1,-1))
cluster_val = pd.DataFrame(np.concatenate(vec_val,axis=0))
cluster_val.head()
#find optimal number of clusters
from sklearn.cluster import KMeans

inertia = []
for k in range(2, 50):
    print(k)
    kmeans = KMeans(n_clusters=k)
    kmeans.fit(cluster_val)
    inertia.append(kmeans.inertia_)

fig = plt.figure(figsize=(15, 5))
plt.plot(range(2, 50), inertia)
kmeans = KMeans(n_clusters=10)
kmeans.fit(cluster_val)
Counter(kmeans.labels_)
df_val['kmeans_lab']=kmeans.labels_
#cluster containing a lot of english reviews
df_val.query('kmeans_lab==0').head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11932195878573011, 'fraction_numerical': 0.025371150011079105, 'mean_word_length': 6.124704025256511, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '484769', 'license': 'MIT', 'n_tokens_mistral': 3471, 'n_tokens_neox': 3284, 'n_words': 832, 'path': 'notebook/ulmfit_ec.ipynb', 'repo_name': 'titipata/thai2vec', 'size': '9026'}