<jupyter_start><jupyter_text>Tutorial for Chinese Sentiment analysis with hotel review data DependenciesPython 3.5, numpy, pickle, keras, tensorflow, [jieba](https://github.com/fxsjy/jieba) Optional for plottingpylab, scipy<jupyter_code>from os import listdir
from os.path import isfile, join
import jieba
import codecs
from langconv import * # convert Traditional Chinese characters to Simplified Chinese characters
import pickle
import random

from keras.models import Sequential
from keras.layers.embeddings import Embedding
from keras.layers.recurrent import GRU
from keras.preprocessing.text import Tokenizer
from keras.layers.core import Dense
from keras.utils import to_categorical
from keras.preprocessing.sequence import pad_sequences
from keras.callbacks import TensorBoard<jupyter_output><empty_output><jupyter_text>Helper function to pickle and load stuff<jupyter_code>def __pickleStuff(filename, stuff):
    save_stuff = open(filename, "wb")
    pickle.dump(stuff, save_stuff)
    save_stuff.close()
def __loadStuff(filename):
    saved_stuff = open(filename,"rb")
    stuff = pickle.load(saved_stuff)
    saved_stuff.close()
    return stuff<jupyter_output><empty_output><jupyter_text>Get lists of files, positive and negative files<jupyter_code>dataBaseDirPos = "./data/ChnSentiCorp_htl_ba_6000/pos/"
dataBaseDirNeg = "./data/ChnSentiCorp_htl_ba_6000/neg/"
positiveFiles = [dataBaseDirPos + f for f in listdir(dataBaseDirPos) if isfile(join(dataBaseDirPos, f))]
negativeFiles = [dataBaseDirNeg + f for f in listdir(dataBaseDirNeg) if isfile(join(dataBaseDirNeg, f))]<jupyter_output><empty_output><jupyter_text>Show length of samples<jupyter_code>print(len(positiveFiles))
print(len(negativeFiles))<jupyter_output>2916
3000<jupyter_text>Have a look at what's in a file(one hotel review)<jupyter_code>filename = positiveFiles[0]
with codecs.open(filename, "r", encoding="gb2312") as doc_file:
    text=doc_file.read()
    print(text)<jupyter_output>如果是我一个人再到东莞出差的话，我还会住在东莞宾馆，但是如果我和我的老板一起来东莞的话，可能会住到索菲特去。

我喜欢东莞宾馆的环境和它的几个餐厅，身处闹市但是却很安静，很舒服，宾馆的服务也很到位。

对于普通的商务旅行者来说，东莞宾馆的标准间是最物超所值的。<jupyter_text>Test removing stop wordsDemo what it looks like to tokenize the sentence and remove stop words.<jupyter_code>filename = positiveFiles[110]
with codecs.open(filename, "r", encoding="gb2312") as doc_file:
    text=doc_file.read()
    text = text.replace("\n", "")
    text = text.replace("\r", "")
print("==Orginal==:\n\r{}".format(text))
    
stopwords = [ line.rstrip() for line in codecs.open('./data/chinese_stop_words.txt',"r", encoding="utf-8") ]
seg_list = jieba.cut(text, cut_all=False)
final =[]
seg_list = list(seg_list)
for seg in seg_list:
    if seg not in stopwords:
        final.append(seg)
print("==Tokenized==\tToken count:{}\n\r{}".format(len(seg_list)," ".join(seg_list)))
print("==Stop Words Removed==\tToken count:{}\n\r{}".format(len(final)," ".join(final)))<jupyter_output>==Orginal==:

别墅型的酒店，非常特别,离海边很近.消费很平价
==Tokenized==	Token count:15

别墅 型 的 酒店 ， 非常 特别 , 离 海边 很近 . 消费 很 平价
==Stop Words Removed==	Token count:8

别墅 型 酒店 特别 海边 很近 消费 平价<jupyter_text>Prepare "doucments", a list of tuplesSome files contain abnormal encoding characters which encoding GB2312 will complain about. Solution: read as bytes then decode as GB2312 line by line, skip lines with abnormal encodings. We also convert any traditional Chinese characters to simplified Chinese characters.<jupyter_code>documents = []
for filename in positiveFiles:
    text = ""
    with codecs.open(filename, "rb") as doc_file:
        for line in doc_file:
            try:
                line = line.decode("GB2312")
            except:
                continue
            text+=Converter('zh-hans').convert(line)# Convert from traditional to simplified Chinese

            text = text.replace("\n", "")
            text = text.replace("\r", "")
    documents.append((text, "pos"))

for filename in negativeFiles:
    text = ""
    with codecs.open(filename, "rb") as doc_file:
        for line in doc_file:
            try:
                line = line.decode("GB2312")
            except:
                continue
            text+=Converter('zh-hans').convert(line)# Convert from traditional to simplified Chinese

            text = text.replace("\n", "")
            text = text.replace("\r", "")
    documents.append((text, "neg"))<jupyter_output><empty_output><jupyter_text>Optional step to save/load the documents as pickle file<jupyter_code># Uncomment those two lines to save/load the documents for later use since the step above takes a while
# __pickleStuff("./data/chinese_sentiment_corpus.p", documents)
# documents = __loadStuff("./data/chinese_sentiment_corpus.p")
print(len(documents))
print(documents[4000])<jupyter_output>5916
('优点;有电梯(4层开封宾馆连电梯都没有),晚上好像没有小姐电话骚扰(开封宾馆骚扰到最后干脆和按摩小姐电话聊天起来了)缺点;房间象招待所,设备老化,能算3星?价格偏高,性价比不好!门口乱哄哄的,感觉不好!', 'neg')<jupyter_text>shuffle the data<jupyter_code>random.shuffle(documents)<jupyter_output><empty_output><jupyter_text>Prepare the input and output for the modelEach input (hotel review) will be a list of tokens, output will be one token("pos" or "neg"). The stopwords are not removed here since the dataset is relative small and removing the stop words are not saving much traing time.<jupyter_code># Tokenize only
totalX = []
totalY = [str(doc[1]) for doc in documents]
for doc in documents:
    seg_list = jieba.cut(doc[0], cut_all=False)
    seg_list = list(seg_list)
    totalX.append(seg_list)


#Switch to below code to experiment with removing stop words
# Tokenize and remove stop words
# totalX = []
# totalY = [str(doc[1]) for doc in documents]
# stopwords = [ line.rstrip() for line in codecs.open('./data/chinese_stop_words.txt',"r", encoding="utf-8") ]
# for doc in documents:
#     seg_list = jieba.cut(doc[0], cut_all=False)
#     seg_list = list(seg_list)
#     Uncomment below code to experiment with removing stop words
#     final =[]
#     for seg in seg_list:
#         if seg not in stopwords:
#             final.append(seg)
#     totalX.append(final)<jupyter_output><empty_output><jupyter_text>Visualize distribution of sentence lengthDecide the max input sequence, here we cover up to 60% sentences. The longer input sequence, the more training time will take, but could improve  prediction accuracy.<jupyter_code>import numpy as np
import scipy.stats as stats
import pylab as pl
h = sorted([len(sentence) for sentence in totalX])
maxLength = h[int(len(h) * 0.60)]
print("Max length is: ",h[len(h)-1])
print("60% cover length up to: ",maxLength)
h = h[:5000]
fit = stats.norm.pdf(h, np.mean(h), np.std(h))  #this is a fitting indeed

pl.plot(h,fit,'-o')
pl.hist(h,normed=True)      #use this to draw histogram of your data
pl.show()<jupyter_output>Max length is:  1804
60% cover length up to:  68<jupyter_text>Words to number tokens, paddingPad input sequence to max input length if it is shorterSave the input tokenizer, since we need to use the same tokenizer for our new predition data.<jupyter_code>totalX = [" ".join(wordslist) for wordslist in totalX]  # Keras Tokenizer expect the words tokens to be seperated by space 
input_tokenizer = Tokenizer(30000) # Initial vocab size
input_tokenizer.fit_on_texts(totalX)
vocab_size = len(input_tokenizer.word_index) + 1
print("input vocab_size:",vocab_size)
totalX = np.array(pad_sequences(input_tokenizer.texts_to_sequences(totalX), maxlen=maxLength))
__pickleStuff("./data/input_tokenizer_chinese.p", input_tokenizer)<jupyter_output>input vocab_size: 22123<jupyter_text>Output, array of 0s and 1s<jupyter_code>target_tokenizer = Tokenizer(3)
target_tokenizer.fit_on_texts(totalY)
print("output vocab_size:",len(target_tokenizer.word_index) + 1)
totalY = np.array(target_tokenizer.texts_to_sequences(totalY)) -1
totalY = totalY.reshape(totalY.shape[0])
totalY[40:50]<jupyter_output><empty_output><jupyter_text>Turn output 0s and 1s to categories(one-hot vectors)<jupyter_code>totalY = to_categorical(totalY, num_classes=2)
totalY[40:50]
output_dimen = totalY.shape[1] # which is 2<jupyter_output><empty_output><jupyter_text>Save meta data for later preditionmaxLength: the input sequence lengthvocab_size: Input vocab sizeoutput_dimen: which is 2 in this example (pos or neg)sentiment_tag: either ["neg","pos"] or ["pos","neg"] matching the target tokenizer<jupyter_code>target_reverse_word_index = {v: k for k, v in list(target_tokenizer.word_index.items())}
sentiment_tag = [target_reverse_word_index[1],target_reverse_word_index[2]] 
metaData = {"maxLength":maxLength,"vocab_size":vocab_size,"output_dimen":output_dimen,"sentiment_tag":sentiment_tag}
__pickleStuff("./data/meta_sentiment_chinese.p", metaData)<jupyter_output><empty_output><jupyter_text>Build the Model, train and save itThe training data is logged to Tensorboard, we can look at it by cd into directory "./Graph/sentiment_chinese" and run"python -m tensorflow.tensorboard --logdir=."<jupyter_code>embedding_dim = 256

model = Sequential()
model.add(Embedding(vocab_size, embedding_dim,input_length = maxLength))
# Each input would have a size of (maxLength x 256) and each of these 256 sized vectors are fed into the GRU layer one at a time.
# All the intermediate outputs are collected and then passed on to the second GRU layer.
model.add(GRU(256, dropout=0.9, return_sequences=True))
# Using the intermediate outputs, we pass them to another GRU layer and collect the final output only this time
model.add(GRU(256, dropout=0.9))
# The output is then sent to a fully connected layer that would give us our final output_dim classes
model.add(Dense(output_dimen, activation='softmax'))
# We use the adam optimizer instead of standard SGD since it converges much faster
tbCallBack = TensorBoard(log_dir='./Graph/sentiment_chinese', histogram_freq=0,
                            write_graph=True, write_images=True)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()
model.fit(totalX, totalY, validation_split=0.1, batch_size=32, epochs=20, verbose=1, callbacks=[tbCallBack])
model.save('./data/sentiment_chinese_model.HDF5')

print("Saved model!")<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, 68, 256)           5663488   
_________________________________________________________________
gru_1 (GRU)                  (None, 68, 256)           393984    
_________________________________________________________________
gru_2 (GRU)                  (None, 256)               393984    
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 514       
=================================================================
Total params: 6,451,970
Trainable params: 6,451,970
Non-trainable params: 0
_________________________________________________________________
Train on 5324 samples, validate on 592 samples
Epoch 1/20
5324/5324 [==============================] - 49s 9ms/step - loss: 0.6808 -[...]<jupyter_text>Below are prediction codeFunction to load the meta data and the model we just trained.<jupyter_code>model = None
sentiment_tag = None
maxLength = None
def loadModel():
    global model, sentiment_tag, maxLength
    metaData = __loadStuff("./data/meta_sentiment_chinese.p")
    maxLength = metaData.get("maxLength")
    vocab_size = metaData.get("vocab_size")
    output_dimen = metaData.get("output_dimen")
    sentiment_tag = metaData.get("sentiment_tag")
    embedding_dim = 256
    if model is None:
        model = Sequential()
        model.add(Embedding(vocab_size, embedding_dim, input_length=maxLength))
        # Each input would have a size of (maxLength x 256) and each of these 256 sized vectors are fed into the GRU layer one at a time.
        # All the intermediate outputs are collected and then passed on to the second GRU layer.
        model.add(GRU(256, dropout=0.9, return_sequences=True))
        # Using the intermediate outputs, we pass them to another GRU layer and collect the final output only this time
        model.add(GRU(256, dropout=0.9))
        # The output is then sent to a fully connected layer that would give us our final output_dim classes
        model.add(Dense(output_dimen, activation='softmax'))
        # We use the adam optimizer instead of standard SGD since it converges much faster
        model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
        model.load_weights('./data/sentiment_chinese_model.HDF5')
        model.summary()
    print("Model weights loaded!")<jupyter_output><empty_output><jupyter_text>Functions to convert sentence to model input, and predict result<jupyter_code>def findFeatures(text):
    text=Converter('zh-hans').convert(text)
    text = text.replace("\n", "")
    text = text.replace("\r", "") 
    seg_list = jieba.cut(text, cut_all=False)
    seg_list = list(seg_list)
    text = " ".join(seg_list)
    textArray = [text]
    input_tokenizer_load = __loadStuff("./data/input_tokenizer_chinese.p")
    textArray = np.array(pad_sequences(input_tokenizer_load.texts_to_sequences(textArray), maxlen=maxLength))
    return textArray
def predictResult(text):
    if model is None:
        print("Please run \"loadModel\" first.")
        return None
    features = findFeatures(text)
    predicted = model.predict(features)[0] # we have only one sentence to predict, so take index 0
    predicted = np.array(predicted)
    probab = predicted.max()
    predition = sentiment_tag[predicted.argmax()]
    return predition, probab<jupyter_output><empty_output><jupyter_text>Calling the load model function<jupyter_code>loadModel()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_3 (Embedding)      (None, 68, 256)           5663488   
_________________________________________________________________
gru_5 (GRU)                  (None, 68, 256)           393984    
_________________________________________________________________
gru_6 (GRU)                  (None, 256)               393984    
_________________________________________________________________
dense_3 (Dense)              (None, 2)                 514       
=================================================================
Total params: 6,451,970
Trainable params: 6,451,970
Non-trainable params: 0
_________________________________________________________________
Model weights loaded!<jupyter_text>Try some new comments, feel free to try your ownThe result tuple consists the predicted result and likehood.<jupyter_code>predictResult("还好，床很大而且很干净，前台很友好，很满意，下次还来。")
predictResult("床上有污渍，房间太挤不透气，空调不怎么好用。")
predictResult("房间有点小但是设备还齐全，没有异味。")
predictResult("房间还算干净，一般般吧，短住还凑合。")
predictResult("开始不太满意，前台好说话换了一间，房间很干净没有异味。")
predictResult("你是个SB")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11836599136499502, 'fraction_numerical': 0.020192627034207905, 'mean_word_length': 4.904313725490196, 'pattern_counts': {'":': 4, '<': 73, '<?xml version=': 0, '>': 73, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '522128', 'license': 'MIT', 'n_tokens_mistral': 5105, 'n_tokens_neox': 4916, 'n_words': 1230, 'path': 'Chinese_rationality_analasis/chinese_sentiment_analysis.ipynb', 'repo_name': 'yingshaoxo/Chinese_rationality_analasis_or_classification', 'size': '15055'}