<jupyter_start><jupyter_text>tensorflow text classification from the web sitehttps://www.tensorflow.org/tutorials/keras/basic_text_classificationThe purpose is to predict the movie review "sentiment" as either positive or negativedisplay what backend I am using<jupyter_code>from tensorflow import keras
#import keras
import numpy as np

print('backend:',keras.backend.backend())
print('version:', keras.__version__)<jupyter_output>backend: tensorflow
version: 2.2.4-tf<jupyter_text>define the model hereI purposely put the model here because some backends ( such as plaidml ) complained about invalid config/setup```pythonvocab_size = 50000```the `vocab_size` determines how many vocabulary to use. The vocabulary index that is beyond this number will be shown with `` tag<jupyter_code># input shape is the vocabulary count used for the movie reviews 
vocab_size = 50000

model = keras.Sequential()
model.add(keras.layers.Embedding(vocab_size, 16, input_length=256))
model.add(keras.layers.GlobalAveragePooling1D())
model.add(keras.layers.Dense(16, activation='relu'))
model.add(keras.layers.Dense(1, activation='sigmoid'))

model.summary()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, 256, 16)           800000    
_________________________________________________________________
global_average_pooling1d_1 ( (None, 16)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 16)                272       
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 17        
=================================================================
Total params: 800,289
Trainable params: 800,289
Non-trainable params: 0
_________________________________________________________________<jupyter_text>download the dataThe data already has been "converted" from text to numbers<jupyter_code>imdb = keras.datasets.imdb

(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=vocab_size)<jupyter_output><empty_output><jupyter_text>show sample size and content<jupyter_code>print("Training entries: {}, labels: {}".format(len(train_data), len(train_labels)))
print("Test entries: {}, labels: {}".format(len(test_data), len(test_labels)))
print(train_data[0])<jupyter_output>[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 22665, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 21631, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 19193, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 10311, 8, 4, 107, 117, 5952, 15, 256, 4, 31050, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 12118, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]<jupyter_text>not every data (review) has the same length, so we have to "pad" the dataFor this run, we are using 265 as the max length of the review.<jupyter_code>len(train_data[0]), len(train_data[1])

# A dictionary mapping words to an integer index
word_index = imdb.get_word_index()

# The first indices are reserved
word_index = {k:(v+3) for k,v in word_index.items()} 
word_index["<PAD>"] = 0
word_index["<START>"] = 1
word_index["<UNK>"] = 2  # unknown
word_index["<UNUSED>"] = 3

reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])

def decode_review(text):
    return ' '.join([reverse_word_index.get(i, '?') for i in text])<jupyter_output><empty_output><jupyter_text>keras has the padding method<jupyter_code>train_data = keras.preprocessing.sequence.pad_sequences(train_data,
                                                        value=word_index["<PAD>"],
                                                        padding='post',
                                                        maxlen=256)

test_data = keras.preprocessing.sequence.pad_sequences(test_data,
                                                       value=word_index["<PAD>"],
                                                       padding='post',
                                                       maxlen=256)

train_data[470]<jupyter_output><empty_output><jupyter_text>show how the decoded data ( number => text ) looks like  changed to 256 length data<jupyter_code>len(train_data[0]), len(train_data[1])

decode_review(train_data[470])<jupyter_output><empty_output><jupyter_text>train the model to recognize "positive" or "negative" review using word-embedding<jupyter_code>model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

x_val = train_data[:10000]
partial_x_train = train_data[10000:]

y_val = train_labels[:10000]
partial_y_train = train_labels[10000:]

print('train shape', partial_x_train.shape)
history = model.fit(partial_x_train,
                    partial_y_train,
                    epochs=10,
                    batch_size=128,
                    validation_data=(x_val, y_val)
                    )

%matplotlib inline
from matplotlib import pyplot as plt
# show procession of training...
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])

plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper right')
plt.show()

plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])

plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()<jupyter_output><empty_output><jupyter_text>let's see how the model predicts review (positive/negative) on the unseen data<jupyter_code># evaluate againt test data
results = model.evaluate(test_data, test_labels)

print(results)<jupyter_output>25000/25000 [==============================] - 1s 37us/sample - loss: 0.2943 - acc: 0.8799
[0.2942980431747437, 0.87992]<jupyter_text>Let's visualize the prediction against the "test data"<jupyter_code># create two arrays with 0 and 1
predictions = model.predict(test_data)
positive=[]
negative=[]
good_matrix = np.zeros( (2,), dtype='int')
good_lowest = 1.0
bad_highest = 0.0
good_index = 0
bad_index = 0

for i in range(test_labels.shape[0]):
    
    # get lowest "good" and highest "bad"
    pv = predictions[i][0]
    expected = test_labels[i]
    
    # append to pos/neg list
    if expected == 1:
        positive.append(pv)
    else:
        negative.append(pv)
    
    # guess cutoff is 0.5
    guess = 1 if pv > 0.5 else 0
    
    # good count
    if guess == expected:
        good_matrix[expected] +=1
    else:
        #find the worst prediction
        if expected == 1:
            if good_lowest > pv:
                good_lowest = pv
                good_index = i
        else:
            if bad_highest < pv:
                bad_highest = pv
                bad_index = i<jupyter_output><empty_output><jupyter_text>How did this simple model predict?See the histogram to see the distribution of its predictions against positive and negative reviews.The cutoff for positive is > 0.5The cutoff for negative is <= 0.5<jupyter_code>print('Positive Reviews:', len(positive), 
      'guessed correctly:',good_matrix[1], '(' + str(good_matrix[1]*100.0/len(positive)) + '%)',
      'missed:', len(positive) - good_matrix[1], '(' + str( (len(positive) - good_matrix[1])*100.0/len(positive)) + '%)')

print('Negative Reviews:', len(negative), 
      'guessed correctly:',good_matrix[0], '(' + str(good_matrix[0]*100.0/len(negative)) + '%)',
      'missed:', len(negative) - good_matrix[0], '(' + str( (len(negative) - good_matrix[0])*100.0/len(negative)) + '%)')

print()

print('positive median:', np.median(positive))
binwidth=0.01
plt.hist(positive, bins=np.arange(min(positive), max(positive) + binwidth, binwidth))
plt.show()
plt.close()

print('negative median:', np.median(negative))
binwidth=0.01
plt.hist(negative, bins=np.arange(min(negative), max(negative) + binwidth, binwidth))
plt.show()
plt.close()<jupyter_output>Positive Reviews: 12500 guessed correctly: 10934 (87.472%) missed: 1566 (12.528%)
Negative Reviews: 12500 guessed correctly: 11064 (88.512%) missed: 1436 (11.488%)

positive median: 0.9438868<jupyter_text>good review guessed as negativeExpected good review but guessed as negative<jupyter_code>print('guessed weight:', good_lowest, 'review label value:', test_labels[good_index])
decode_review(test_data[good_index])<jupyter_output>guessed weight: 0.0006455183 review label value: 1<jupyter_text>bad review guessed as positive<jupyter_code>print('guessed weight:', bad_highest, 'review label value:', test_labels[bad_index])
decode_review(test_data[bad_index])<jupyter_output>guessed weight: 0.99890584 review label value: 0<jupyter_text>Let's check how this simple model actually  performs against my own review.<jupyter_code>from keras.preprocessing.text import text_to_word_sequence

# predict function
def mypredict(text):
    result = text_to_word_sequence(text)
    print(result)
    mydata = np.zeros(256,dtype='int')
    mydata[0] = 1
    index = 1
    for w in result:
        if w in word_index.keys():
            iv = word_index[w]
            if iv >= vocab_size:
                iv = 2
        else:
            iv=2
        mydata[index]=iv
        index += 1

    print(mydata)
    #decode_review(mydata)
    indata = np.reshape(mydata, (1,256))
    #print(indata.shape)
    #print(indata)
    gr = model.predict(indata)
    return gr[0][0]
    
mypredict("<START> this film was the worst movie I ever saw. Anyone planning to go, please spend your money else where")<jupyter_output>['start', 'this', 'film', 'was', 'the', 'worst', 'movie', 'i', 'ever', 'saw', 'anyone', 'planning', 'to', 'go', 'please', 'spend', 'your', 'money', 'else', 'where']
[   1  380   14   22   16    4  249   20   13  126  219  259 3601    8
  140  591 1142  129  278  334  121    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    [...]
{'fraction_non_alphanumeric': 0.13390161605398687, 'fraction_numerical': 0.09163558870538092, 'mean_word_length': 3.021063905747947, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '304428', 'license': 'Apache-2.0', 'n_tokens_mistral': 4507, 'n_tokens_neox': 3676, 'n_words': 1073, 'path': 'word_embed.ipynb', 'repo_name': 'sspak9/ml', 'size': '11262'}