<jupyter_start><jupyter_text>Dataset 这个项目主要是做情感分类，顺带学习一下词嵌入的相关内容。用到的数据集叫aclImdb_v1，是用于二进制情绪分类的大型电影评论数据集，其涵盖比基准数据集更多的数据，其中有 25,000 条电影评论用于训练，25,000 条用于测试，还有其他未经标记的数据可供使用，该数据集包含原始文本和已处理的单词格式包。<jupyter_code>url = "https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"
dataset = tf.keras.utils.get_file("aclImdb_v1.tar.gz", url,
                                    untar=True, cache_dir='.',
                                    cache_subdir='')
dataset_dir = os.path.join(os.path.dirname(dataset), 'aclImdb')
os.listdir(dataset_dir)
train_dir = os.path.join(dataset_dir, 'train')
os.listdir(train_dir)<jupyter_output><empty_output><jupyter_text>unsup文件夹用不到，需要删除，防止影响我们构建Dataset。<jupyter_code>remove_dir = os.path.join(train_dir, 'unsup')
shutil.rmtree(remove_dir)<jupyter_output><empty_output><jupyter_text>从训练集里分20%作为验证集<jupyter_code>batch_size = 64
seed = 123
train_ds = tf.keras.preprocessing.text_dataset_from_directory(
    os.path.join(dataset_dir,'train'), batch_size=batch_size, validation_split=0.2, 
    subset='training', seed=seed)
val_ds = tf.keras.preprocessing.text_dataset_from_directory(
    os.path.join(dataset_dir,'train'), batch_size=batch_size, validation_split=0.2, 
    subset='validation', seed=seed)<jupyter_output>Found 25000 files belonging to 2 classes.
Using 20000 files for training.
Found 25000 files belonging to 2 classes.
Using 5000 files for validation.<jupyter_text>看看数据集大概长啥样<jupyter_code>for text_batch, label_batch in train_ds.take(1):
      for i in range(5):
            print(label_batch[i].numpy(), text_batch.numpy()[i])
AUTOTUNE = tf.data.AUTOTUNE

train_ds = train_ds.cache().shuffle(10000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)<jupyter_output><empty_output><jupyter_text>Text Preprocessing  标准化 文本预处理要处理几件事：* 把所有字母转成小写字母* 把"\"替换成空格* 去除所有标点符号<jupyter_code>def custom_standardization(input_data):
    lowercase = tf.strings.lower(input_data)
    stripped_html = tf.strings.regex_replace(lowercase, '<br />', ' ')
    return tf.strings.regex_replace(stripped_html, '[%s]' % re.escape(string.punctuation), '')<jupyter_output><empty_output><jupyter_text>看看处理效果<jupyter_code>for text_batch, label_batch in train_ds.take(1):
      for i in range(5):
            print(label_batch[i].numpy(), custom_standardization(text_batch.numpy()[i]))<jupyter_output>1 tf.Tensor(b'this is a good example a film that in spite of the low rating is more than worth watching the story is engaging and it doesnt take long before the chemistry between nicole kidman and <NAME> grabs your attention the acting is first class and the characters are represented well sometimes it feels like the director couldnt decide himself between drama and romantic comedy ben chillers portrayal of the law abiding and shy englishman with porn sm magazines hidden in the bedroom creates plenty of moments for laughs as does the look in nicole kidmans eyes when she is offering john his first taste of intimacy in a long time other times the actors and especially nicole kidman give this comedy quite expertly a dramatic slant', shape=(), dtype=string)
1 tf.Tensor(b'i recently had the pleasure of seeing the big bad swim at the ft lauderdale film festival and i must say it is the best film i have seen all year and the only film i have ever felt inspired to write a commentreview on[...]<jupyter_text>Vectorization Layertf提供了vectorization layer来将字符串标准化、切割、将单个字符串映射成一个整数。<jupyter_code>vocab_size = 10000

vectorize_layer = TextVectorization(
    standardize=custom_standardization,
    max_tokens=vocab_size,
    output_mode='int')<jupyter_output><empty_output><jupyter_text>把字典构造出来<jupyter_code>text_ds = train_ds.map(lambda x, y: x) #去除label
vectorize_layer.adapt(text_ds)<jupyter_output><empty_output><jupyter_text>看看字典长啥样，比较长取20个看看<jupyter_code>vectorize_layer.get_vocabulary()[:20]<jupyter_output><empty_output><jupyter_text>Build Model<jupyter_code>embedding_dim=16

model = Sequential([
  vectorize_layer,
  Embedding(vocab_size, embedding_dim, name="embedding",mask_zero=True),
  GlobalAveragePooling1D(),
  Dense(16, activation='relu'),
  Dense(1)
])<jupyter_output><empty_output><jupyter_text>GlobalAveragePooling1D()是把一个batch里的词向量求平均值，忽略了语序等影响。后面会考虑使用RNN<jupyter_code>model.compile(optimizer='adam',
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>结果只有positive和negative，是二元分类，所以loss函数用的是BinaryCrossentropy<jupyter_code>history1 = model.fit(
    train_ds,
    validation_data=val_ds, 
    epochs=15)
show_acc.plot(history1)<jupyter_output><empty_output><jupyter_text>过拟合了，最高准确率大概89%  Save Learned Word Embedding有时候在大规模预料上训练得到的embedding layer是可以继续拿来在其他task中使用的<jupyter_code>weights = model.get_layer('embedding').get_weights()[0]
vocab = vectorize_layer.get_vocabulary()<jupyter_output><empty_output><jupyter_text>在Embedding Projector中可视化[Embedding Projector](http://projector.tensorflow.org/)是一个高维数据可视化工具，可以用来可视化我们的word embedding结果<jupyter_code>out_v = io.open('vectors.tsv', 'w', encoding='utf-8')
out_m = io.open('metadata.tsv', 'w', encoding='utf-8')

for index, word in enumerate(vocab):
    if  index == 0: continue # skip 0, it's padding.
    vec = weights[index] 
    out_v.write('\t'.join([str(x) for x in vec]) + "\n")
    out_m.write(word + "\n")
out_v.close()
out_m.close()<jupyter_output><empty_output><jupyter_text>去upload试一下吧  使用RNN<jupyter_code>model2=tf.keras.Sequential([
    vectorize_layer,
    Embedding(vocab_size, 64, name="embedding",mask_zero=True),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1)
])
model2.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(1e-4),
              metrics=['accuracy'])
history2 = model2.fit(train_ds, epochs=15,
                    validation_data=val_ds)
show_acc.plot(history2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11277330264672036, 'fraction_numerical': 0.01660364951504192, 'mean_word_length': 6.242857142857143, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '159614', 'license': 'MIT', 'n_tokens_mistral': 2430, 'n_tokens_neox': 2440, 'n_words': 375, 'path': 'notebooks/3_sentiment_classification.ipynb', 'repo_name': 'nofacer/TensorCertificate', 'size': '6088'}