<jupyter_start><jupyter_text>作业更换TokenEmbedding预训练模型，使用VisualDL查看相应的TokenEmbedding可视化效果，并尝试更换后的TokenEmbedding计算句对语义相似度。本作业详细步骤，可参考[Day01作业教程](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/education/day01.md)，记得star PaddleNLP，收藏起来，随时跟进最新功能噢。**作业结果提交**：1. 截图提交可视化结果（图片注明作业可视化结果）。2. 通篇执行每段代码，并保留执行结果。  PaddleNLP词向量应用展示6.7日NLP直播打卡课开始啦**[直播链接请戳这里，每晚20:00-21:30👈](http://live.bilibili.com/21689802)****[课程地址请戳这里👈](https://aistudio.baidu.com/aistudio/course/introduce/24177)**欢迎来课程**QQ群**（群号:618354318）交流吧~~词向量（Word embedding），即把词语表示成实数向量。“好”的词向量能体现词语直接的相近关系。词向量已经被证明可以提高NLP任务的性能，例如语法分析和情感分析。 图1：词向量示意图PaddleNLP已预置多个公开的预训练Embedding，您可以通过使用`paddlenlp.embeddings.TokenEmbedding`接口加载预训练Embedding，从而提升训练效果。本篇教程将依次介绍`paddlenlp.embeddings.TokenEmbedding`的初始化和文本表示效果，并通过文本分类训练的例子展示其对训练提升的效果。<jupyter_code>!pip install --upgrade paddlenlp -i https://pypi.org/simple<jupyter_output><empty_output><jupyter_text>加载TokenEmbedding`TokenEmbedding()`参数- `embedding_name`将模型名称以参数形式传入TokenEmbedding，加载对应的模型。默认为`w2v.baidu_encyclopedia.target.word-word.dim300`的词向量。- `unknown_token`未知token的表示，默认为[UNK]。- `unknown_token_vector`未知token的向量表示，默认生成和embedding维数一致，数值均值为0的正态分布向量。- `extended_vocab_path`扩展词汇列表文件路径，词表格式为一行一个词。如引入扩展词汇列表，trainable=True。- `trainable`Embedding层是否可被训练。True表示Embedding可以更新参数，False为不可更新。默认为True。<jupyter_code>from paddlenlp.embeddings import TokenEmbedding

# 初始化TokenEmbedding， 预训练embedding未下载时会自动下载并加载数据
# 需要更换所选的词向量
token_embedding = TokenEmbedding(embedding_name="w2v.people_daily.target.bigram-char.dim300")

# 查看token_embedding详情
print(token_embedding)<jupyter_output>100%|██████████| 389084/389084 [00:12<00:00, 32125.66it/s]
[2021-06-09 09:48:48,907] [    INFO] - Loading token embedding...
[2021-06-09 09:48:54,604] [    INFO] - Finish loading embedding vector.
[2021-06-09 09:48:54,607] [    INFO] - Token Embedding info:             
Unknown index: 356053             
Unknown token: [UNK]             
Padding index: 356054             
Padding token: [PAD]             
Shape :[356055, 300]<jupyter_text>**首先查看一下可以用的tokenembedding有哪些**这里根据训练集，维度，语言，功能有很多区分<jupyter_code>from paddlenlp.embeddings import TokenEmbedding
import paddlenlp
paddlenlp.embeddings.list_embedding_name()<jupyter_output><empty_output><jupyter_text>认识一下Embedding**`TokenEmbedding.search()`**获得指定词汇的词向量。<jupyter_code>test_token_embedding = token_embedding.search("中国")
print(test_token_embedding)<jupyter_output>[[ 0.245299 -0.063509  0.213156 -0.429391  0.825422 -0.119011  0.02137
   0.076543  0.085277  0.492101 -0.050074  0.035595  0.509381  0.179038
   0.456816  0.108855  0.041408 -0.153931  0.129657 -0.565065 -0.335973
  -0.515457  0.396934  0.96231  -0.265134  0.164116 -0.725352 -0.539651
   0.760359 -0.151472  0.506642  0.635705  0.648893 -0.183843 -0.131642
   0.649702  0.496052  0.159072 -0.06204  -0.512253 -0.295271  0.127087
  -0.288607 -0.292584 -0.016321 -0.586327 -0.114622  0.00445  -0.068454
  -0.230694  0.510607  0.191056 -0.35957   0.287977 -0.031446  0.468448
  -0.016571 -0.764257 -0.014768 -0.600075  0.472969 -0.027864 -0.406401
   0.575796  0.381489 -0.385134 -0.342893  0.127585 -0.649147  0.057982
   0.379719  0.335534 -0.504159 -0.258878 -0.094977  0.002911  0.134969
   0.804434 -0.657511 -0.011848  0.491697  0.186991 -0.310367 -0.203108
  -0.284629 -0.065764 -0.19618   0.245264  0.813726 -0.534868  0.143717
  -0.348675 -0.391534 -0.074511 -0.207123  0.833282 -0.147526  0.[...]<jupyter_text>**`TokenEmbedding.cosine_sim()`**计算词向量间余弦相似度，语义相近的词语余弦相似度更高，说明预训练好的词向量空间有很好的语义表示能力。<jupyter_code>score1 = token_embedding.cosine_sim("女孩", "女人")
score2 = token_embedding.cosine_sim("女孩", "书籍")
print('score1:', score1)
print('score2:', score2)<jupyter_output>score1: 0.61954707
score2: 0.19051214<jupyter_text>词向量映射到低维空间使用深度学习可视化工具[VisualDL](https://github.com/PaddlePaddle/VisualDL)的[High Dimensional](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/components/README_CN.mdHigh-Dimensional--%E6%95%B0%E6%8D%AE%E9%99%8D%E7%BB%B4%E7%BB%84%E4%BB%B6)组件可以对embedding结果进行可视化展示，便于对其直观分析，步骤如下：1. 升级 VisualDL 最新版本。`pip install --upgrade visualdl`2. 创建LogWriter并将记录词向量。3. 点击左侧面板中的可视化tab，选择‘token_hidi’作为文件并启动VisualDL可视化<jupyter_code>!pip install --upgrade visualdl
# 获取词表中前2000个单词
labels = token_embedding.vocab.to_tokens(list(range(0, 2000)))

# 取出这2000个单词对应的Embedding
test_token_embedding = token_embedding.search(labels)

# 引入VisualDL的LogWriter记录日志
from visualdl import LogWriter

with LogWriter(logdir='./token_hidi') as writer:
    writer.add_embeddings(tag='0-2000', mat=[i for i in test_token_embedding], metadata=labels)<jupyter_output><empty_output><jupyter_text>**效果如图：****T-SNE降维方法的结果**![](https://ai-studio-static-online.cdn.bcebos.com/b153ffcb0d9f4164a84727ee192a026e6c77da1679604c68812913b07663d6d5)**UMAP方法的结果**![](https://ai-studio-static-online.cdn.bcebos.com/6ad823e5b4d844428df56a6196fac57081fceeddb52e49c7a22c6b687dddb3d6)**PCA方法的结果**![](https://ai-studio-static-online.cdn.bcebos.com/83dc95ef8a9f4497ba46358b57ece92b62d9cb0088f34b55a5b0f24e1a5fdd53)可以看出来，UMAP的直观效果最好 数据降维的过程中：这里顺便查了一下TSNE，PCA，UMAP算法的异同参考了一篇文章：[Dimensionality Reduction for Data Visualization: PCA vs TSNE vs UMAP](https://towardsdatascience.com/dimensionality-reduction-for-data-visualization-pca-vs-tsne-vs-umap-be4aa7b1cb29)可以看原文的[机翻](http://www.weainfo.net/news/detail/453570)降维的两种主要方法：投影和流形学习。投影：这种技术处理将每个高维的数据点投影到子空间适合的低维空间上，从而近似地保持点之间的距离。流形学习：许多降维算法通过对训练实例所在的流形进行建模来工作；这称为流形学习。它依赖于流形假设或假设，它认为大多数真实世界的高维数据集都接近一个低维的流形，这种假设在大多数情况下是基于观察或经验，而不是理论或纯逻辑。>PCA主成分分析：>>The axis that explains the maximum amount of variance in the training set is called the Principal Components.>>The axis orthogonal to this axis is called the second principal component. As we go for higher dimensions, PCA would find a third component orthogonal to the other two components and so on, for visualization purposes we always stick to 2 or maximum 3 principal components.>>这段解释了什么是主成分以及过程：find a third component orthogonal to the other two components>>缺点：极值和非线性：The main drawback of PCA is that it is highly influenced by outliers present in the data. Moreover, PCA is a linear projection, which means it can’t capture non-linear dependencies.>t-SNE(T-distributed stochastic neighbour embedding)>>(t-SNE) takes a high dimensional data set and reduces it to a low dimensional graph that retains a lot of the original information. It does so by giving each data point a location in a two or three-dimensional map. This technique finds clusters in data thereby making sure that an embedding preserves the meaning in the data. t-SNE reduces dimensionality while trying to keep similar instances close and dissimilar instances apart.>>[详细解释](https://www.oreilly.com/content/an-illustrated-introduction-to-the-t-sne-algorithm/)>>原来的数据点$x_i,x_j$,映射的对应点$y_i,y_j$,$x$之间的距离：$p_{j|i} = \frac{\exp\left(-\left| x_i – x_j\right|^2 \big/ 2\sigma_i^2\right)}{\displaystyle\sum_{k \neq i} \exp\left(-\left| x_i – x_k\right|^2 \big/ 2\sigma_i^2\right)}$,得到相似矩阵$p_{ij} = \frac{p_{j|i} + p_{i|j}}{2N}$>>映射点的相似矩阵：$q_{ij} = \frac{f(\left| x_i – x_j\right|)}{\displaystyle\sum_{k \neq i} f(\left| x_i – x_k\right|)} \quad \textrm{with} \quad f(z) = \frac{1}{1+z^2}$>>这里有个物理意义的解释：Let’s assume that our map points are all connected with springs. The stiffness of a spring connecting points  and  depends on the mismatch between the similarity of the two data points and the similarity of the two map points, that is, . Now, we let the system evolve according to the laws of physics. If two map points are far apart while the data points are close, they are attracted together. If they are nearby while the data points are dissimilar, they are repelled.>>也就是说把这些点当作用弹簧连起来的，如果一侧距离近就会把另一侧带着拉近>>距离之间的KL散度：$KL(P||Q) = \sum_{i, j} p_{ij} \, \log \frac{p_{ij}}{q_{ij}}.$,求对$y_i$的偏导：$\frac{\partial \, KL(P || Q)}{\partial y_i} = 4 \sum_j (p_{ij} – q_{ij}) g\left( \left| x_i – x_j\right| \right) u_{ij} \quad \textrm{where} \, g(z) = \frac{z}{1+z^2}.$,使用梯度下降法就能得到结果，VDL中也可以修改学习率>UMAP(Uniform Manifold Approximation and Projection)>>具有非线性，算力消耗比tsne少的特点>>算力消耗少的原因： 稀疏矩阵：it can be applied directly to sparse matrices thereby eliminating the need to applying any Dimensionality reduction such as PCA or Truncated SVD(Singular Value Decomposition) as a prior pre-processing step>  基于TokenEmbedding衡量句子语义相似度在许多实际应用场景（如文档检索系统）中， 需要衡量两个句子的语义相似程度。此时我们可以使用词袋模型（Bag of Words，简称BoW）计算句子的语义向量。**首先**，将两个句子分别进行切词，并在TokenEmbedding中查找相应的单词词向量（word embdding）。**然后**，根据词袋模型，将句子的word embedding叠加作为句子向量（sentence embedding）。**最后**，计算两个句子向量的余弦相似度。 基于TokenEmbedding的词袋模型使用`BoWEncoder`搭建一个BoW模型用于计算句子语义。* `paddlenlp.TokenEmbedding`组建word-embedding层* `paddlenlp.seq2vec.BoWEncoder`组建句子建模层<jupyter_code>import paddle
import paddle.nn as nn
import paddlenlp


class BoWModel(nn.Layer):
    def __init__(self, embedder):
        super().__init__()
        self.embedder = embedder
        emb_dim = self.embedder.embedding_dim
        self.encoder = paddlenlp.seq2vec.BoWEncoder(emb_dim)
        self.cos_sim_func = nn.CosineSimilarity(axis=-1)

    def get_cos_sim(self, text_a, text_b):
        text_a_embedding = self.forward(text_a)
        text_b_embedding = self.forward(text_b)
        cos_sim = self.cos_sim_func(text_a_embedding, text_b_embedding)
        return cos_sim

    def forward(self, text):
        # Shape: (batch_size, num_tokens, embedding_dim)
        embedded_text = self.embedder(text)

        # Shape: (batch_size, embedding_dim)
        summed = self.encoder(embedded_text)

        return summed

model = BoWModel(embedder=token_embedding)<jupyter_output><empty_output><jupyter_text>构造Tokenizer使用TokenEmbedding词表构造Tokenizer。<jupyter_code>from data import Tokenizer
tokenizer = Tokenizer()
tokenizer.set_vocab(vocab=token_embedding.vocab)<jupyter_output><empty_output><jupyter_text>相似句对数据读取以提供的样例数据text_pair.txt为例，该数据文件每行包含两个句子。<jupyter_code>text_pairs = {}
with open("text_pair.txt", "r", encoding="utf8") as f:
    for line in f:
        text_a, text_b = line.strip().split("\t")
        if text_a not in text_pairs:
            text_pairs[text_a] = []
        text_pairs[text_a].append(text_b)<jupyter_output><empty_output><jupyter_text>查看相似语句相关度<jupyter_code>for text_a, text_b_list in text_pairs.items():
    text_a_ids = paddle.to_tensor([tokenizer.text_to_ids(text_a)])

    for text_b in text_b_list:
        text_b_ids = paddle.to_tensor([tokenizer.text_to_ids(text_b)])
        print("text_a: {}".format(text_a))
        print("text_b: {}".format(text_b))
        print("cosine_sim: {}".format(model.get_cos_sim(text_a_ids, text_b_ids).numpy()[0]))
        print()<jupyter_output>text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: 多项式矩阵的左共轭积及其应用
cosine_sim: 0.8273115754127502

text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: 退化阻尼对高维可压缩欧拉方程组经典解的影响
cosine_sim: 0.7478786110877991

text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: Burgers方程基于特征正交分解方法的数值解法研究
cosine_sim: 0.7302016019821167

text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: 有界对称域上解析函数空间的若干性质
cosine_sim: 0.6930968165397644

text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: 基于卷积神经网络的图像复杂度研究与应用
cosine_sim: 0.6603596806526184

text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: Cartesian发射机中线性功率放大器的研究
cosine_sim: 0.7091397643089294

text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: CFRP加固WF型梁侧扭屈曲的几何非线性有限元分析
cosine_sim: 0.7540827989578247

text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: 基于线性CCD自适应成像的光刻机平台调平方法研究
cosine_sim: 0.7331011295318604

text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: 基于变分贝叶斯理论的图像复原方法研究
cosine_sim: 0.7067816853523254

text_a: 多项式矩阵左共轭积对偶Sylvester共轭和数学算子完备参数解
text_b: 网格资源分配中混合并行蚁群算法方式研究
cosine[...]<jupyter_text>使用VisualDL查看句子向量<jupyter_code># 引入VisualDL的LogWriter记录日志
import numpy as np
from visualdl import LogWriter    
# 获取句子以及其对应的向量
label_list = []
embedding_list = []

for text_a, text_b_list in text_pairs.items():
    text_a_ids = paddle.to_tensor([tokenizer.text_to_ids(text_a)])
    embedding_list.append(model(text_a_ids).flatten().numpy())
    label_list.append(text_a)

    for text_b in text_b_list:
        text_b_ids = paddle.to_tensor([tokenizer.text_to_ids(text_b)])
        embedding_list.append(model(text_b_ids).flatten().numpy())
        label_list.append(text_b)


with LogWriter(logdir='./sentence_hidi') as writer:
    writer.add_embeddings(tag='句子', mat=embedding_list, metadata=label_list)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11704896697576053, 'fraction_numerical': 0.09687652513421181, 'mean_word_length': 6.0337528604119, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 71, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '198033', 'license': 'Apache-2.0', 'n_tokens_mistral': 6678, 'n_tokens_neox': 6420, 'n_words': 773, 'path': 'PaddleNLP_learn/day1.ipynb', 'repo_name': 'NexusXi/Paddle_Learn', 'size': '12294'}