<jupyter_start><jupyter_text>**CogDL Notebook**created by CogDL Team[<EMAIL>]This notebook shows how to build applications based on CogDL pipeline. CogDL Link: https://github.com/THUDM/CogDLColab Link: https://colab.research.google.com/drive/1V47IIanXxDxi0Qsd6feOvvyYuqXcFP6P?usp=sharing **使用cogdl的pipeline API来搭建应用**--- 1. 通过pip install来安装最新的cogdl。同时安装用于可视化的seaborn库。Notes: 如果安装失败，请手动从https://github.com/THUDM/cogdl 下载源码进行安装。（记得点个star支持一下我们）<jupyter_code>!pip install git+https://github.com/THUDM/cogdl
!pip install seaborn<jupyter_output>Collecting git+https://github.com/THUDM/cogdl
  Cloning https://github.com/THUDM/cogdl to /tmp/pip-req-build-bd6quyfj
  Running command git clone -q https://github.com/THUDM/cogdl /tmp/pip-req-build-bd6quyfj
  Installing build dependencies ... [?25l[?25hdone
  Getting requirements to build wheel ... [?25l[?25hdone
    Preparing wheel metadata ... [?25l[?25hdone
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from cogdl==0.4.0) (1.4.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from cogdl==0.4.0) (1.19.5)
Requirement already satisfied: gensim<4.0 in /usr/local/lib/python3.7/dist-packages (from cogdl==0.4.0) (3.6.0)
Collecting emoji
  Downloading emoji-1.4.2.tar.gz (184 kB)
[K     |████████████████████████████████| 184 kB 5.0 MB/s 
[?25hRequirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from cogdl==0.4.0) (0.22.2.post1)
Collecting pre-commit
  Downloading pre_commit-2.14.0-py2[...]<jupyter_text>2. 从cogdl中加载Yelp2018数据集（训练数据保存在train_cf下）<jupyter_code>from cogdl.datasets import build_dataset_from_name

dataset = build_dataset_from_name("yelp2018")
data = dataset[0]<jupyter_output>No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
Failed to load C version of sampling, use python version instead.
Downloading https://cloud.tsinghua.edu.cn/d/ddbbff157971449eb163/files/?p=%2Fyelp2018.zip&dl=1
unpacking yelp2018.zip
Processing...
reading train and test user-item set ...
building the adj mat ...
loading done.
Done!<jupyter_text>3. 从原图中抽样一个小图：取前100000条交互数据，同时只保留编号小于3000的商品<jupyter_code>import numpy as np

train_data = data.train_cf[:100000].copy()
train_data = train_data[train_data[:, 1] < 3000]
print(train_data)

# 对user/item重新进行编号
users = list(set(train_data[:, 0]))
users.sort()
num_users = len(users)
user_map = dict(zip(users, list(range(num_users))))

items = list(set(train_data[:, 1]))
items.sort()
num_items = len(items)
item_map = dict(zip(items, list(range(num_items))))

for i in range(len(train_data)):
    train_data[i, 0] = user_map[train_data[i, 0]]
    train_data[i, 1] = item_map[train_data[i, 1]]

print("重标号后的用户/商品数目：", num_users, num_items)

train_data[:, 1] += num_users
train_data = np.concatenate((train_data, np.array([(y, x) for (x, y) in train_data])), axis=0)
print(train_data)<jupyter_output>[[   0    0]
 [   0    1]
 [   0    2]
 ...
 [1990 1683]
 [1990 2724]
 [1990  807]]
重标号后的用户/商品数目： 1949 3000
[[   0 1949]
 [   0 1950]
 [   0 1951]
 ...
 [3632 1948]
 [4673 1948]
 [2756 1948]]<jupyter_text>4. 通过cogdl的pipeline API来获取训练集中用户和商品的表示<jupyter_code>from cogdl import pipeline

# 使用cogdl的pipelie API来生成用户和商品的表示(可以选用prone模型)
generator = pipeline("generate-emb", model="prone")
embs = generator(train_data)

user_embs = embs[:num_users]
item_embs = embs[num_users:]
print(user_embs.shape, item_embs.shape)<jupyter_output>(1949, 128) (3000, 128)<jupyter_text>5. 将得到的用户/商品表示进行可视化<jupyter_code>from sklearn.manifold import TSNE 
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
 
# 使用TSNE将embs降到两维，并将结果保存在X_tsne中
tsne = TSNE(n_components=2) 
X_tsne = tsne.fit_transform(embs) 

y = np.array([0] * num_users + [1] * num_items)
X_tsne_data = np.vstack((X_tsne.T, y)).T 
df_tsne = pd.DataFrame(X_tsne_data, columns=['Dim1', 'Dim2', 'type']) 
df_tsne.head()

plt.figure(figsize=(8, 8))
sns.scatterplot(data=df_tsne, hue='type', x='Dim1', y='Dim2') 
plt.show()<jupyter_output><empty_output><jupyter_text>6. 得到用户和商品的表示之后搭建一个简易的推荐应用<jupyter_code>import numpy as np

def recommend(user):
    user_emb = user_embs[user].reshape((-1, 128))
    # 计算user_emb和item_embs之间的点积相似度，并保存在scores里
    # user_emb的shape为(1, 128), item_embs的shape为(3000, 128), 得到的scores的shape为(3000,)
    scores = np.multiply(user_emb, item_embs)
    scores = np.sum(scores, -1)

    idx = np.argsort(scores)[::-1] # 对scores进行从大到小排序
    print(idx[:20])                # 输出top 20的商品编号
    print(scores[idx[:20]])        # 输出top 20的分数

idx = 10
recommend(idx)                     # 给用户idx推荐商品<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1498371335504886, 'fraction_numerical': 0.05884907709011943, 'mean_word_length': 5.2075471698113205, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 22, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '177089', 'license': 'MIT', 'n_tokens_mistral': 2237, 'n_tokens_neox': 2073, 'n_words': 335, 'path': 'examples/notebooks/build_gnn_applications.ipynb', 'repo_name': 'jasmine-yu/cogdl', 'size': '4617'}