<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License.  DKN : Deep Knowledge-Aware Network for News RecommendationDKN \[1\] is a deep learning model which incorporates information from knowledge graph for better news recommendation. Specifically, DKN uses TransX \[2\] method for knowledge graph representation learning, then applies a CNN framework, named KCNN, to combine entity embedding with word embedding and generate a final embedding vector for a news article. CTR prediction is made via an attention-based neural scorer.  Properties of DKN:- DKN is a content-based deep model for CTR prediction rather than traditional ID-based collaborative filtering. - It makes use of knowledge entities and common sense in news content via joint learning from semantic-level and knowledge-level representations of news articles.- DKN uses an attention module to dynamically calculate a user's aggregated historical representaition. Data format: DKN takes several files as input as follows:- **training / validation / test files**: each line in these files represents one instance. Impressionid is used to evaluate performance within an impression session, so it is only used when evaluating, you can set it to 0 for training data. The format is :  `[label] [userid] [CandidateNews]%[impressionid] ` e.g., `1 train_U1 N1%0`  - **user history file**: each line in this file represents a users' click history. You need to set `history_size` parameter in the config file, which is the max number of user's click history we use. We will automatically keep the last `history_size` number of user click history, if user's click history is more than `history_size`, and we will automatically pad with 0 if user's click history is less than `history_size`. the format is :  `[Userid] [newsid1,newsid2...]`e.g., `train_U1 N1,N2`  - **document feature file**: It contains the word and entity features for news articles. News articles are represented by aligned title words and title entities. To take a quick example, a news title may be: "Trump to deliver State of the Union address next week", then the title words value may be `CandidateNews:34,45,334,23,12,987,3456,111,456,432` and the title entitie value may be: `entity:45,0,0,0,0,0,0,0,0,0`. Only the first value of entity vector is non-zero due to the word "Trump". The title value and entity value is hashed from 1 to `n` (where `n` is the number of distinct words or entities). Each feature length should be fixed at k (`doc_size` parameter), if the number of words in document is more than k, you should truncate the document to k words, and if the number of words in document is less than k, you should pad 0 to the end. the format is like:  `[Newsid] [w1,w2,w3...wk] [e1,e2,e3...ek]`- **word embedding/entity embedding/ context embedding files**: These are `*.npy` files of pretrained embeddings. After loading, each file is a `[n+1,k]` two-dimensional matrix, n is the number of words(or entities) of their hash dictionary, k is dimension of the embedding, note that we keep embedding 0 for zero padding. In this experiment, we used GloVe\[4\] vectors to initialize the word embedding. We trained entity embedding using TransE\[2\] on knowledge graph and context embedding is the average of the entity's neighbors in the knowledge graph. MIND datasetMIND dataset\[3\] is a large-scale English news dataset. It was collected from anonymized behavior logs of Microsoft News website. MIND contains 1,000,000 users, 161,013 news articles and 15,777,377 impression logs. Every news article contains rich textual content including title, abstract, body, category and entities. Each impression log contains the click events, non-clicked events and historical news click behaviors of this user before this impression.In this notebook we are going to use a subset of MIND dataset, **MIND demo**. MIND demo contains 500 users, 9,432 news articles  and 6,134 impression logs. For this quick start notebook, we are providing directly all the necessary word embeddings, entity embeddings and context embedding files.  Global settings and imports<jupyter_code>import sys
sys.path.append("../../")

import os
from tempfile import TemporaryDirectory
import papermill as pm
import tensorflow as tf

from reco_utils.recommender.deeprec.deeprec_utils import download_deeprec_resources, prepare_hparams
from reco_utils.recommender.deeprec.models.dkn import DKN
from reco_utils.recommender.deeprec.io.dkn_iterator import DKNTextIterator

print(f"System version: {sys.version}")
print(f"Tensorflow version: {tf.__version__}")<jupyter_output>System version: 3.6.11 | packaged by conda-forge | (default, Nov 27 2020, 18:57:37) 
[GCC 9.3.0]
Tensorflow version: 1.15.2<jupyter_text>Download and load data<jupyter_code>tmpdir = TemporaryDirectory()
data_path = os.path.join(tmpdir.name, "mind-demo-dkn")

yaml_file = os.path.join(data_path, r'dkn.yaml')
train_file = os.path.join(data_path, r'train_mind_demo.txt')
valid_file = os.path.join(data_path, r'valid_mind_demo.txt')
test_file = os.path.join(data_path, r'test_mind_demo.txt')
news_feature_file = os.path.join(data_path, r'doc_feature.txt')
user_history_file = os.path.join(data_path, r'user_history.txt')
wordEmb_file = os.path.join(data_path, r'word_embeddings_100.npy')
entityEmb_file = os.path.join(data_path, r'TransE_entity2vec_100.npy')
contextEmb_file = os.path.join(data_path, r'TransE_context2vec_100.npy')
if not os.path.exists(yaml_file):
    download_deeprec_resources(r'https://recodatasets.blob.core.windows.net/deeprec/', tmpdir.name, 'mind-demo-dkn.zip')<jupyter_output>100%|██████████| 11.3k/11.3k [00:03<00:00, 3.50kKB/s]<jupyter_text>Create hyper-parameters<jupyter_code>epochs = 10
history_size = 50
batch_size = 100
hparams = prepare_hparams(yaml_file,
                          news_feature_file = news_feature_file,
                          user_history_file = user_history_file,
                          wordEmb_file=wordEmb_file,
                          entityEmb_file=entityEmb_file,
                          contextEmb_file=contextEmb_file,
                          epochs=epochs,
                          history_size=history_size,
                          batch_size=batch_size)
print(hparams)<jupyter_output>WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

kg_file=None,user_clicks=None,FEATURE_COUNT=None,FIELD_COUNT=None,data_format=dkn,PAIR_NUM=None,DNN_FIELD_NUM=None,n_user=None,n_item=None,n_user_attr=None,n_item_attr=None,iterator_type=None,SUMMARIES_DIR=None,MODEL_DIR=None,wordEmb_file=/tmp/tmppqp68_ph/mind-demo-dkn/word_embeddings_100.npy,entityEmb_file=/tmp/tmppqp68_ph/mind-demo-dkn/TransE_entity2vec_100.npy,contextEmb_file=/tmp/tmppqp68_ph/mind-demo-dkn/TransE_context2vec_100.npy,news_feature_file=/tmp/tmppqp68_ph/mind-demo-dkn/doc_feature.txt,user_history_file=/tmp/tmppqp68_ph/mind-demo-dkn/user_history.txt,use_entity=True,use_context=True,doc_size=10,[...]<jupyter_text>Train the DKN model<jupyter_code>model = DKN(hparams, DKNTextIterator)
print(model.run_eval(valid_file))
model.fit(train_file, valid_file)<jupyter_output>at epoch 1
train info: logloss loss:0.6992590076337426
eval info: auc:0.5233, group_auc:0.4937, mean_mrr:0.1625, ndcg@10:0.2226, ndcg@5:0.16
at epoch 1 , train time: 9.5 eval time: 3.7
at epoch 2
train info: logloss loss:0.6215811174804882
eval info: auc:0.5266, group_auc:0.5233, mean_mrr:0.1735, ndcg@10:0.2307, ndcg@5:0.1802
at epoch 2 , train time: 8.7 eval time: 3.7
at epoch 3
train info: logloss loss:0.5888907697241185
eval info: auc:0.5285, group_auc:0.5271, mean_mrr:0.1774, ndcg@10:0.236, ndcg@5:0.1812
at epoch 3 , train time: 8.6 eval time: 3.7
at epoch 4
train info: logloss loss:0.5635578234569502
eval info: auc:0.5291, group_auc:0.5327, mean_mrr:0.1825, ndcg@10:0.2407, ndcg@5:0.1886
at epoch 4 , train time: 8.7 eval time: 3.7
at epoch 5
train info: logloss loss:0.5442607703350358
eval info: auc:0.5842, group_auc:0.5506, mean_mrr:0.1822, ndcg@10:0.2493, ndcg@5:0.1934
at epoch 5 , train time: 8.6 eval time: 3.7
at epoch 6
train info: logloss loss:0.5262203899733091
eval info: au[...]<jupyter_text>Evaluate the DKN modelNow we can check the performance on the test set:<jupyter_code>res = model.run_eval(test_file)
print(res)
pm.record("res", res)<jupyter_output>/home/ec2-user/anaconda3/envs/reco_full/lib/python3.6/site-packages/ipykernel/__main__.py:1: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.
  if __name__ == '__main__':
{'fraction_non_alphanumeric': 0.0935235759670048, 'fraction_numerical': 0.05261397837476313, 'mean_word_length': 4.965425531914893, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 16, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '368261', 'license': 'MIT', 'n_tokens_mistral': 3137, 'n_tokens_neox': 2841, 'n_words': 939, 'path': 'examples/00_quick_start/dkn_MIND.ipynb', 'repo_name': 'leemengtaiwan/recommenders', 'size': '8971'}