<jupyter_start><jupyter_text>Matrix Factorisation by Minimising the Top Push Loss  GoalLearn latent features of songs and playlists by factorising the song-playlist matrix and optimising the Top Push bipartite ranking loss (see [Method](Method) for details). MethodGiven a dataset with $M$ songs and $N$ playlists,let function $$f(m, n) = \mathbf{s}_m^\top \mathbf{p}_n + b_m, \ m \in \{1,\dots,M\}, \, n \in \{1,\dots,N\},$$where $\mathbf{s}_m \in \mathbb{R}^{D \times 1}$ ($D > 0$) is the feature vector of song $m$ ,$\mathbf{p}_n \in \mathbb{R}^{D \times 1}$ is the feature vector of playlist $n$,and $b_m$ is the bias of song $m$.Intuitively, $f(m, n)$ measures the affinity between song $m$ and playlist $n$.We minimising the number of songs appeared in the playlist but are scored lower than the *highest* scored song which is not appeared in the playlist,$$\min_{\mathbf{s}, \mathbf{p}, \mathbf{b}} \ \frac{1}{N} \sum_{n=1}^N \frac{1}{M_+^n}\sum_{m: y_m^n = 1} I\left( f(m, n) \le \max_{m': y_{m'}^n = 0} f(m', n) \right)$$where $M_+^n$ is the number of songs in playlist $n$, $y_m^n = 1$ means song $n$ appeared in playlist $n$ and $y_{m'}^n = 0$ means song $m'$ didn't appear in playlist $n$, $I(\cdot)$ is the indicator function.This is known as the *Top Push Loss* for bipartite ranking problem, which was proposed in [Li et al. Top Rank Optimization in Linear Time, NIPS'14](https://arxiv.org/abs/1410.1462) Practical concernsWe use the *squared hinge loss* $\ell(z) = [\max(0, 1-z)]^2$ as the convex surrogate of the 0-1 loss (i.e., the indicator function $I(\cdot)$) in the Top Push loss, and optimise the objective using mini-batch sub-gradient descent via tensorflow.<jupyter_code>import os
import sys
import gzip
import time
import numpy as np
import pickle as pkl
from scipy.sparse import isspmatrix_csr
import tensorflow as tf
RAND_SEED = 0
data_dir = 'data'
fmftrain = os.path.join(data_dir, 'mftrain_mpd.pkl.gz')<jupyter_output><empty_output><jupyter_text>Load the song-playlist matrix<jupyter_code>Y, _ = pkl.load(gzip.open(fmftrain, 'rb'))
assert isspmatrix_csr(Y)<jupyter_output><empty_output><jupyter_text>Specify the feature dimension, the number of training epochs and batch size<jupyter_code>feature_dim = 200
n_epochs = 65
batch_size = 64<jupyter_output><empty_output><jupyter_text>File to store the learned parameters<jupyter_code>fparam = os.path.join(data_dir, 'mf-%d-mpd.npy' % feature_dim)<jupyter_output><empty_output><jupyter_text>This model was trained using 4 Tesla P100-SXM2-16GB GPUs for 65 epochs, each epoch took about 40 minutes.<jupyter_code>N, M = Y.shape
D = feature_dim
n_batches = int((N - 1) / batch_size) + 1

# Inputs
# Note that the shape of some inputs are fixed so that
# the cost can be computed in a way supported by tensorflow
pIx = tf.placeholder(dtype=tf.int32, shape=[batch_size])     # playlist indices in current batch
posIx = tf.placeholder(dtype=tf.int32, shape=[None, 2])      # indices of positive entries in current batch
Mplus = tf.placeholder(dtype=tf.float32, shape=[batch_size]) # number of songs in each playlist
nPos = tf.placeholder(dtype=tf.int32)                        # total number of positive entries in current batch

# Parameters
tf.set_random_seed(RAND_SEED)
S = tf.Variable(initial_value=1e-3 * tf.random_uniform([D, M]))  # latent features of songs
b = tf.Variable(initial_value=1e-3 * tf.random_uniform([1, M]))  # latent features of playlists
P = tf.Variable(initial_value=1e-3 * tf.random_uniform([N, D]))  # biases of songs

# Compute cost
Pb = tf.gather(params=P, indices=pIx, axis=0)  # Nb by D, Nb = batch_size
T = tf.matmul(Pb, S) + b  # scores: Nb by M
pos_vec = tf.gather_nd(params=T, indices=posIx)  # scores of positive entries

Tn = T + tf.scatter_nd(shape=T.shape, indices=posIx, updates=tf.tile([-np.inf], [nPos]))  # mask positive entries
max_negs = tf.reduce_max(Tn, axis=1)  # element `n`: maximum score of songs not in playlist `n` in current batch

row_ix = posIx[:, 0]  # row indices of positive entries
relu_vec = tf.nn.relu(tf.divide(1 - pos_vec + tf.gather(max_negs, row_ix), tf.gather(Mplus, row_ix)))
cost = tf.reduce_sum(tf.multiply(relu_vec, relu_vec)) / tf.cast(T.shape[0], tf.float32)  # mean squared hinge loss

# optimiser
optimiser = tf.train.AdamOptimizer(learning_rate=1e-3).minimize(cost)

# train
init_op = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init_op)
    for epoch in range(n_epochs):
        rand_ix = np.random.permutation(N)
        for nb in range(n_batches):
            ix_start = nb * batch_size
            ix_end = min((nb + 1) * batch_size, N)
            indices = rand_ix[ix_start:ix_end]
            
            # make sure each batch is of the specified size
            if len(indices) < batch_size:
                indices = np.r_[indices, rand_ix[:batch_size - len(indices)]]

            Y_nb = Y[indices, :]
            Mplus_nb = Y_nb.sum(axis=1).A.reshape(-1)
            Y_coo = Y_nb.tocoo()

            _, J = sess.run([optimiser, cost],
                            feed_dict={pIx: indices,
                                       Mplus: Mplus_nb,
                                       nPos: Y_coo.row.shape[0],
                                       posIx: np.hstack([Y_coo.row.reshape(-1, 1), Y_coo.col.reshape(-1, 1)])})
            print('%d / %d / %d: %g' % (epoch + 1, nb + 1, n_batches, J))

    # save parameters
    w = np.r_[S.eval(), b.eval(), P.eval()]
    np.save(fparam, w, allow_pickle=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13036393264530147, 'fraction_numerical': 0.015028064457722253, 'mean_word_length': 3.6071726438698914, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '140155', 'license': 'Apache-2.0', 'n_tokens_mistral': 2019, 'n_tokens_neox': 1926, 'n_words': 570, 'path': 'src/Train.ipynb', 'repo_name': 'cdawei/playlist_recommendation', 'size': '5523'}