title a transformerbased recommendation system khalid salamahttps www linkedin cominkhalidsalama24403144 date created 20201230 last modified 20231025 description rating rate prediction using the behavior sequence transformer bst model on the movielens accelerator gpu introduction this example demonstrates the behavior sequence transformer bsthttps arxiv orgabs1905 06874 model by qiwei chen et al using the movielens datasethttps grouplens orgdatasetsmovielens the bst model leverages the sequential behaviour of the users in watching and rating movies as well as user profile and movie features to predict the rating of the user to a target movie more precisely the bst model aims to predict the rating of a target movie by accepting the following inputs 1 a fixedlength sequence of movieids watched by a user 2 a fixedlength sequence of the ratings for the movies watched by a user 3 a set of user features including userid sex occupation and agegroup 4 a set of genres for each movie in the input sequence and the target movie 5 a targetmovieid for which to predict the rating this example modifies the original bst model in the following ways 1 we incorporate the movie features genres into the processing of the embedding of each movie of the input sequence and the target movie rather than treating them as other features outside the transformer layer 2 we utilize the ratings of movies in the input sequence along with the their positions in the sequence to update them before feeding them into the selfattention layer note that this example should be run with tensorflow 2 4 or higher the dataset we use the 1m version of the movielens datasethttps grouplens orgdatasetsmovielens1m the dataset includes around 1 million ratings from 6000 users on 4000 movies along with some user features movie genres in addition the timestamp of each usermovie rating is provided which allows creating sequences of movie ratings for each user as expected by the bst model setup prepare the data download and prepare the dataframes first let s download the movielens data the downloaded folder will contain three data files users dat movies dat and ratings dat then we load the data into pandas dataframes with their proper column names here we do some simple data processing to fix the data types of the columns each movie has multiple genres we split them into separate columns in the movies dataframe transform the movie ratings data into sequences first let s sort the the ratings data using the unixtimestamp and then group the movieid values and the rating values by userid the output dataframe will have a record for each userid with two ordered lists sorted by rating datetime the movies they have rated and their ratings of these movies now let s split the movieids list into a set of sequences of a fixed length we do the same for the ratings set the sequencelength variable to change the length of the input sequence to the model you can also change the stepsize to control the number of sequences to generate for each user after that we process the output to have each sequence in a separate records in the dataframe in addition we join the user features with the ratings data with sequencelength of 4 and stepsize of 2 we end up with 498 623 sequences finally we split the data into training and testing splits with 85 and 15 of the instances respectively and store them to csv files define metadata create tf data dataset for training and evaluation the last movie id in the sequence is the target movie the last rating in the sequence is the target for the model to predict create model inputs encode input features the encodeinputfeatures method works as follows 1 each categorical user feature is encoded using layers embedding with embedding dimension equals to the square root of the vocabulary size of the feature the embeddings of these features are concatenated to form a single input tensor 2 each movie in the movie sequence and the target movie is encoded layers embedding where the dimension size is the square root of the number of movies 3 a multihot genres vector for each movie is concatenated with its embedding vector and processed using a nonlinear layers dense to output a vector of the same movie embedding dimensions 4 a positional embedding is added to each movie embedding in the sequence and then multiplied by its rating from the ratings sequence 5 the target movie embedding is concatenated to the sequence movie embeddings producing a tensor with the shape of batch size sequence length embedding size as expected by the attention layer for the transformer architecture 6 the method returns a tuple of two elements encodedtransformerfeatures and encodedotherfeatures encode user features convert the string input values into integer indices compute embedding dimensions create an embedding layer with the specified dimensions convert the index values to embedding representations create a single embedding vector for the user features create a movie embedding encoder create a lookup to convert string values to integer indices create an embedding layer with the specified dimensions create a vector lookup for movie genres create a processing layer for genres define a function to encode a given movie id convert the string input values into integer indices encoding targetmovieid encoding sequence movieids create positional embedding retrieve sequence ratings to incorporate them into the encoding of the movie add the positional encoding to the movie encodings and multiply them by rating construct the transformer inputs create a bst model create a multiheaded attention layer transformer block included the other features fullyconnected layers run training and evaluation experiment compile the model read the training data fit the model with the training data read the test data evaluate the model on the test data you should achieve a mean absolute error mae at or around 0 7 on the test data conclusion the bst model uses the transformer layer in its architecture to capture the sequential signals underlying users behavior sequences for recommendation you can try training this model with different configurations for example by increasing the input sequence length and training the model for a larger number of epochs in addition you can try including other features like movie release year and customer zipcode and including cross features like sex x genre introduction this example demonstrates the behavior sequence transformer bst https arxiv org abs 1905 06874 model by qiwei chen et al using the movielens dataset https grouplens org datasets movielens the bst model leverages the sequential behaviour of the users in watching and rating movies as well as user profile and movie features to predict the rating of the user to a target movie more precisely the bst model aims to predict the rating of a target movie by accepting the following inputs 1 a fixed length sequence of movie_ids watched by a user 2 a fixed length sequence of the ratings for the movies watched by a user 3 a set of user features including user_id sex occupation and age_group 4 a set of genres for each movie in the input sequence and the target movie 5 a target_movie_id for which to predict the rating this example modifies the original bst model in the following ways 1 we incorporate the movie features genres into the processing of the embedding of each movie of the input sequence and the target movie rather than treating them as other features outside the transformer layer 2 we utilize the ratings of movies in the input sequence along with the their positions in the sequence to update them before feeding them into the self attention layer note that this example should be run with tensorflow 2 4 or higher the dataset we use the 1m version of the movielens dataset https grouplens org datasets movielens 1m the dataset includes around 1 million ratings from 6000 users on 4000 movies along with some user features movie genres in addition the timestamp of each user movie rating is provided which allows creating sequences of movie ratings for each user as expected by the bst model setup prepare the data download and prepare the dataframes first let s download the movielens data the downloaded folder will contain three data files users dat movies dat and ratings dat then we load the data into pandas dataframes with their proper column names here we do some simple data processing to fix the data types of the columns each movie has multiple genres we split them into separate columns in the movies dataframe transform the movie ratings data into sequences first let s sort the the ratings data using the unix_timestamp and then group the movie_id values and the rating values by user_id the output dataframe will have a record for each user_id with two ordered lists sorted by rating datetime the movies they have rated and their ratings of these movies now let s split the movie_ids list into a set of sequences of a fixed length we do the same for the ratings set the sequence_length variable to change the length of the input sequence to the model you can also change the step_size to control the number of sequences to generate for each user after that we process the output to have each sequence in a separate records in the dataframe in addition we join the user features with the ratings data with sequence_length of 4 and step_size of 2 we end up with 498 623 sequences finally we split the data into training and testing splits with 85 and 15 of the instances respectively and store them to csv files define metadata create tf data dataset for training and evaluation the last movie id in the sequence is the target movie the last rating in the sequence is the target for the model to predict create model inputs encode input features the encode_input_features method works as follows 1 each categorical user feature is encoded using layers embedding with embedding dimension equals to the square root of the vocabulary size of the feature the embeddings of these features are concatenated to form a single input tensor 2 each movie in the movie sequence and the target movie is encoded layers embedding where the dimension size is the square root of the number of movies 3 a multi hot genres vector for each movie is concatenated with its embedding vector and processed using a non linear layers dense to output a vector of the same movie embedding dimensions 4 a positional embedding is added to each movie embedding in the sequence and then multiplied by its rating from the ratings sequence 5 the target movie embedding is concatenated to the sequence movie embeddings producing a tensor with the shape of batch size sequence length embedding size as expected by the attention layer for the transformer architecture 6 the method returns a tuple of two elements encoded_transformer_features and encoded_other_features encode user features convert the string input values into integer indices compute embedding dimensions create an embedding layer with the specified dimensions convert the index values to embedding representations create a single embedding vector for the user features create a movie embedding encoder create a lookup to convert string values to integer indices create an embedding layer with the specified dimensions create a vector lookup for movie genres create a processing layer for genres define a function to encode a given movie id convert the string input values into integer indices encoding target_movie_id encoding sequence movie_ids create positional embedding retrieve sequence ratings to incorporate them into the encoding of the movie add the positional encoding to the movie encodings and multiply them by rating construct the transformer inputs create a bst model create a multi headed attention layer transformer block included the other features fully connected layers run training and evaluation experiment compile the model read the training data fit the model with the training data read the test data evaluate the model on the test data you should achieve a mean absolute error mae at or around 0 7 on the test data conclusion the bst model uses the transformer layer in its architecture to capture the sequential signals underlying users behavior sequences for recommendation you can try training this model with different configurations for example by increasing the input sequence length and training the model for a larger number of epochs in addition you can try including other features like movie release year and customer zipcode and including cross features like sex x genre
import os
import math
from zipfile import ZipFile
from urllib.request import urlretrieve
import numpy as np
import pandas as pd
import tensorflow as tf
import keras
from keras import layers
from keras.layers import StringLookup




urlretrieve("http://files.grouplens.org/datasets/movielens/ml-1m.zip", "movielens.zip")
ZipFile("movielens.zip", "r").extractall()



users = pd.read_csv(
    "ml-1m/users.dat",
    sep="::",
    names=["user_id", "sex", "age_group", "occupation", "zip_code"],
)

ratings = pd.read_csv(
    "ml-1m/ratings.dat",
    sep="::",
    names=["user_id", "movie_id", "rating", "unix_timestamp"],
)

movies = pd.read_csv(
    "ml-1m/movies.dat", sep="::", names=["movie_id", "title", "genres"], encoding='latin-1'
)



users["user_id"] = users["user_id"].apply(lambda x: f"user_{x}")
users["age_group"] = users["age_group"].apply(lambda x: f"group_{x}")
users["occupation"] = users["occupation"].apply(lambda x: f"occupation_{x}")

movies["movie_id"] = movies["movie_id"].apply(lambda x: f"movie_{x}")

ratings["movie_id"] = ratings["movie_id"].apply(lambda x: f"movie_{x}")
ratings["user_id"] = ratings["user_id"].apply(lambda x: f"user_{x}")
ratings["rating"] = ratings["rating"].apply(lambda x: float(x))



genres = [
    "Action",
    "Adventure",
    "Animation",
    "Children's",
    "Comedy",
    "Crime",
    "Documentary",
    "Drama",
    "Fantasy",
    "Film-Noir",
    "Horror",
    "Musical",
    "Mystery",
    "Romance",
    "Sci-Fi",
    "Thriller",
    "War",
    "Western",
]

for genre in genres:
    movies[genre] = movies["genres"].apply(
        lambda values: int(genre in values.split("|"))
    )




ratings_group = ratings.sort_values(by=["unix_timestamp"]).groupby("user_id")

ratings_data = pd.DataFrame(
    data={
        "user_id": list(ratings_group.groups.keys()),
        "movie_ids": list(ratings_group.movie_id.apply(list)),
        "ratings": list(ratings_group.rating.apply(list)),
        "timestamps": list(ratings_group.unix_timestamp.apply(list)),
    }
)




sequence_length = 4
step_size = 2


def create_sequences(values, window_size, step_size):
    sequences = []
    start_index = 0
    while True:
        end_index = start_index + window_size
        seq = values[start_index:end_index]
        if len(seq) < window_size:
            seq = values[-window_size:]
            if len(seq) == window_size:
                sequences.append(seq)
            break
        sequences.append(seq)
        start_index += step_size
    return sequences


ratings_data.movie_ids = ratings_data.movie_ids.apply(
    lambda ids: create_sequences(ids, sequence_length, step_size)
)

ratings_data.ratings = ratings_data.ratings.apply(
    lambda ids: create_sequences(ids, sequence_length, step_size)
)

del ratings_data["timestamps"]



ratings_data_movies = ratings_data[["user_id", "movie_ids"]].explode(
    "movie_ids", ignore_index=True
)
ratings_data_rating = ratings_data[["ratings"]].explode("ratings", ignore_index=True)
ratings_data_transformed = pd.concat([ratings_data_movies, ratings_data_rating], axis=1)
ratings_data_transformed = ratings_data_transformed.join(
    users.set_index("user_id"), on="user_id"
)
ratings_data_transformed.movie_ids = ratings_data_transformed.movie_ids.apply(
    lambda x: ",".join(x)
)
ratings_data_transformed.ratings = ratings_data_transformed.ratings.apply(
    lambda x: ",".join([str(v) for v in x])
)

del ratings_data_transformed["zip_code"]

ratings_data_transformed.rename(
    columns={"movie_ids": "sequence_movie_ids", "ratings": "sequence_ratings"},
    inplace=True,
)



random_selection = np.random.rand(len(ratings_data_transformed.index)) <= 0.85
train_data = ratings_data_transformed[random_selection]
test_data = ratings_data_transformed[~random_selection]

train_data.to_csv("train_data.csv", index=False, sep="|", header=False)
test_data.to_csv("test_data.csv", index=False, sep="|", header=False)



CSV_HEADER = list(ratings_data_transformed.columns)

CATEGORICAL_FEATURES_WITH_VOCABULARY = {
    "user_id": list(users.user_id.unique()),
    "movie_id": list(movies.movie_id.unique()),
    "sex": list(users.sex.unique()),
    "age_group": list(users.age_group.unique()),
    "occupation": list(users.occupation.unique()),
}

USER_FEATURES = ["sex", "age_group", "occupation"]

MOVIE_FEATURES = ["genres"]




def get_dataset_from_csv(csv_file_path, shuffle=False, batch_size=128):
    def process(features):
        movie_ids_string = features["sequence_movie_ids"]
        sequence_movie_ids = tf.strings.split(movie_ids_string, ",").to_tensor()

        
        features["target_movie_id"] = sequence_movie_ids[:, -1]
        features["sequence_movie_ids"] = sequence_movie_ids[:, :-1]

        ratings_string = features["sequence_ratings"]
        sequence_ratings = tf.strings.to_number(
            tf.strings.split(ratings_string, ","), tf.dtypes.float32
        ).to_tensor()

        
        target = sequence_ratings[:, -1]
        features["sequence_ratings"] = sequence_ratings[:, :-1]

        return features, target

    dataset = tf.data.experimental.make_csv_dataset(
        csv_file_path,
        batch_size=batch_size,
        column_names=CSV_HEADER,
        num_epochs=1,
        header=False,
        field_delim="|",
        shuffle=shuffle,
    ).map(process)

    return dataset





def create_model_inputs():
    return {
        "user_id": layers.Input(name="user_id", shape=(1,), dtype="string"),
        "sequence_movie_ids": layers.Input(
            name="sequence_movie_ids", shape=(sequence_length - 1,), dtype="string"
        ),
        "target_movie_id": layers.Input(
            name="target_movie_id", shape=(1,), dtype="string"
        ),
        "sequence_ratings": layers.Input(
            name="sequence_ratings", shape=(sequence_length - 1,), dtype="float32"
        ),
        "sex": layers.Input(name="sex", shape=(1,), dtype="string"),
        "age_group": layers.Input(name="age_group", shape=(1,), dtype="string"),
        "occupation": layers.Input(name="occupation", shape=(1,), dtype="string"),
    }





def encode_input_features(
    inputs,
    include_user_id=True,
    include_user_features=True,
    include_movie_features=True,
):
    encoded_transformer_features = []
    encoded_other_features = []

    other_feature_names = []
    if include_user_id:
        other_feature_names.append("user_id")
    if include_user_features:
        other_feature_names.extend(USER_FEATURES)

    
    for feature_name in other_feature_names:
        
        vocabulary = CATEGORICAL_FEATURES_WITH_VOCABULARY[feature_name]
        idx = StringLookup(vocabulary=vocabulary, mask_token=None, num_oov_indices=0)(
            inputs[feature_name]
        )
        
        embedding_dims = int(math.sqrt(len(vocabulary)))
        
        embedding_encoder = layers.Embedding(
            input_dim=len(vocabulary),
            output_dim=embedding_dims,
            name=f"{feature_name}_embedding",
        )
        
        encoded_other_features.append(embedding_encoder(idx))

    
    if len(encoded_other_features) > 1:
        encoded_other_features = layers.concatenate(encoded_other_features)
    elif len(encoded_other_features) == 1:
        encoded_other_features = encoded_other_features[0]
    else:
        encoded_other_features = None

    
    movie_vocabulary = CATEGORICAL_FEATURES_WITH_VOCABULARY["movie_id"]
    movie_embedding_dims = int(math.sqrt(len(movie_vocabulary)))
    
    movie_index_lookup = StringLookup(
        vocabulary=movie_vocabulary,
        mask_token=None,
        num_oov_indices=0,
        name="movie_index_lookup",
    )
    
    movie_embedding_encoder = layers.Embedding(
        input_dim=len(movie_vocabulary),
        output_dim=movie_embedding_dims,
        name=f"movie_embedding",
    )
    
    genre_vectors = movies[genres].to_numpy()
    movie_genres_lookup = layers.Embedding(
        input_dim=genre_vectors.shape[0],
        output_dim=genre_vectors.shape[1],
        embeddings_initializer=keras.initializers.Constant(genre_vectors),
        trainable=False,
        name="genres_vector",
    )
    
    movie_embedding_processor = layers.Dense(
        units=movie_embedding_dims,
        activation="relu",
        name="process_movie_embedding_with_genres",
    )

    
    def encode_movie(movie_id):
        
        movie_idx = movie_index_lookup(movie_id)
        movie_embedding = movie_embedding_encoder(movie_idx)
        encoded_movie = movie_embedding
        if include_movie_features:
            movie_genres_vector = movie_genres_lookup(movie_idx)
            encoded_movie = movie_embedding_processor(
                layers.concatenate([movie_embedding, movie_genres_vector])
            )
        return encoded_movie

    
    target_movie_id = inputs["target_movie_id"]
    encoded_target_movie = encode_movie(target_movie_id)

    
    sequence_movies_ids = inputs["sequence_movie_ids"]
    encoded_sequence_movies = encode_movie(sequence_movies_ids)
    
    position_embedding_encoder = layers.Embedding(
        input_dim=sequence_length,
        output_dim=movie_embedding_dims,
        name="position_embedding",
    )
    positions = keras.ops.arange(start=0, stop=sequence_length - 1, step=1)
    encodded_positions = position_embedding_encoder(positions)
    
    sequence_ratings = keras.ops.expand_dims(inputs["sequence_ratings"], -1)
    
    encoded_sequence_movies_with_poistion_and_rating = layers.Multiply()(
        [(encoded_sequence_movies + encodded_positions), sequence_ratings]
    )

    
    for encoded_movie in keras.ops.unstack(
        encoded_sequence_movies_with_poistion_and_rating, axis=1
    ):
        encoded_transformer_features.append(keras.ops.expand_dims(encoded_movie, 1))
    encoded_transformer_features.append(encoded_target_movie)

    encoded_transformer_features = layers.concatenate(
        encoded_transformer_features, axis=1
    )

    return encoded_transformer_features, encoded_other_features




include_user_id = False
include_user_features = False
include_movie_features = False

hidden_units = [256, 128]
dropout_rate = 0.1
num_heads = 3


def create_model():
    inputs = create_model_inputs()
    transformer_features, other_features = encode_input_features(
        inputs, include_user_id, include_user_features, include_movie_features
    )

    
    attention_output = layers.MultiHeadAttention(
        num_heads=num_heads, key_dim=transformer_features.shape[2], dropout=dropout_rate
    )(transformer_features, transformer_features)

    
    attention_output = layers.Dropout(dropout_rate)(attention_output)
    x1 = layers.Add()([transformer_features, attention_output])
    x1 = layers.LayerNormalization()(x1)
    x2 = layers.LeakyReLU()(x1)
    x2 = layers.Dense(units=x2.shape[-1])(x2)
    x2 = layers.Dropout(dropout_rate)(x2)
    transformer_features = layers.Add()([x1, x2])
    transformer_features = layers.LayerNormalization()(transformer_features)
    features = layers.Flatten()(transformer_features)

    
    if other_features is not None:
        features = layers.concatenate(
            [features, layers.Reshape([other_features.shape[-1]])(other_features)]
        )

    
    for num_units in hidden_units:
        features = layers.Dense(num_units)(features)
        features = layers.BatchNormalization()(features)
        features = layers.LeakyReLU()(features)
        features = layers.Dropout(dropout_rate)(features)

    outputs = layers.Dense(units=1)(features)
    model = keras.Model(inputs=inputs, outputs=outputs)
    return model


model = create_model()




model.compile(
    optimizer=keras.optimizers.Adagrad(learning_rate=0.01),
    loss=keras.losses.MeanSquaredError(),
    metrics=[keras.metrics.MeanAbsoluteError()],
)


train_dataset = get_dataset_from_csv("train_data.csv", shuffle=True, batch_size=265)


model.fit(train_dataset, epochs=5)


test_dataset = get_dataset_from_csv("test_data.csv", batch_size=265)


_, rmse = model.evaluate(test_dataset, verbose=0)
print(f"Test MAE: {round(rmse, 3)}")