task_categories:multiple-choice task_categories:question-answering task_ids:multiple-choice-qa task_ids:open-domain-qa annotations_creators:expert-generated language_creators:crowdsourced multilinguality:monolingual size_categories:n<1K source_datasets:original language:en license:apache-2.0 movie-recommendation collaborative-filtering movielens film doi:10.57967/hf/0257 region:us
Movie recommendation task based on the Movielens dataset
@InProceedings{sileodreclm22,
author="Sileo, Damien
and Vossen, Wout
and Raymaekers, Robbe",
editor="Hagen, Matthias
and Verberne, Suzan
and Macdonald, Craig
and Seifert, Christin
and Balog, Krisztian
and N{\o}rv{\aa}g, Kjetil
and Setty, Vinay",
title="Zero-Shot Recommendation as Language Modeling",
booktitle="Advances in Information Retrieval",
year="2022",
publisher="Springer International Publishing",
address="Cham",
pages="223--230",
abstract="Recommendation is the task of ranking items (e.g. movies or products) according to individual user needs. Current systems rely on collaborative filtering and content-based techniques, which both require structured training data. We propose a framework for recommendation with off-the-shelf pretrained language models (LM) that only used unstructured text corpora as training data. If a user u liked Matrix and Inception, we construct a textual prompt, e.g. ``Movies like Matrix, Inception, {\$}{\$}{\{}<{\}}m{\{}>{\}}{\$}{\$}<m>'' to estimate the affinity between u and m with LM likelihood. We motivate our idea with a corpus analysis, evaluate several prompt structures, and we compare LM-based recommendation with standard matrix factorization trained on different data regimes. The code for our experiments is publicly available (https://colab.research.google.com/drive/...?usp=sharing).",
isbn="978-3-030-99739-7"
}
---
annotations_creators:
- expert-generated
language_creators:
- crowdsourced
language:
- en
license:
- apache-2.0
multilinguality:
- monolingual
pretty_name: 'movie_recommendation'
size_categories:
- n<1K
source_datasets:
- original
task_categories:
- multiple-choice
- question-answering
task_ids:
- multiple-choice-qa
- open-domain-qa
tags:
- movie-recommendation
- collaborative-filtering
- movielens
- film
---


# Dataset for evaluation of (zero-shot) recommendation with language models

We showed that pretrained large language models can act as a recommender system, and compare few-shot learning results to matrix factorization baselines.
This is the BIG-Bench version of our language-based movie recommendation dataset.

<https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/movie_recommendation>

GPT-2 has a 48.8% accuracy, chance is 25%.
Human accuracy is 60.4%.

# Citation
```
@InProceedings{sileodreclm22,
    author="Sileo, Damien
    and Vossen, Wout
    and Raymaekers, Robbe",
    editor="Hagen, Matthias
    and Verberne, Suzan
    and Macdonald, Craig
    and Seifert, Christin
    and Balog, Krisztian
    and N{\o}rv{\aa}g, Kjetil
    and Setty, Vinay",
    title="Zero-Shot Recommendation as Language Modeling",
    booktitle="Advances in Information Retrieval",
    year="2022",
    publisher="Springer International Publishing",
    address="Cham",
    pages="223--230",
    isbn="978-3-030-99739-7"
}
```