There is a movie recommendation task for language models in the BigBench framework based on a Movielens benchmark.https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/movie_recommendation/README.md However, this task is formulated specifically for a language model, in the form of a multiple choice task as opposed to a regression or matrix completion task. Our formulation of this task allows us to directly compare general purpose language models to special purpose systems on the same axis in order to assess a more realistic capability. Finally, we note that we can extend the analysis we do here to other economically valuable real-world tasks such as those in the recent Real-World Few-Shot Text-Classification (RAFT) benchmark [1]}.
