DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation
Recently, one critical issue looms large in the field of recommender systems
-- there are no effective benchmarks for rigorous evaluation -- which
consequently leads to unreproducible evaluation and unfair comparison. We,
therefore, conduct studies from the perspectives of practical theory and
experiments, aiming at benchmarking recommendation for rigorous evaluation.
Regarding the theoretical study, a series of hyper-factors affecting
recommendation performance throughout the whole evaluation chain are
systematically summarized and analyzed via an exhaustive review on 141 papers
published at eight top-tier conferences within 2017-2020. We then classify them
into model-independent and model-dependent hyper-factors, and different modes
of rigorous evaluation are defined and discussed in-depth accordingly. For the
experimental study, we release DaisyRec 2.0 library by integrating these
hyper-factors to perform rigorous evaluation, whereby a holistic empirical
study is conducted to unveil the impacts of different hyper-factors on
recommendation performance. Supported by the theoretical and experimental
studies, we finally create benchmarks for rigorous evaluation by proposing
standardized procedures and providing performance of ten state-of-the-arts
across six evaluation metrics on six datasets as a reference for later study.
Overall, our work sheds light on the issues in recommendation evaluation,
provides potential solutions for rigorous evaluation, and lays foundation for
further investigation.