We experiment with multiple setups to evaluate algorithm performance:
1) Users Past/Future within One Dataset, a simple setup that uses the first 80% of every user's data as the training set, and the remaining 20% as the testing set in each DS.
2) Leave-One-Dataset-Out, a cross-dataset setup that uses three DS as the training set, and the other as the testing set.
3) Pre/Post-COVID, another cross-dataset setup to measure the effect of the pandemic, using DS1&2 (before COVID) as the training set and DS3&4 (after COVID) as the testing set, and then swapping the two sides.
4) Overlapping Users across Datasets, a cross-dataset setup that only focuses on overlapping users in multiple datasets to measure the time effect, which trains a model with overlapping users from one dataset, and tests it on overlapping users from other datasets.
We employ balanced accuracy (the average of sensitivity and specificity) as the metric, as it has been shown to be more robust to class-imbalance [1]}.
