<jupyter_start><jupyter_text>Demonstration of a simple evaluation pipeline Polara supports various evaluation regimes and can be tuned flexibly to achieve the setup you need. Note, that particular evaluation settings may not be directly supported by some models and may require a certain model modification.For example, matrix factorization models are not directly applicable in the *warm-start scenario* (when test users are not present in the training set) until the *folding-in* technique is implemented for generating recommendations. Keep this in mind when creating your custom solutions.  In this example we demonstrate basic evaluation of two well-known factorization models - `PureSVD` and `iALS` - in both standard and warm-start settings.The models can be easily tested in both scenarios without any modification due to automatic handling of the warm-start case, provided by Polara.  Preparing Movielens-1M data<jupyter_code>from polara.recommender.data import RecommenderData
from polara.datasets.movielens import get_movielens_data
data = get_movielens_data() # will automatically download it, or you can specify a path to the local copy
data.head()
data.shape
data_model = RecommenderData(data, 'userid', 'movieid', 'rating', seed=0)
data_model.get_configuration()<jupyter_output><empty_output><jupyter_text>Standard scenario with known users<jupyter_code>data_model.random_holdout = True # allow not only top-rated items in evaluation, this reduces evaluation biases
data_model.warm_start = False # standard case
data_model.prepare()<jupyter_output>Preparing data...
Done.
There are 996585 events in the training and 3624 events in the holdout.<jupyter_text>Let's check for demonstration purposes that all test users are present in the training set:<jupyter_code>data_model.test.holdout['userid'].isin(data_model.index.userid.training.new).all()<jupyter_output><empty_output><jupyter_text>PureSVD<jupyter_code>from polara.recommender.models import SVDModel
svd = SVDModel(data_model) # create model
svd.switch_positive = 4 # mark ratings below 4 as negative feedback and treat them accordingly in evaluation
svd.build() # fit model
svd.evaluate() # by default it calculates the total number of hits<jupyter_output>PureSVD training time: 0.149s<jupyter_text>iALS<jupyter_code># implicit library must be installed separately, follow instructions at https://github.com/benfred/implicit 
from polara.recommender.external.implicit.ialswrapper import ImplicitALS
als = ImplicitALS(data_model) # create model
als.switch_positive = 4 # same as for PureSVD, affects only evaluation
als.build()
als.evaluate()<jupyter_output>WARNING:root:Intel MKL BLAS detected. Its highly recommend to set the environment variable 'export MKL_NUM_THREADS=1' to disable its internal multithreading<jupyter_text>The maximum possible number of correct recomendations is:<jupyter_code>data_model.test.holdout.query('rating>=4').shape[0]<jupyter_output><empty_output><jupyter_text>Both models correctly retrieve around a quarter of all items. Let's look on the averaged relevance scores:<jupyter_code>svd.evaluate('relevance')
als.evaluate('relevance')<jupyter_output><empty_output><jupyter_text>Warm-start scenario This will split test users from the training data.<jupyter_code>data_model.warm_start = True # warm-start case
data_model.prepare()<jupyter_output>Preparing data...
19 unique movieid's within 26 testset interactions were filtered. Reason: not in the training data.
1 unique movieid's within 1 holdout interactions were filtered. Reason: not in the training data.
1 of 1208 userid's were filtered out from holdout. Reason: incompatible number of items.
1 userid's were filtered out from testset. Reason: inconsistent with holdout.
Done.
There are 807458 events in the training and 3621 events in the holdout.<jupyter_text>There's no intersection between test and training users:<jupyter_code>data_model.index.userid.test.old.isin(data_model.index.userid.training.old).any()<jupyter_output><empty_output><jupyter_text>Polara makes a certain level of efforts to preserve data sanity and consistency.For example, as can be seen from the log message above, it filters out the items that are happen to be in the test split but are not a part of the training set.  PureSVD<jupyter_code>svd.build()
svd.evaluate()<jupyter_output>PureSVD training time: 0.107s<jupyter_text>Note, that you do not have to recreate the models as they operate on top of the `data_model` instance.  In fact, the state change in `data_model` is synchronized with the dependent models' states. It will force models to rebuild themselves even if you do not explicitly specify it (even though it is recommended to be explicit to conform with the Zen of Python).  iALS<jupyter_code>als.evaluate()<jupyter_output>WARNING:root:Intel MKL BLAS detected. Its highly recommend to set the environment variable 'export MKL_NUM_THREADS=1' to disable its internal multithreading<jupyter_text>The maximum possible number of correct recomendations is:<jupyter_code>data_model.test.holdout.query('rating>=4').shape[0]<jupyter_output><empty_output><jupyter_text>Check relevance scores:<jupyter_code>svd.evaluate('relevance')
als.evaluate('relevance')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06978967495219886, 'fraction_numerical': 0.009751434034416827, 'mean_word_length': 6.62536443148688, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '571871', 'license': 'MIT', 'n_tokens_mistral': 1564, 'n_tokens_neox': 1468, 'n_words': 551, 'path': 'examples/Warm_start_and_standard_scenarios.ipynb', 'repo_name': 'almirb/polara', 'size': '5230'}