<jupyter_start><jupyter_text>Create your own metricsThis Notebook illustrates how to create your own metrics. At the end of this guide, you'll be able to add new metrics to existing models. For a guide on how to create a new model, please see `advanced-models.ipynb`. In what follows, we assume you are familiar with the main concepts of the framework shown in `complete-guide.ipynb`.Custom metrics allow you to measure any quantity of interest at every timestep of the simulation. For example, at a high level, you might want to track how the diversity of recommendations, user utility, or item popularity evolve over the duration of the simulation. Metrics allow you to do all of this.  Concrete exampleThe `measure()` method is called before the simulation begins and at the end of each timestep. If the metric has no meaning before the simulation starts, you can simply return `None` from the `measure` method. The argument to the `measure()` method is always the `recommender` model (an instance of `trecs.models.BaseRecmomender`).At the end of the `measure()` method, you should call the `self.observe(value_of_metric)` to actually record the value of the metric.<jupyter_code>from trecs.metrics import Measurement
import numpy as np

class SampleMetric(Measurement):
    def __init__(self, name="sample_metric", verbose=False):
        Measurement.__init__(self, name, verbose)

    def measure(self, recommender):
        """
        The purpose of this metric is to calculate the average cosine similarity
        between every user profile's and the item the user interacted with. 

        Parameters
        ------------
            recommender: :class:`~models.recommender.BaseRecommender`
                Model that inherits from
                :class:`~models.recommender.BaseRecommender`.
        """
        similarity = 0
        interactions = recommender.interactions
        if interactions.size == 0:
            self.observe(None) # no interactions yet
            return
        # interacted items in shape |I| x num_attributes
        interacted_items = recommender.predicted_item_attributes[:, interactions].T
        item_norm = np.linalg.norm(interacted_items, axis=1)
        # user profiles in shape |U| x num_attributes
        user_profs = recommender.predicted_user_profiles
        user_norm = np.linalg.norm(user_profs, axis=1)
        # calculate mean cosine similarity between each user and their item
        sim_vals = (interacted_items * user_profs).sum(axis=1) / (item_norm * user_norm)
        
        # to complete the measurement, call `self.observe(metric_value)`
        self.observe(sim_vals.mean())<jupyter_output><empty_output><jupyter_text>Incorporating your metric into a simulationTo add your metric into a simulation, you can pass an instance of your `Measurement` object into the `metrics` argument when initializing a new simulation.<jupyter_code>from trecs.models import ContentFiltering
import pandas as pd

content_sim = ContentFiltering(num_users=100, num_items=500, measurements=[SampleMetric()])
content_sim.run(timesteps=10)<jupyter_output>100%|██████████| 10/10 [00:01<00:00,  8.67it/s]<jupyter_text>Now we can visualize the results of the metric by calling `get_measurements()` on our simulation object. This returns a dictionary that maps the name of the metric to a list of its values. (The `__init__` method of the `Measurement` class defines the default name of our metric, which in this case was `sample_metric`). The dictionary also additionally contains a key-value pair for the timesteps of the simulation.(Pro-tip: use `pd.DataFrame` to visualize the metric alongside the timesteps!) Note that timestep 0 corresponds to the initial state before the simulation starts. The value of our metric is `None` for timestep 0, since no users have any interactions yet.<jupyter_code>results = content_sim.get_measurements()
pd.DataFrame({'sample_metric': results['sample_metric'], 'timesteps': results['timesteps']})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0762902019446522, 'fraction_numerical': 0.008227374719521317, 'mean_word_length': 4.104325699745547, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '342751', 'license': 'MIT', 'n_tokens_mistral': 1101, 'n_tokens_neox': 1050, 'n_words': 443, 'path': 'examples/advanced-metrics.ipynb', 'repo_name': 'claramaine/t-recs', 'size': '4011'}