<jupyter_start><jupyter_text>Benchmarks consist of a target assembly and a metric to compare assemblies.They accept a source assembly to compare against and yield a score.  Pre-defined benchmarks Brainscore defines benchmarks, which can be run on brain models. To implement a model, the BrainModel interface has to be implemented by the model to be tested. A very simple implementation could look like this:<jupyter_code>import numpy as np
from typing import List, Tuple
from brainscore.benchmarks.screen import place_on_screen

from brainscore.model_interface import BrainModel
from brainio_base.assemblies import DataAssembly

class RandomITModel(BrainModel):
    def __init__(self):
        self._num_neurons = 50
        # to note which time we are recording
        self._time_bin_start = None
        self._time_bin_end = None
    
    def look_at(self, stimuli, **kwargs):
        print(f"Looking at {len(stimuli)} stimuli")
        rnd = np.random.RandomState(0)
        recordings = DataAssembly(rnd.rand(len(stimuli), self._num_neurons, 1),
                              coords={'image_id': ('presentation', stimuli['image_id']),
                                      'object_name': ('presentation', stimuli['object_name']),
                                      'neuroid_id': ('neuroid', np.arange(self._num_neurons)),
                                      'region': ('neuroid', ['IT'] * self._num_neurons),
                                      'time_bin_start': ('time_bin', [self._time_bin_start]),
                                      'time_bin_end': ('time_bin', [self._time_bin_end])},
                              dims=['presentation', 'neuroid', 'time_bin'])
        recordings.name = 'random_it_model'
        return recordings
    
    def start_task(self, task, **kwargs):
        print(f"Starting task {task}")
        if task != BrainModel.Task.passive:
            raise NotImplementedError()

    def start_recording(self, recording_target=BrainModel.RecordingTarget, time_bins=List[Tuple[int]]):
        print(f"Recording from {recording_target} during {time_bins} ms")
        if str(recording_target) != "IT":
            raise NotImplementedError(f"RandomITModel only supports IT, not {recording_target}")
        if len(time_bins) != 1:
            raise NotImplementedError(f"RandomITModel only supports a single start-end time-bin, not {time_bins}")
        time_bins = time_bins[0].tolist()
        self._time_bin_start, self._time_bin_end = time_bins[0], time_bins[1]
    
    def visual_degrees(self):
        print("Declaring model to have a visual field size of 8 degrees")
        return 8

model = RandomITModel()<jupyter_output>Loading lookup from /braintree/home/msch/brainio_collection/brainio_collection/lookup.csv<jupyter_text>The implementation maps a given brain region to a neural network layer. In the look_at method, the class just creates a mock result and returns it. The other two methods only check for correctness of the input values. The following lines load the public benchmark `MajajHong2015public.IT-pls`,consisting of neural recordings in macaque IT from `<NAME> et al. 2015`and a neural predictivity metric based on PLS regression to compare between model predictions and actual data. Running the benchmark with the `RandomITModel` then returns a score of the model's brain-likeness under this particular benchmark.<jupyter_code>from brainscore import score_model
score = score_model(model_identifier='mymodel', model=model, benchmark_identifier='dicarlo.MajajHong2015public.IT-pls')
print(score)<jupyter_output><xarray.Score (aggregation: 2)>
array([1.329276e-05, 4.042570e-03])
Coordinates:
  * aggregation  (aggregation) <U6 'center' 'error'
Attributes:
    raw:                   <xarray.Score (aggregation: 2)>\narray([-0.002974,...
    ceiling:               <xarray.Score (aggregation: 2)>\narray([0.815799, ...
    model_identifier:      mymodel
    benchmark_identifier:  dicarlo.MajajHong2015public.IT-pls<jupyter_text>The benchmark (1) recorded from the model as its response to 2560 stimuli, (2) applied the neural predictivity metric to compare the predicted model recordings with the actual primate recordings to yield a score, (3) normalized the score by the ceiling.Since the benchmark already cross-validated results, the resulting score now contains the center (i.e. the average of the splits, in this case the mean) and the error (in this case standard-error-of-the-mean).<jupyter_code>center, error = score.sel(aggregation='center'), score.sel(aggregation='error')
print(f"score: {center.values:.3f}+-{error.values:.3f}")<jupyter_output>score: 0.000+-0.004<jupyter_text>The score tells us that random features don't predict IT recordings well.We can also check the raw unceiled values...<jupyter_code>unceiled_scores = score.raw
print(unceiled_scores)<jupyter_output><xarray.Score (aggregation: 2)>
array([-0.002974,  0.004043])
Coordinates:
  * aggregation  (aggregation) <U6 'center' 'error'
Attributes:
    raw:      <xarray.Score (split: 10, neuroid: 168)>\narray([[-0.070138, -0...<jupyter_text>...as well as the per-neuroid, per-split correlations.<jupyter_code>raw_scores = score.raw.raw
print(raw_scores)<jupyter_output><xarray.Score (split: 10, neuroid: 168)>
array([[-0.070138, -0.135265, -0.003982, ...,  0.012797,  0.004866,  0.059825],
       [-0.024563,  0.026108,  0.036809, ...,  0.055484,  0.088264,  0.040734],
       [-0.036944, -0.008812,  0.034396, ..., -0.005293,  0.032865,  0.060247],
       ...,
       [-0.024136, -0.004739,  0.012494, ..., -0.04916 ,  0.02889 , -0.041967],
       [ 0.004991, -0.038394, -0.008638, ...,  0.095637,  0.101677,  0.046107],
       [-0.022872, -0.02758 ,  0.043983, ...,  0.049692,  0.033474, -0.0456  ]])
Coordinates:
  * split       (split) int64 0 1 2 3 4 5 6 7 8 9
  * neuroid     (neuroid) MultiIndex
  - neuroid_id  (neuroid) object 'Chabo_L_A_2_4' ... 'Tito_L_M_9_8'
  - arr         (neuroid) object 'A' 'A' 'A' 'A' 'A' 'A' ... 'M' 'M' 'M' 'M' 'M'
  - col         (neuroid) int64 4 3 5 0 1 2 3 4 5 6 2 ... 4 5 6 7 8 1 3 4 5 7 8
  - hemisphere  (neuroid) object 'L' 'L' 'L' 'L' 'L' 'L' ... 'L' 'L' 'L' 'L' 'L'
  - subregion   (neuroid) object 'cIT' 'cIT' 'aIT' 'cIT'[...]<jupyter_text>Custom benchmarks We can also define our own benchmarks.The benchmark fulfills two purposes:1. reproduce the primate experiment on the model2. apply a similarity metric to compare predictions with actual measurements3. normalize the match with the ceiling, i.e. an upper bound on how well a model could doThe following example implements a simple benchmark that show-cases these three steps.<jupyter_code>import numpy as np
import brainscore
from brainscore.benchmarks import Benchmark
from brainscore.metrics.regression import CrossRegressedCorrelation, pls_regression, pearsonr_correlation
from brainscore.metrics.ceiling import InternalConsistency
from brainscore.benchmarks._neural_common import explained_variance, average_repetition

# Let's say, we want to test the model's match to IT recordings between 100-120 ms.
# We'll use the same Majaj et al. 2015 data from primates passively fixating.

class MyBenchmark(Benchmark):
    def __init__(self):
        # both StimulusSets as well as assemblies are packaged through https://github.com/brain-score/brainio_contrib
        assembly = brainscore.get_assembly('dicarlo.MajajHong2015.temporal.public')  # this will take a while to download and open
        assembly = assembly[{'time_bin': [start == 100 for start in assembly['time_bin_start'].values]}]
        # also, let's only look at a subset of the images
        image_ids = np.unique(assembly['image_id'].values)[:1000]
        assembly = assembly.loc[{'presentation': [image_id in image_ids for image_id in assembly['image_id'].values]}]
        stimulus_set = assembly.stimulus_set  # assemblies always have a StimulusSet attached to them
        stimulus_set = stimulus_set[stimulus_set['image_id'].isin(image_ids)]
        assembly.attrs['stimulus_set'] = stimulus_set
        # reduce to presentation x neuroid for simplicity (we only have one time_bin)
        assembly = assembly.squeeze('time_bin')
        self._assembly = assembly  # note that this assembly still has repetitions which we need for the ceiling
        self._similarity_metric = CrossRegressedCorrelation(
                                       regression=pls_regression(), correlation=pearsonr_correlation(),
                                       crossvalidation_kwargs=dict(splits=3, stratification_coord='object_name'))
        self._ceiler = InternalConsistency()
    
    @property
    def identifier(self):  # for storing results
        return "my-dummy-benchmark"
    
    def __call__(self, candidate: BrainModel):
        # since the candidate follows the BrainModel interface, we can easily treat all models the same way.
        # (1) reproduce the experiment on the model. 
        candidate.start_task(task=BrainModel.Task.passive)
        candidate.start_recording(recording_target="IT", time_bins=[np.array((100, 120))])
        # since different models can have different fields of view, we adjust the image sizes accordingly.
        # for instance, a stimulus of 2 degree should take up little space for a model with a field of view of 10 degree
        # while the same stimulus would take up much more space for a model of 4 degrees.
        stimulus_set = place_on_screen(self._assembly.stimulus_set, target_visual_degrees=candidate.visual_degrees(),
                                       # for reference, we know this experiment was run at 8 degrees for the primates.
                                       source_visual_degrees=8)
        predictions = candidate.look_at(stimuli=stimulus_set)
        # (2) compute similarity between predictions and measurements
        assembly = average_repetition(self._assembly)  # average over repetitions
        predictions = predictions.squeeze('time_bin')
        print("Computing model-match")
        unceiled_score = self._similarity_metric(predictions, assembly)
        # (3) normalize by our estimate of how well the ideal model could do
        ceiled_score = explained_variance(unceiled_score, self.ceiling)
        return ceiled_score
        
    
    @property
    def ceiling(self):
        print("Computing ceiling")
        return self._ceiler(self._assembly)

my_benchmark = MyBenchmark()
model = RandomITModel()  # we'll use the same model from before
score = my_benchmark(model)
print(score)<jupyter_output>Starting task Task.passive
Recording from IT during [array([100, 120])] ms
Declaring model to have a visual field size of 8 degrees
Looking at 1000 stimuli
Computing model-match
Computing ceiling
<xarray.Score (aggregation: 2)>
array([9.656137e-05, 1.310140e-02])
Coordinates:
  * aggregation  (aggregation) <U6 'center' 'error'
Attributes:
    raw:      <xarray.Score (aggregation: 2)>\narray([0.006589, 0.013101])\nC...
    ceiling:  <xarray.Score (aggregation: 2)>\narray([0.670578, 0.002014])\nC...
{'fraction_non_alphanumeric': 0.10030864197530864, 'fraction_numerical': 0.04547930283224401, 'mean_word_length': 3.34253054789121, 'pattern_counts': {'":': 1, '<': 32, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '41041', 'license': 'MIT', 'n_tokens_mistral': 3695, 'n_tokens_neox': 3264, 'n_words': 1008, 'path': 'examples/benchmarks.ipynb', 'repo_name': 'hermessuen/brain-score', 'size': '11021'}