<jupyter_start><jupyter_text>The Rekall Auto-TunerRekall queries often have a lot of "magic numbers" that need to be tuned to get the best performance. The Rekall [tuner package](https://rekallpy.readthedocs.io/en/latest/source/rekall.tuner.html) provides a number of tuning algorithms that will help you select the best parameters for your query given some ground truth annotations.This notebook will help introduce you to the tuner package's API, and walk you through the tuning of a query for the parking space detection task. The Tuner APIHere's an example of what a call to the tuner API looks like:```Pythonfrom rekall.tuner import RandomTunerdef eval_config(params):     Run the Rekall query    query_results = query(params)     Evaluate the results    score = evaluate(query_results)    return scoresearch_space = {    'param1': [0.0, 1.0, 2.0],           discrete    'param2': { 'range': (10.0, 20.0) }  linear range}tuner = RandomTuner(search_space, eval_config, budget = 50)best_score, best_config, score_history, execution_times, total_cost = tuner.tune()```You define a search space of parameters to search through, and give it a function that runs the Rekall query and returns a score to optimize (we assume you want to maximize the score).You pick a tuner (we picked `RandomTuner` in this example) and give it a budget, and the tuning algorithm will search through the search space and return the best score, as well as the best parameter configuration it found.Here's a list of tuners that we provide as of this writing (see the [documentation](https://rekallpy.readthedocs.io/en/latest/source/rekall.tuner.html) for details and for the most up-to-date list):* `RandomTuner`* `GridTuner`* `CoordinateDescentTuner`* `SuccessiveHalvingTuner`* `HyperbandTuner`Anecdotally, we have observed that running the `CoordinateDescentTuner` from a few random initializations, or running the `HyperbandTuner` with the `CoordinateDescentTuner` in the inner loop tend to work the best. An example: tuning an empty parking space detection queryLet's tune an empty parking space detection query as an example. We'll load up some ground truth annotations (some tips on collecting those at the bottom of this notebook), put together a simple query with some magic numbers, and see how well we can do. Load Data<jupyter_code>from empty_parking_space_tutorial_helpers import *

bboxes = get_maskrcnn_bboxes()<jupyter_output>100%|██████████| 2/2 [00:00<00:00, 39.14it/s]<jupyter_text>Load Ground TruthLoad up some ground truth annotations<jupyter_code>ground_truth_ism = get_ground_truth()<jupyter_output>100%|██████████| 2/2 [00:00<00:00, 520.58it/s]<jupyter_text>Parking Spaces QueryHere's an example parking spaces query, with some magic numbers.<jupyter_code>def parking_spaces_query(bboxes_ism, params={}):
    magic_numbers = {
        'EMPTY_SPOT_IOU': 0.25,
        'COALESCE_IOU': 0.5,
        'MIN_TIME': 30
    }
    
    magic_numbers.update(params)
    
    EMPTY_SPOT_IOU = magic_numbers['EMPTY_SPOT_IOU']
    COALESCE_IOU = magic_numbers['COALESCE_IOU']
    MIN_TIME = magic_numbers['MIN_TIME']
    
    first_key = sorted(list(bboxes_ism.keys()))[0]
    parking_spot_candidates = bboxes_ism[first_key].filter(
        lambda intrvl: intrvl['t1'] == 0.0 and intrvl['payload']['class'] == 'car'
    )
    
    video_durations = bboxes_ism.coalesce(
        axis=('t1', 't2'),
        bounds_merge_op=Bounds3D.span
    )
    
    parking_spots = IntervalSetMapping({
        key: IntervalSet([
            Interval(Bounds3D(
                t1 = t,
                t2 = t + 30,
                x1 = parking_spot['x1'],
                x2 = parking_spot['x2'],
                y1 = parking_spot['y1'],
                y2 = parking_spot['y2']
            ))
            for parking_spot in parking_spot_candidates.get_intervals()
            for t in range(0, int(video_durations[key].get_intervals()[0]['t2']), 30)
        ])
        for key in video_durations
    })
    
    empty_parking_spots = parking_spots.minus(
        bboxes_ism.filter(
            lambda intrvl: intrvl['payload']['class'] in ['car', 'truck']
        ),
        predicate = and_pred(
            Bounds3D.T(overlaps()),
            Bounds3D.X(overlaps()),
            Bounds3D.Y(overlaps()),
            iou_at_least(EMPTY_SPOT_IOU)
        ),
        window=0.0
    )
    
    # Get rid of predictions that are empty for less than four minutes
    empty_spots_coalesced_and_filtered = empty_parking_spots.map(
        lambda intrvl: Interval(intrvl['bounds'], [intrvl])
    ).coalesce(
        ('t1', 't2'),
        bounds_merge_op = Bounds3D.span,
        payload_merge_op = lambda p1, p2: p1 + p2,
        predicate = iou_at_least(COALESCE_IOU)
    ).filter_size(min_size=MIN_TIME).split(
        lambda intrvl: IntervalSet(intrvl['payload'])
    )
    
    return empty_spots_coalesced_and_filtered<jupyter_output><empty_output><jupyter_text>Visualize Example Results(this will take a few seconds)<jupyter_code>parking_spaces_result = parking_spaces_query(bboxes)
visualize_helper([
    parking_spaces_result,
    ground_truth_ism
])<jupyter_output><empty_output><jupyter_text>Computing Average PrecisionThis is a function that computes average precision based on the ground truth (you may need to install scikit-learn in your environment to run this).<jupyter_code>from sklearn.metrics import average_precision_score
import numpy as np

def evaluate_preds(predictions, gt):
    true_positives = predictions.filter_against(
        gt,
        predicate = and_pred(
            Bounds3D.T(equal()),
            iou_at_least(0.5)
        ),
        window = 0.0
    )
    false_positives = predictions.minus(
        true_positives,
        predicate = and_pred(
            Bounds3D.T(equal()),
            iou_at_least(0.5)
        ),
        window = 0.0
    )
    false_negatives = gt.minus(
        predictions,
        predicate = and_pred(
            Bounds3D.T(equal()),
            iou_at_least(0.5)
        ),
        window = 0.0
    )
    
    tp_count = sum(true_positives.size().values())
    fp_count = sum(false_positives.size().values())
    fn_count = sum(false_negatives.size().values())
    
    y_true = np.concatenate([
        np.ones(tp_count),
        np.zeros(fp_count),
        np.zeros(1000 * sum(gt.size().values())),
        np.ones(fn_count),
    ])
    y_scores = np.concatenate([
        np.ones(tp_count),
        np.ones(fp_count),
        np.ones(1000 * sum(gt.size().values())) - 0.1,
        np.zeros(fn_count),
    ])
    
    return average_precision_score(y_true, y_scores)<jupyter_output><empty_output><jupyter_text>Now let's run it (this will take a few seconds).<jupyter_code>evaluate_preds(parking_spaces_result, ground_truth_ism)<jupyter_output><empty_output><jupyter_text>Tuning the QueryTo tune the query, you need to:* Define a search space over query parameters* Define a single function that will take in a configuration from the search space (a set of values) and evaluate the query results with those parameters  Defining the search spaceExample of a search space:```Pythonsearch_space = {    'param1': [0.0, 1.0, 2.0],           discrete    'param2': { 'range': (10.0, 20.0) }  linear range}```We'll define three continuous parameters for our query:<jupyter_code>search_space = {
    'EMPTY_SPOT_IOU': { 'range': [0.1, 0.9] },
    'COALESCE_IOU': { 'range': [0.1, 0.9] },
    'MIN_TIME': { 'range': [10, 500] }
}<jupyter_output><empty_output><jupyter_text>Evaluating a configurationThis function will take a config, run the query with those parameters, and return the average precision score. The config will be of this format:```Python{    'EMPTY_SPOT_IOU': value,    'COALESCE_IOU': value,    'MIN_TIME': value}```Luckily, we wrote our query to take exactly this kind of input in the params argument.<jupyter_code>def evaluate_config(config):
    query_results = parking_spaces_query(bboxes, params = config)
    
    ap_score = evaluate_preds(query_results, ground_truth_ism)
    
    return ap_score<jupyter_output><empty_output><jupyter_text>Tuning our query!Let's tune our query with a few tuning algorithms!<jupyter_code>from rekall.tuner import *
random_tuner = RandomTuner(search_space, evaluate_config, budget = 10)
(best_score, best_config, scores,
     execution_times, cost) = random_tuner.tune()<jupyter_output>100%|██████████| 10/10 [02:34<00:00, 15.47s/it]<jupyter_text>Let's see how the best configs improved during our tuning session:<jupyter_code>import matplotlib.pyplot as plt

def get_best_so_far(scores):
    best_so_far = []
    best = scores[0]
    
    for score in scores:
        if score > best:
            best = score
        best_so_far.append(best)
    
    return best_so_far

plt.plot(get_best_so_far(scores))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10207553589656346, 'fraction_numerical': 0.01916751729613247, 'mean_word_length': 3.5641821946169774, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '400756', 'license': 'Apache-2.0', 'n_tokens_mistral': 3016, 'n_tokens_neox': 2834, 'n_words': 754, 'path': 'tutorials/04 The Rekall Auto-Tuner.ipynb', 'repo_name': 'lsheiba/rekall', 'size': '8817'}