<commit_before>h1. Panako - Benchmark

To verify the computational efficiency of Panako there is a benchmark script available. The script measures runtimes to store audio and to query the index. The main aim is to see how performance is affected when the index gets larger.

Note that correctness is not checked with this script. Please have a look at the evaluation script for correctness.

The script needs access to the "FMA-Medium dataset":https://github.com/mdeff/fma/. The FMA medium set contains 25, tracks of 30s spread over 16 unbalanced genres (22 GiB)

The evaluation script does the following: starting from an empty database first 2 audio files are stored. Next, the database size is doubled a couple of times until 16k files are stored (or about 130 hours of audio). The time it takes to double the database size (indexing audio) is measured and, after a doubling in size, the database is queried with a number of 30s fragments. The time it takes to query the database is also reported.

h2. Benchmark results

The following results are measured on an Apple M1 Pro with 8 cores. The speeds are reported in number seconds of audio that can be processed per wall clock second (s/s). For example, if a store speed of 750 is reported then 750 seconds of audio are indexed in a single second.

The absolute values are not that relevant. These might differ significantly from one machine to another. However, the fact that store speed and query speed stay in the same range when the index grows is relevant. This limited effect of index size is expected to be similar on all machines.

<div align="center">
!./olaf_results.svg(Benchmark results for Olaf)!
<small>Fig. Benchmark results for Olaf.</small>
</div>


<div align="center">
!./panako_results.svg(Benchmark results for Panako)!
<small>Fig. Benchmark results for Panako.</small>
</div><commit_msg>benchmark<commit_after>h1. Panako - Benchmark

To verify the computational efficiency of Panako there is a benchmark script available. The script measures runtimes to store audio and to query the index. The main aim is to see how performance is affected when the index gets larger.

Note that correctness is not checked with this script. Please have a look at the evaluation script for correctness.

The script needs access to the "FMA-Medium dataset":https://github.com/mdeff/fma/. The FMA medium set contains 25, tracks of 30s spread over 16 unbalanced genres (22 GiB)

The evaluation script does the following: starting from an empty database first 2 audio files are stored. Next, the database size is doubled a couple of times until 16k files are stored (or about 130 hours of audio). The time it takes to double the database size (indexing audio) is measured and, after a doubling in size, the database is queried with a number of 30s fragments. The time it takes to query the database is also reported.

h2. Benchmark results

The following results are measured on an Apple M1 Pro with 8 cores. The speeds are reported in number seconds of audio that can be processed per wall clock second (s/s). For example, if a store speed of 750 is reported then 750 seconds of audio are indexed in a single second.

The absolute values are not that relevant. These might differ significantly from one machine to another. However, the fact that store speed and query speed stay in the same range when the index grows is relevant. This limited effect of index size is expected to be similar on all machines.

<div align="center">
!./olaf_benchmark_results.svg(Benchmark results for Olaf)!
<small>Fig. Benchmark results for Olaf.</small>
</div>


<div align="center">
!./panako_benchmark_results.svg(Benchmark results for Panako)!
<small>Fig. Benchmark results for Panako.</small>
</div>
{'fraction_non_alphanumeric': 0.05024051309460182, 'fraction_numerical': 0.01389631213254944, 'mean_word_length': 4.416787264833575, 'pattern_counts': {'":': 2, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '665896', 'n_tokens_mistral': 985, 'n_tokens_neox': 904, 'n_words': 585}