<commit_before>BUGS
====

* BUG: "python perf timeit --compare-to=pypy" doesn't tune correctly the runner
  for pypy. pypy requires more warmup samples (10) than cpython (1).
* BUG: --duplicate of timeit must be ignored in PyPy, see the discussion
  on the speed mailing list.


TODO
====

* Implement a few system operations on macOS
* store metadata common to all benchmarks at the root in JSON
* --append should get the number of loops from the existing file and avoid
  calibration, at least when no JIT is used.
* Implement a system-wide lock to prevent running two benchmarks at the same
  time, especially using CPU pinning
* system: display uptime
* Need to write unit test for Runner.timeit()
* load(): remove '-' special case
* add stdev() method, maybe median_stdev()
* Calibration run: display time per iteration and total duration
* system:

  * advice: use --affinity when no CPU is isolated
  * set the CPU scaling governor when intel_pstate is not used.
    Use "userland" governor with a fixed CPU speed (max speed)?
  * warn if HyperThread is enabled?
  * check that isolated CPU "respect HyperThreading" (are part of the
    same physical cores)
  * check that rcu_nocbs respects isolated CPUs
  * detect NUMA

* compare: check also if benchmarks are unstable
* External tool (?) to produce nice HTML reports:

  * https://magic.io/blog/uvloop-blazing-fast-python-networking/

    - HTML, JS: d3 graphic library
    - min, max, std dev
    - https://github.com/MagicStack/pgbench
    - https://github.com/MagicStack/vmbench

  * http://hdrhistogram.github.io/HdrHistogram/
  * probability density graphs

    - https://hydra.snabb.co/build/589970/download/2/report.html
    - https://en.wikipedia.org/wiki/Probability_density_function
    - enhance hist_scipy.py to support more than one benchmark
    - https://docs.scipy.org/doc/scipy/reference/tutorial/stats.html
<commit_msg>Update TODO<commit_after>BUGS
====

* BUG: "python perf timeit --compare-to=pypy" doesn't tune correctly the runner
  for pypy. pypy requires more warmup samples (10) than cpython (1).
* BUG: --duplicate of timeit must be ignored in PyPy, see the discussion
  on the speed mailing list.


TODO
====

* Implement a few system operations on macOS
* store metadata common to all benchmarks at the root in JSON
* --append should get the number of loops from the existing file and avoid
  calibration, at least when no JIT is used.
* Implement a system-wide lock to prevent running two benchmarks at the same
  time, especially using CPU pinning
* system: display uptime
* Need to write unit test for Runner.timeit()
* Calibration run: display time per iteration and total duration
* system:

  * advice: use --affinity when no CPU is isolated
  * set the CPU scaling governor when intel_pstate is not used.
    Use "userland" governor with a fixed CPU speed (max speed)?
  * warn if HyperThread is enabled?
  * check that isolated CPU "respect HyperThreading" (are part of the
    same physical cores)
  * check that rcu_nocbs respects isolated CPUs
  * detect NUMA

* compare: check also if benchmarks are unstable
* External tool (?) to produce nice HTML reports:

  * https://magic.io/blog/uvloop-blazing-fast-python-networking/

    - HTML, JS: d3 graphic library
    - min, max, std dev
    - https://github.com/MagicStack/pgbench
    - https://github.com/MagicStack/vmbench

  * http://hdrhistogram.github.io/HdrHistogram/
  * probability density graphs

    - https://hydra.snabb.co/build/589970/download/2/report.html
    - https://en.wikipedia.org/wiki/Probability_density_function
    - enhance hist_scipy.py to support more than one benchmark
    - https://docs.scipy.org/doc/scipy/reference/tutorial/stats.html

{'fraction_non_alphanumeric': 0.08635996771589992, 'fraction_numerical': 0.005918751681463546, 'mean_word_length': 3.607187112763321, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5459524', 'n_tokens_mistral': 1139, 'n_tokens_neox': 1082, 'n_words': 432}