<commit_before>module Benchmark
    exposing
        ( Benchmark
        , benchmark
        , compare
        , describe
        , done
        , progress
        , series
        , step
        , withRuntime
        )

{-| Benchmark Elm Programs

@docs Benchmark


# Creating and Organizing Benchmarks

@docs benchmark, compare, series, withRuntime, describe


# Running

@docs done, step, progress

-}

import Benchmark.Benchmark exposing (Benchmark(..))
import Benchmark.LowLevel as LowLevel exposing (Error(..))
import Benchmark.Status as Status exposing (Status(..))
import Task exposing (Task)
import Time exposing (Time)


-- Benchmarks and Suites


{-| Benchmarks that contain potential, in-progress, and completed runs.

To make these, try [`benchmark`](#benchmark), [`compare`](#compare), or
[`series`](#series), and organize them with [`describe`](#describe).

-}
type alias Benchmark =
    Benchmark.Benchmark.Benchmark


{-| Set the expected runtime for a [`Benchmark`](#Benchmark). This is how we
determine when the benchmark is "done". Note that this sets the _expected_
runtime, not _actual_ runtime. In other words, your function will be run for _at
least_ this long. In practice, it will usually be slightly more.

The default runtime is 5 seconds, which is good enough for most applications. If
you are benchmarking a very expensive function, this may not be high enough. You
will know that this is the case when you see a low number of total samples
(rough guide: under 10,000.)

On the other hand, 5 seconds is almost never _too much_ time, and usually hits
the sweet spot between getting enough data to be useful while not keeping you
waiting. While there's nothing preventing you from lowering this value, think
about it for a long time before you do. It's easy way to get bad data.

All that said, to set the expected runtime to 10 seconds, pass your constructed
benchmark into this function:

    benchmark "list head" (\_ -> List.head [1])
        |> withRuntime (10 * Time.second)

This works with all the kinds of benchmarks you can create. If you provide a
composite benchmark (a series or group) the same expected runtime will be set
for all members recursively.

-}
withRuntime : Time -> Benchmark -> Benchmark
withRuntime time benchmark =
    case benchmark of
        Single inner _ ->
            Single inner (ToSize time)

        Series name inners ->
            Series name <| List.map (\( inner, _ ) -> ( inner, ToSize time )) inners

        Group name benchmarks ->
            Group name <| List.map (withRuntime time) benchmarks



-- Creation and Organization


{-| Group a number of benchmarks together. Grouping benchmarks using `describe`
will never effect measurement, only organization.

You'll typically have at least one call to this in your benchmark program, at
the top level:

    describe "your program"
        [ -- all your benchmarks
        ]

-}
describe : String -> List Benchmark -> Benchmark
describe =
    Group


{-| Benchmark a named function.

    benchmark "head" (\_ -> List.head [1])

The name here should be short and descriptive. Ideally, it should also uniquely
identify a single benchmark among your whole suite.

Your code is wrapped in an anonymous function, which we will call repeatedly to
measure speed. Note that this is slightly slower than calling functions
directly. This is OK! The point of this library is to _reliably_ measure
execution speed. In this case, we get more consistent results between functions
and runs by calling them inside thunks like this.

Now, a note about benchmark design: when you first write benchmarks, you usually
think something along the lines of "I need to test the worst possible case."
This is a fair observation and a useful thing to measure eventually, but it's
not a good _first_ step. Unless you're careful, you'll end up with too small a
sample size to be useful, and a very difficult benchmark to maintain.

Instead, benchmark the smallest _real_ sample. If your typical use of a data
structure has 20 items, measure with 20 items. If your model is in one
particular state 90% of the time, measure that. It's helpful to get edge cases
eventually, but better to get the basics right first. Solve the problems you
know are real instead of fabricating more out of whole cloth.

When you get the point where you _know_ you need to test a bunch of different
sizes, we've got your back: that's what [`series`](#series) is for.

-}
benchmark : String -> (() -> a) -> Benchmark
benchmark name fn =
    Single (LowLevel.benchmark name fn) Status.init


{-| Specify that two benchmarks are meant to be directly compared.

As with [`benchmark`](#benchmark), the first argument is the name for the
comparison.

    compare "initialize"
        (benchmark2 "HAMT" HAMT.initialize 10000 identity)
        (benchmark2 "Core" Array.initialize 10000 identity)

When you're doing comparisons, try as hard as possible to **make the arguments
the same**. The comparison above wouldn't be accurate if we told HAMT to
initialize an array with only 5,000 elements. Likewise, try to **use the same
benchmark function**. For example, use only `benchmark2` instead of mixing
`benchmark` and `benchmark2`. The difference between the different benchmark
functions is small, but not so small that it won't influence your results.
See the chart in the README for more on the runtime cost of different functions.

-}
compare : String -> String -> (() -> a) -> String -> (() -> b) -> Benchmark
compare name name1 fn1 name2 fn2 =
    Series name
        [ ( LowLevel.benchmark name1 fn1, Status.init )
        , ( LowLevel.benchmark name2 fn2, Status.init )
        ]


{-| Create (and compmare) a series of benchmarks.

This is especially good for testing out the performance of your data structures
at various scales.

    -- TODO: example!

Beware that large series can make very intensive benchmarks, and adjust your
size and expectations accordingly!

-}
series : String -> List ( String, () -> a ) -> Benchmark
series name series =
    series
        |> List.map
            (\( subName, fn ) ->
                ( LowLevel.benchmark subName fn
                , Status.init
                )<commit_msg>update docs for compare<commit_after>module Benchmark
    exposing
        ( Benchmark
        , benchmark
        , compare
        , describe
        , done
        , progress
        , series
        , step
        , withRuntime
        )

{-| Benchmark Elm Programs

@docs Benchmark


# Creating and Organizing Benchmarks

@docs benchmark, compare, series, withRuntime, describe


# Running

@docs done, step, progress

-}

import Benchmark.Benchmark exposing (Benchmark(..))
import Benchmark.LowLevel as LowLevel exposing (Error(..))
import Benchmark.Status as Status exposing (Status(..))
import Task exposing (Task)
import Time exposing (Time)


-- Benchmarks and Suites


{-| Benchmarks that contain potential, in-progress, and completed runs.

To make these, try [`benchmark`](#benchmark), [`compare`](#compare), or
[`series`](#series), and organize them with [`describe`](#describe).

-}
type alias Benchmark =
    Benchmark.Benchmark.Benchmark


{-| Set the expected runtime for a [`Benchmark`](#Benchmark). This is how we
determine when the benchmark is "done". Note that this sets the _expected_
runtime, not _actual_ runtime. In other words, your function will be run for _at
least_ this long. In practice, it will usually be slightly more.

The default runtime is 5 seconds, which is good enough for most applications. If
you are benchmarking a very expensive function, this may not be high enough. You
will know that this is the case when you see a low number of total samples
(rough guide: under 10,000.)

On the other hand, 5 seconds is almost never _too much_ time, and usually hits
the sweet spot between getting enough data to be useful while not keeping you
waiting. While there's nothing preventing you from lowering this value, think
about it for a long time before you do. It's easy way to get bad data.

All that said, to set the expected runtime to 10 seconds, pass your constructed
benchmark into this function:

    benchmark "list head" (\_ -> List.head [1])
        |> withRuntime (10 * Time.second)

This works with all the kinds of benchmarks you can create. If you provide a
composite benchmark (a series or group) the same expected runtime will be set
for all members recursively.

-}
withRuntime : Time -> Benchmark -> Benchmark
withRuntime time benchmark =
    case benchmark of
        Single inner _ ->
            Single inner (ToSize time)

        Series name inners ->
            Series name <| List.map (\( inner, _ ) -> ( inner, ToSize time )) inners

        Group name benchmarks ->
            Group name <| List.map (withRuntime time) benchmarks



-- Creation and Organization


{-| Group a number of benchmarks together. Grouping benchmarks using `describe`
will never effect measurement, only organization.

You'll typically have at least one call to this in your benchmark program, at
the top level:

    describe "your program"
        [ -- all your benchmarks
        ]

-}
describe : String -> List Benchmark -> Benchmark
describe =
    Group


{-| Benchmark a named function.

    benchmark "head" (\_ -> List.head [1])

The name here should be short and descriptive. Ideally, it should also uniquely
identify a single benchmark among your whole suite.

Your code is wrapped in an anonymous function, which we will call repeatedly to
measure speed. Note that this is slightly slower than calling functions
directly. This is OK! The point of this library is to _reliably_ measure
execution speed. In this case, we get more consistent results between functions
and runs by calling them inside thunks like this.

Now, a note about benchmark design: when you first write benchmarks, you usually
think something along the lines of "I need to test the worst possible case."
This is a fair observation and a useful thing to measure eventually, but it's
not a good _first_ step. Unless you're careful, you'll end up with too small a
sample size to be useful, and a very difficult benchmark to maintain.

Instead, benchmark the smallest _real_ sample. If your typical use of a data
structure has 20 items, measure with 20 items. If your model is in one
particular state 90% of the time, measure that. It's helpful to get edge cases
eventually, but better to get the basics right first. Solve the problems you
know are real instead of fabricating more out of whole cloth.

When you get the point where you _know_ you need to test a bunch of different
sizes, we've got your back: that's what [`series`](#series) is for.

-}
benchmark : String -> (() -> a) -> Benchmark
benchmark name fn =
    Single (LowLevel.benchmark name fn) Status.init


{-| Specify two benchmarks which are meant to be compared directly. This is most
useful when optimizing data structures and other situations where you can make
apples-to-apples comparisons between different approaches.

As with [`benchmark`](#benchmark), the first argument is the name for the
comparison. The other string arguments are the names of the functions that
follow them directly.

    compare "initialize"
        "HAMT"
        (\_ -> Array.HAMT.initialize 10000 identity)
        "Core"
        (\_ -> Array.initialize 10000 identity)

The same advice as single benchmarks applies to comparison benchmarks. In
addition, try as hard as possible to make the arguments the same. It wouldn't be
a valid comparison if, in the example above, we told `Array.HAMT` to use 5,000
items instead of 10,000. In the cases where you can't get _exactly_ the same
arguments, at least try to match functionality.

-}
compare : String -> String -> (() -> a) -> String -> (() -> b) -> Benchmark
compare name name1 fn1 name2 fn2 =
    Series name
        [ ( LowLevel.benchmark name1 fn1, Status.init )
        , ( LowLevel.benchmark name2 fn2, Status.init )
        ]


{-| Create (and compmare) a series of benchmarks.

This is especially good for testing out the performance of your data structures
at various scales.

    -- TODO: example!

Beware that large series can make very intensive benchmarks, and adjust your
size and expectations accordingly!

-}
series : String -> List ( String, () -> a ) -> Benchmark
series name series =
    series
        |> List.map
            (\( subName, fn ) ->
                ( LowLevel.benchmark subName fn
                , Status.init
                )
{'fraction_non_alphanumeric': 0.06987179487179487, 'fraction_numerical': 0.007291666666666667, 'mean_word_length': 3.2993455046503617, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1410104', 'n_tokens_mistral': 3431, 'n_tokens_neox': 3139, 'n_words': 1688}