<commit_before>Various parts of the language, including function calls, lambda
syntaxes, and control flow constructs take an additional argument,
which is a hint for the debugger. This hint can specify information
for entries in a log file, for frames in a stack trace, for levels of
verbosity in a stack trace, and for other purposes:

- Arbitrary information for application-specific debugging support.
  - NOTE: It would seem useful but challenging for this information
    to refer to state resources that are considered external to the
    program (i.e. outliving the program).
- Acceptable criteria for an alternate way to resume the program (e.g.
  a type and unit test suite for any debugger-inserted function
  result).
- Inhibitors for tail call elimination.
- Hints about whether the program may be experiencing a non-"stuck"
  error (e.g. using more stack frames or allocations than
  anticipated).

In "<expr [] lin1 lin2 L1 L2 ; nl1 nl2 NL1 NL2>", the capitalized
tokens represent sets of variables. The variables that come before the
semicolon are linear (usable exactly once), while the rest are
nonlinear (usable zero, one, or more times).

The brackets "[]" contain a sequence of continuation types, and
sometimes they contain the capitalized variable "W" representing an
arbitrary subsequence. The types are `*`, `hint`, and `hinted`.

<expr [*] ; NL> ::= (done)
<expr [] var ; NL> ::= (get var)
<expr [] ; var NL> ::= (get var)
<expr [W] L1 L2 ; NL> ::=
  (let <expr [hint] ; NL (error description)> x <expr [] L1 ; NL>
    <expr [W] L2 ; x NL>)
  -- May result in an error if `x` is used nonlinearly and would be
  -- bound to a linear value.
<expr [W] L1 L2 ; NL> ::= (do <expr [*] L1 ; NL> <expr [W] L2 ; NL>)
<expr [] L1 L2 ; NL> ::=
  (call <expr [hint] ; NL (error description)>
    <expr [] L1 ; NL (function)> <expr [] L2 ; NL (args)>)
  -- May result in an error if a non-(fn ...) is called. May also
  -- result in an error if the function's `args` is used nonlinearly
  -- and would be bound to a linear value.
<expr [W] L ; NL> ::=
  (hints <expr [hint] ; NL> <expr [hinted W] L ; NL>)
<expr [] L ; NL> ::=
  (fn <expr [hint] ; NL (hint for memory inspector)>
    contour args
    <expr [hinted hinted] L ; contour args NL
      (body with entry and exit hints)>)
  -- Results in a linear value if `L` is nonempty.
  --
  -- NOTE: The `contour` variable is only for use with
  -- (leave-contour ...). The code in this function that isn't inside
  -- (leave-contour ...) is considered to be the function's "lexical
  -- internals." For any given function call at run time, as the
  -- program dynamically enters and exits the lexical internals (even
  -- after the function has returned), the time it spends inside that
  -- lexical boundary is considered the "dynamic internals" of that
  -- function call.
  --
  -- Not all function authors will have the discipline to remember to
  -- put (leave-contour ...) in their higher-order functions, in which
  -- case the internal abstraction layers might not be quite as easy
  -- to distinguish from the higher-order client calls. Fortunately,
  -- if a function hogs the stack trace, a user can write a function
  -- that delegates to it inside the appropriate (leave-contour ...)
  -- boundary.
<expr [hinted W] L ; NL> ::=
  (hinted <expr [hint] ; NL> <expr [W] L ; NL>)
<expr [W] L ; contour NL> ::=
  (leave-contour contour
    <expr [hinted hinted W] L ; NL (body with entry and exit hints)>)
<expr [W] L1 L2 L3 ; NL> ::=
  (yield <expr [hint] ; NL (yield emigration hint)><commit_msg>In untyped-debugger-lang, changed (hints ...) to (enter-contour ...).<commit_after>Various parts of the language, including function calls, lambda
syntaxes, and control flow constructs take an additional argument,
which is a hint for the debugger. This hint can specify information
for entries in a log file, for frames in a stack trace, for levels of
verbosity in a stack trace, and for other purposes:

- Arbitrary information for application-specific debugging support.
  - NOTE: It would seem useful but challenging for this information
    to refer to state resources that are considered external to the
    program (i.e. outliving the program).
- Acceptable criteria for an alternate way to resume the program (e.g.
  a type and unit test suite for any debugger-inserted function
  result).
- Inhibitors for tail call elimination.
- Hints about whether the program may be experiencing a non-"stuck"
  error (e.g. using more stack frames or allocations than
  anticipated).

In "<expr [] lin1 lin2 L1 L2 ; nl1 nl2 NL1 NL2>", the capitalized
tokens represent sets of variables. The variables that come before the
semicolon are linear (usable exactly once), while the rest are
nonlinear (usable zero, one, or more times).

The brackets "[]" contain a sequence of continuation types, and
sometimes they contain the capitalized variable "W" representing an
arbitrary subsequence. The types are `*`, `hint`, and `hinted`.

<expr [*] ; NL> ::= (done)
<expr [] var ; NL> ::= (get var)
<expr [] ; var NL> ::= (get var)
<expr [W] L1 L2 ; NL> ::=
  (let <expr [hint] ; NL (error description)> x <expr [] L1 ; NL>
    <expr [W] L2 ; x NL>)
  -- May result in an error if `x` is used nonlinearly and would be
  -- bound to a linear value.
<expr [W] L1 L2 ; NL> ::= (do <expr [*] L1 ; NL> <expr [W] L2 ; NL>)
<expr [] L1 L2 ; NL> ::=
  (call <expr [hint] ; NL (error description)>
    <expr [] L1 ; NL (function)> <expr [] L2 ; NL (args)>)
  -- May result in an error if a non-(fn ...) is called. May also
  -- result in an error if the function's `args` is used nonlinearly
  -- and would be bound to a linear value.
<expr [W] L ; NL> ::=
  (enter-contour contour
    <expr [hinted hinted W] L ; NL (body with entry and exit hints)>)
<expr [] L ; NL> ::=
  (fn <expr [hint] ; NL (hint for memory inspector)>
    contour args
    <expr [hinted hinted] L ; contour args NL
      (body with entry and exit hints)>)
  -- Results in a linear value if `L` is nonempty.
  --
  -- NOTE: Variables named `contour` are only for use with
  -- (leave-contour ...). The code in this function that isn't inside
  -- (leave-contour ...) is considered to be the function's "lexical
  -- internals." For any given function call at run time, as the
  -- program dynamically enters and exits the lexical internals (even
  -- after the function has returned), the time it spends inside that
  -- lexical boundary is considered the "dynamic internals" of that
  -- function call.
  --
  -- Not all function authors will have the discipline to remember to
  -- put (leave-contour ...) in their higher-order functions, in which
  -- case the internal abstraction layers might not be quite as easy
  -- to distinguish from the higher-order client calls. Fortunately,
  -- if a function hogs the stack trace, a user can write a function
  -- that delegates to it inside the appropriate (leave-contour ...)
  -- boundary.
<expr [hinted W] L ; NL> ::=
  (hinted <expr [hint] ; NL> <expr [W] L ; NL>)
<expr [W] L ; contour NL> ::=
  (leave-contour contour
    <expr [hinted hinted W] L ; NL (body with entry and exit hints)>)
<expr [W] L1 L2 L3 ; NL> ::=
  (yield <expr [hint] ; NL (yield emigration hint)>
{'fraction_non_alphanumeric': 0.11034387515536528, 'fraction_numerical': 0.006352713713575473, 'mean_word_length': 3.31328171530673, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5272609', 'n_tokens_mistral': 2208, 'n_tokens_neox': 2122, 'n_words': 1076}