<commit_before>Various parts of the language, including function calls, lambda
syntaxes, and control flow constructs take an additional argument,
which is a hint for the debugger. This hint can specify information
for entries in a log file, for frames in a stack trace, for levels of
verbosity in a stack trace, and for other purposes:

- Arbitrary information for application-specific debugging support.
- Handles to state resources that are considered external to the
  program (i.e. outliving the program).
- Acceptable criteria for an alternate way to resume the program (e.g.
  a type and unit test suite for any debugger-inserted function
  result).
- Inhibitors for tail call elimination.
- Hints about whether the program may be experiencing a non-"stuck"
  error (e.g. using more stack frames or allocations than
  anticipated).

In "<expr [] lin1 lin2 L1 L2 ; nl1 nl2 NL1 NL2>", the capitalized
tokens represent sets of variables. The variables that come before the
semicolon are linear (usable exactly once), while the rest are
nonlinear (usable zero, one, or more times).

The brackets "[]" contain a sequence of continuation types, and
sometimes they contain the capitalized variable "W" representing an
arbitrary subsequence. The types are `hinted`, `yields`, and
`coroutine-output`.

<expr [] var ; NL> ::= (get var)
<expr [] ; var NL> ::= (get var)
<expr [W] L1 L2 ; NL> ::=
  (let <hint NL (error description)> x <expr [] L1 ; NL>
    <expr [W] L2 ; x NL>)
  -- May result in an error if `x` is used nonlinearly and would be
  -- bound to a linear value.
<expr [yields W] L1 L2 ; NL> ::=
  (let-yields <hint NL (error description)> x <expr [yields] L1 ; NL>
    <expr [yields W] L2 ; x NL>)
  -- May result in an error if `x` is used nonlinearly and would be
  -- bound to a linear value.
<expr [] L1 L2 ; NL> ::=
  (call <hint NL (error description)>
    <expr [] L1 ; NL (function)> <expr [] L2 ; NL (args)>)
  -- May result in an error if a non-(fn ...) is called, or if the
  -- args are linear and the function's `sealed-args` is used
  -- nonlinearly.
<expr [yields] L1 L2 L3 ; NL> ::=
  (yields-call <hint NL (error description)>
    <expr [yields] L1 ; NL (function)> <expr [yields] L2 ; NL (args)>)
  -- May result in an error if a non-(yields-fn ...) is called, or if
  -- the args are linear and the function's `sealed-args` is used
  -- nonlinearly.
<expr [W] L ; NL> ::= (hints <hint NL> <expr [hinted W] L ; NL>)
<expr [yields W] L ; NL> ::=
  (yields-hints <hint NL> <expr [yields hinted W] L ; NL>)
<expr [] L ; NL> ::=
  (fn <hint NL (hint for memory inspector)>
    (sealed-vars-to-capture ...) sealed-args
    <expr [hinted hinted] L ; sealed-args NL>)
  -- NOTE: This results in a linear value if `L` is nonempty or if any
  -- of the `sealed-vars-to-capture` is linear.
<expr [] L ; NL> ::=
  (yields-fn <hint NL (hint for memory inspector)>
    (sealed-vars-to-capture ...) sealed-args
    <expr [hinted yields hinted] L ; sealed-args NL>)
  -- NOTE: This results in a linear value if `L` is nonempty or if any
  -- of the `sealed-vars-to-capture` is linear.
<expr [yields W] L ; NL> ::= (yieldless <expr [W] L ; NL>)
<expr [coroutine-output] L1 L2 ; NL> ::=
  (coroutine-output <hint NL (coroutine exit hint)>
    <expr [] L1 ; NL (parent coroutine)>
    <expr [] L2 ; NL (main output)>)
  -- May result in an error if the parent coroutine isn't the right
  -- coroutine.
<expr [hinted W] L ; NL> ::= (hinted <hint NL> <expr [W] L ; NL>)
<expr [W] sealed-x L ; NL> ::=
  (unseal x sealed-x <hint NL> <expr [hinted W] x L ; NL>)
<expr [W] L ; sealed-x NL> ::=
  (unseal x sealed-x <hint x NL> <expr [hinted W] L ; x NL>)
  -- NOTE: By designing the language so a function's `sealed-args`
  -- must be passed through (unseal ...) before use, we make it so the
  -- function's implementation in between can have an arbitrary amount
  -- of abstraction without hogging the stack trace.
<expr [yields W] L1 L2 L3 ; NL> ::=
  (yield <hint NL (yield entry hint)>
    coroutine-after result
    <expr [] L1 ; NL (coroutine before)>
    <expr [hinted] L2 ; NL (coroutine input)>
    <expr [hinted yields W] coroutine-after L3 ; result NL>)
  -- May result in an error if the coroutine is complete, if it isn't
  -- a coroutine, or if it's a coroutine outside the sandbox. May also
  -- result in an error if `result` or the coroutine's
  -- `first-coroutine-input` is used nonlinearly and would be bound to
  -- a linear value.
<expr [yields W] L1 L2 L3 ; NL> ::=
  (yield-last <hint NL (yield entry hint)>
    result
    <expr [] L1 ; NL (coroutine)>
    <expr [hinted] L2 ; NL (coroutine input)>
    <expr [hinted yields W] L3 ; result NL>)
  -- May result in an error if the coroutine yields again rather than
  -- completing, if it isn't a coroutine, or if it's a coroutine
  -- outside the sandbox. May also result in an error if `result` or
  -- the coroutine's `first-coroutine-input` is used nonlinearly and
  -- would be bound to a linear value.
<expr [W] L1 ; NL> ::= (yields-sandbox <expr [yields W] L1 ; NL>)
<expr [yields] L ; NL> ::=
  (coroutine <hint NL (hint for memory inspector)>
    (sealed-vars-to-capture ...)
    parent-coroutine first-coroutine-input
    <expr [yields coroutine-output]
      parent-coroutine L ; first-coroutine-input NL>)
  -- Returns a linear value.
<expr [] L1 L2 ; NL> ::=
  (demander-sandbox <hint NL (error description)>
    demander <expr [yields] demander L1 ; NL (effectful code)>
    state demand <expr [] ; demand state NL (fold loop)>)
  -- Waits for the given effectful code to complete, and then returns
  -- the full list of demands, as folded by the given loop. The first
  -- value of `state` is the result of the effectful code.
  --
  -- May result in an error if `demand` or `state` is used nonlinearly
  -- and would be bound to a linear value.
  --
  -- TODO: Figure out what hints we might want for demanders.
<expr [yields W] L1 L2 ; NL> ::=
  (drop-demander <hint NL (error description)>
    <expr [yields] L1 ; NL>
    <expr [yields W] L2 ; NL>)
  -- May result in an error if the demander isn't a demander, or if
  -- it's a demander outside the sandbox.
<expr [yields W] L1 L2 ; NL> ::=
  (split-demander <hint NL (error description)>
    demander-prefix demander-suffix <expr [yields] L1 ; NL>
    <expr [yields W] demander-prefix demander-suffix L2 ; NL>)
  -- The list of demands for the two split-off demanders are
  -- concatenated in order to determine the list of demands for the
  -- original demander.
  --
  -- May result in an error if the demander isn't a demander, or if
  -- it's a demander outside the sandbox.
<expr [yields W] L1 L2 L3 ; NL> ::=
  (demand <hint NL (error description)>
    <expr [yields] L1 ; NL (demander)>
    <expr [yields] L2 ; NL (demand)>
    <expr [yields W] L3 ; NL (afterward)>)
  -- May result in an error if the demander isn't a demander, or if
  -- it's a demander outside the sandbox.

<hint NL> ::= (no-hint)
<hint NL> ::= (expr <expr [] ; NL>)


Notice that the expressions (coroutine ...) and (demander-sandbox ...)
can be used in a completely nonlinear context, but still introduce
linear variables. Because of this, linear variables still make a
showing in the language even if no linear variables are used in the
program's top level or I/O.

Notice that in this language, the nesting of CPS-style abstractions
can be visualized a lot like non-CPS nested function calls thanks to
(unseal ...). And thanks to coroutines, certain implementations of
language interpreters written in this language can inherit this
language's debugger support without hogging the main stack!<commit_msg>In untyped-debugger-lang.txt, went with two kinds of convenient side effects.<commit_after>Various parts of the language, including function calls, lambda
syntaxes, and control flow constructs take an additional argument,
which is a hint for the debugger. This hint can specify information
for entries in a log file, for frames in a stack trace, for levels of
verbosity in a stack trace, and for other purposes:

- Arbitrary information for application-specific debugging support.
- Handles to state resources that are considered external to the
  program (i.e. outliving the program).
- Acceptable criteria for an alternate way to resume the program (e.g.
  a type and unit test suite for any debugger-inserted function
  result).
- Inhibitors for tail call elimination.
- Hints about whether the program may be experiencing a non-"stuck"
  error (e.g. using more stack frames or allocations than
  anticipated).

In "<expr [] lin1 lin2 L1 L2 ; nl1 nl2 NL1 NL2>", the capitalized
tokens represent sets of variables. The variables that come before the
semicolon are linear (usable exactly once), while the rest are
nonlinear (usable zero, one, or more times).

The brackets "[]" contain a sequence of continuation types, and
sometimes they contain the capitalized variable "W" representing an
arbitrary subsequence. The types are `result`, `hint`, `hinted`, and
`coroutine-output`. The `result` type actually appears implicitly at
the end of "[]" brackets unless they end with a capitalized variable
or a *.

Before the "[]" section, the word `writes` or `yields` may occur,
representing the following three levels of effects:

- <expr [] ...>: A pure computation.
- <expr writes [] ...>:
  A computation that writes to ordered lists. The order of each list
  is NOT determined by the order of the code, but instead by the
  aliasing ancestry of the `writer-fn` created during
  `writer-sandbox`. This allows the ordering to be controlled
  independently of the code's causality and independently of
  concurrent writers.
- <expr yields [] ...>:
  Shorthand for <expr writes yields [] ...>. Not only is this
  computation not referentially transparent, but it's also fully
  sequentialized. There's an evaluation order that allows causality to
  to take paths other than the usual path from return values to
  parameters.

A capitalized variable may also occur, standing in for any set of
`writes` or `yields`:

- <expr E [] ...>
- <expr writes E [] ...>

<expr E [*] ; NL> ::= (done)
<expr E [] var ; NL> ::= (get var)
<expr E [] ; var NL> ::= (get var)
<expr E [W] L1 L2 ; NL> ::=
  (let <expr [hint] ; NL (error description)> x <expr E [] L1 ; NL>
    <expr E [W] L2 ; x NL>)
  -- May result in an error if `x` is used nonlinearly and would be
  -- bound to a linear value.
<expr E [W] L1 L2 ; NL> ::=
  (do <expr E [*] L1 ; NL> <expr E [W] L2 ; NL>)
<expr E [] L1 L2 ; NL> ::=
  (call <expr [hint] ; NL (error description)>
    <expr E [] L1 ; NL (function)> <expr E [] L2 ; NL (args)>)
  -- May result in an error if a non-(fn ...) is called, or if the
  -- args are linear and the function's `sealed-args` is used
  -- nonlinearly.
<expr writes E [] L1 L2 ; NL> ::=
  (writes-call <expr [hint] ; NL (error description)>
    <expr writes E [] L1 ; NL (function)>
    <expr writes E [] L2 ; NL (args)>)
  -- May result in an error if a non-(writes-fn ...) is called, or if
  -- the args are linear and the function's `sealed-args` is used
  -- nonlinearly.
<expr yields [] L1 L2 ; NL> ::=
  (yields-call <expr [hint] ; NL (error description)>
    <expr yields [] L1 ; NL (function)>
    <expr yields [] L2 ; NL (args)>)
  -- May result in an error if a non-(yields-fn ...) is called, or if
  -- the args are linear and the function's `sealed-args` is used
  -- nonlinearly.
<expr E [W] L ; NL> ::=
  (hints <expr [hint] ; NL> <expr E [hinted W] L ; NL>)
<expr E [] L ; NL> ::=
  (fn <expr [hint] ; NL (hint for memory inspector)>
    (sealed-vars-to-capture ...) sealed-args
    <expr [hint] ; sealed-args NL (entry hint)>
    <expr [hinted] L ; sealed-args NL>)
  -- NOTE: This results in a linear value if `L` is nonempty or if any
  -- of the `sealed-vars-to-capture` is linear.
<expr E [] L ; NL> ::=
  (writes-fn <expr [hint] ; NL (hint for memory inspector)>
    (sealed-vars-to-capture ...) sealed-args
    <expr [hint] ; sealed-args NL (entry hint)>
    <expr writes [hinted] L ; sealed-args NL>)
  -- NOTE: This results in a linear value if `L` is nonempty or if any
  -- of the `sealed-vars-to-capture` is linear.
<expr E [] L ; NL> ::=
  (yields-fn <expr [hint] ; NL (hint for memory inspector)>
    (sealed-vars-to-capture ...) sealed-args
    <expr [hint] ; sealed-args NL (entry hint)>
    <expr yields [hinted] L ; sealed-args NL>)
  -- NOTE: This results in a linear value if `L` is nonempty or if any
  -- of the `sealed-vars-to-capture` is linear.
<expr E [coroutine-output W] L1 L2 ; NL> ::=
  (coroutine-output <expr [hint] ; NL (coroutine exit hint)>
    <expr E [] L1 ; NL (parent coroutine)>
    <expr E [W] L2 ; NL (main output)>)
  -- May result in an error if the parent coroutine isn't the right
  -- coroutine.
<expr E [hinted W] L ; NL> ::=
  (hinted <expr [hint] ; NL> <expr E [W] L ; NL>)
<expr E [W] sealed-x L ; NL> ::=
  (unseal x sealed-x <expr [hint] ; NL> <expr E [hinted W] x L ; NL>)
<expr E [W] L ; sealed-x NL> ::=
  (unseal x sealed-x <expr [hint] ; x NL>
    <expr E [hinted W] L ; x NL>)
  -- NOTE: By designing the language so a function's `sealed-args`
  -- must be passed through (unseal ...) before use, we make it so the
  -- function's implementation in between can have an arbitrary amount
  -- of abstraction without hogging the stack trace.
<expr E [W] L1 L2 L3 ; NL> ::=
  (yield <expr [hint] ; NL (yield entry hint)>
    coroutine-after result
    <expr E [] L1 ; NL (coroutine before)>
    <expr E [hinted] L2 ; NL (coroutine input)>
    <expr [hint] ; result NL (yield reentry hint)>
    <expr E [W] coroutine-after L3 ; result NL>)
  -- May result in an error if the coroutine is complete, if it
  -- completes rather than yielding again, or if it isn't a coroutine.
  -- May also result in an error if `result` or the coroutine's
  -- `first-coroutine-input` is used nonlinearly and would be bound to
  -- to a linear value.
<expr E [W] L1 L2 L3 ; NL> ::=
  (yield-last <expr [hint] ; NL (yield entry hint)>
    result
    <expr E [] L1 ; NL (coroutine)>
    <expr E [hinted] L2 ; NL (coroutine input)>
    <expr [hint] ; result NL (yield reentry hint)>
    <expr E [W] L3 ; result NL>)
  -- May result in an error if the coroutine yields again rather than
  -- completing or if it isn't a coroutine. May also result in an
  -- error if `result` or the coroutine's `first-coroutine-input` is
  -- used nonlinearly and would be bound to a linear value.
<expr E [] L ; NL> ::=
  (coroutine <expr [hint] ; NL (hint for memory inspector)>
    (sealed-vars-to-capture ...)
    parent-coroutine first-coroutine-input
    <expr yields [coroutine-output]
      parent-coroutine L ; first-coroutine-input NL>)
  -- Returns a linear value.
<expr E [] L1 L2 L3 ; NL> ::=
  (writer-sandbox <expr [hint] ; NL (error description)>
    writer <expr writes E [*] writer L1 ; NL (effectful code)>
    <expr E [] L2 ; NL (fold initial state)>
    state writ <expr E [] L3 ; state writ NL (fold loop)>)
  -- Waits for the given effectful code to complete, and then returns
  -- the full list of writes, as folded by the given loop. The first
  -- value of `state` is the result of the effectful code.
  --
  -- May result in an error if not all split-apart fragments of the
  -- writer are taken care of before the effectful code completes. May
  -- also result in an error if `writ` or `state` is used nonlinearly
  -- and would be bound to a linear value, or if `L3` is nonempty and
  -- there isn't exactly one writ.
  --
  -- TODO: Figure out what hints we might want for writers.
<expr E [*] L ; NL> ::=
  (drop-writer <expr [hint] ; NL (error description)>
    <expr E [] L ; NL>)
  -- May result in an error if the writer isn't a writer.
<expr E [W] L1 L2 ; NL> ::=
  (split-writer <expr [hint] ; NL (error description)>
    writer-for-prefix writer-for-suffix <expr E [] L1 ; NL>
    <expr E [W] writer-for-prefix writer-for-suffix L2 ; NL>)
  -- May result in an error if the writer isn't a writer.
<expr E [*] L1 L2 ; NL> ::=
  (write <expr [hint] ; NL (error description)>
    <expr E [] L1 ; NL (writer)>
    <expr E [] L2 ; NL (writ)>)
  -- May result in an error if the writer isn't a writer.
<expr E [hint] ; NL> ::= (no-hint)
<expr E [hint] L ; NL> ::= (hint <expr E [] L ; NL>)


Notice that the expressions (coroutine ...) and (writer-sandbox ...)
can be used in a completely nonlinear context, but still introduce
linear variables. Because of this, linear variables still make a
showing in the language even if no linear variables are used in the
program's top level or I/O.

Notice that in this language, the nesting of CPS-style abstractions
can be visualized a lot like non-CPS nested function calls thanks to
(unseal ...). And thanks to coroutines, certain implementations of
language interpreters written in this language can inherit this
language's debugger support without hogging the main stack!
{'fraction_non_alphanumeric': 0.11964497041420118, 'fraction_numerical': 0.007041420118343195, 'mean_word_length': 3.1865246470151103, 'pattern_counts': {'":': 0, '<': 175, '<?xml version=': 0, '>': 175, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5808273', 'n_tokens_mistral': 5507, 'n_tokens_neox': 5370, 'n_words': 2499}