<commit_before>Some thoughts about states, an idea for a replacement for threaded structs, and an equivalent more broadly for classes:

A state would be a new type of top-level element defining a type, looking vaguely like this (need a real example):

state MyState {
  foo: Foo
  bar: Bar

  initializer(fooArg: Foo, barArg: Bar) {
    foo = fooArg
    bar = barArg
  }

  mutator setFoo(newFoo: Foo) {
    foo = newFoo
  }

  view getFoo(): Foo {
    foo
  }
}

//Using looks like this if we have .-access
let myState: &MyState = MyState(foo, bar)
if (blah) {
    myState.setFoo(neverMindThisFooIsBetter)
}
let theChosenFoo = myState.getFoo()

I'm being ambiguous here about whether Foo is a value or reference type; not sure about the implications yet.

But part of the idea here is that you have a bunch of fields joined together like you'd have in a struct, except the
internals are private instead of exposed, plus there's a sort of state machine component with the mutators. This reflects
the design of a lot of "business logic" (i.e. non-data) classes in OO languages.

A selection of thoughts to iron out further:

1) If these functions all had to live inside the state block, it would be kind of a weird exception to how things
generally work in sem1. I suspect we'll want something more like a "mutator function MyState.setFoo(...) { ... }", and
it just has to be in the same module. Then language dialects (or linters for sem1 itself) can enforce the locality of
the functions being near/in the same file/inside the state definition.
2) Can an initializer reject a set of inputs, like a struct with a requires block? Should it return the state via having
an implicit "this" or "init" function that can actually generate a MyState? (That feels cleaner than saying "you have to
return all these things".)
3) How do we make states composable? Related: If I want to modify Foo, is that always allowed? Should I just put
@Constant or @Final if a field shouldn't get changed by any mutators?
4) Implicit assumption here: Mutators should always be called in a well-defined serialized order, due to the restrictions
of unaliasable references. (These are reference types.)
5) Views are meant to pull non-reference values out of the state. Are there cases where we'd want to allow pulling
reference values out (e.g. fields that are other state types)? This goes back to the question of what composition looks
like.
6) StringBuilder in this case would just be a state containing a ListBuilder. Would ListBuilder be implemented as a
state that just contains a List, with the assumption that runtimes do something smarter than copy-on-write? Feels iffy...
7) This assumes a struct-like setup. What if we want a union of states (including, say, a state equivalent of a linked
list or Optional)?
8) Are there better names for any of the special function types?
9) Can functions mutate multiple states at once, or are they tied to a single state?
10) Should mutators use the same "return the output of an implicit init function" trick that I'm considering for
constructors? Should mutators be able to return values to the caller in addition to mutating the state? The latter seems
expected, but precludes the former. But the init function/global setter approach seems like it might be nice in that it
avoids adding a new statement type and seems to add more atomicity to the "setting" operation.
11) Following up more broadly: should there be a guarantee that nothing outside a mutator function sees the state in a
partially mutated state? What if the mutator is calling another function? Can it pass itself to that function? Can we
call another mutator function and see our state as it changed from that? The state from before we called it? Should we
even have access to "this" to pass elsewhere, vs. just the fields? (Anything that messes with the ability to extract/
inline/compose functions is probably not the right approach...)

// Not necessarily a good implementation (doubly-linked would usually be better)
// Would also want to retry a version of this where Node is a union type itself that can be a "null" value
@Export state MutableLinkedList<T> {
  head: Maybe<MLL.Node<T>>

  mutator addFront(item: T) {
    let newHead = MLL.Node(item, head)
    head = Maybe.of(newHead)
  }

  mutator addBack(item: T) {
    let newBack = MLL.Node(item)
    if (head.isSuccess()) {
      // Pretend there's type gating
      // Actually, separate construct idea for sem2: head.ifSuccess({ /* block in which "head" is success value... "*/ })
      // function Maybe.ifSuccess<T,U>(theMaybe: Maybe<T>, @UseFirstArgAsInputName doThis: (T) -> U): Maybe<U>
      // Can keep them separate by explicitly naming the input: head.ifSuccess((newName) -> { /* ... */ })
      head.value.addBack(item)
      // So the naive implementation of this is inefficient, but the obvious fix (keep a reference to the end) requires
      // an alias, which is frustrating.
    }
  }

  // What if we had some construct like...
  @KeepInstantiatedAndUpdated
  ??? getLastField(): MLL.Node {
    head.getLastField()
  }
  // Alternatively:
  alias(head) lastField: Maybe<MLL.Node> // alias(head) means the main reference is either head or referenced by head,
                                         // so for example, we shouldn't pass both lastField and head to a function that
                                         // isn't expecting aliasable references
                                         // So... when/how do we enforce this? Seems like "simultaneous" setters would
                                         // be a good thing? (Also: Is it intended to be a requirement that it be an
                                         // alias, or just a new possibility? Probably the latter, but is that clear? In
                                         // that case, is it really only relaxing restrictions, not adding them? But how
                                         // about the case where we have e.g. current and last, and we want to make sure
                                         // they're *not* the same at view time? That might be a better case to think
                                         // about...
  // Alternative notation: lastField: Maybe<MLL.Node<T>> aliasing head
  // ^ Interesting thing about having it as part of the type is that it's caring about something in the local area (a
  // variable) rather than a global... but we have a similar notion established for whether an ExpressionType could have
  // an existing alias...
}

state MutableLinkedList.Node<T> {
  item: T // final?
  next: Maybe<MLL.Node<T>> // does this need to be a separate RefMaybe type? (which would often confuse people)

  // Some notional notation for overloading?
  constructor.$1(item: T, next: MLL.Node) {
    construct(item, Maybe.success(next))
  }
  constructor.$2(item: T) {
    construct(item, Maybe.failure())
  }
}

// Another question is if there should be a layer of "wrapping"/inaccessibility between MLL's users and the MLL.Node type?<commit_msg>Add some older notes on state<commit_after>Some thoughts about states, an idea for a replacement for threaded structs, and an equivalent more broadly for classes:

A state would be a new type of top-level element defining a type, looking vaguely like this (need a real example):

state MyState {
  foo: Foo
  bar: Bar

  initializer(fooArg: Foo, barArg: Bar) {
    foo = fooArg
    bar = barArg
  }

  mutator setFoo(newFoo: Foo) {
    foo = newFoo
  }

  view getFoo(): Foo {
    foo
  }
}

//Using looks like this if we have .-access
let myState: &MyState = MyState(foo, bar)
if (blah) {
    myState.setFoo(neverMindThisFooIsBetter)
}
let theChosenFoo = myState.getFoo()

I'm being ambiguous here about whether Foo is a value or reference type; not sure about the implications yet.

But part of the idea here is that you have a bunch of fields joined together like you'd have in a struct, except the
internals are private instead of exposed, plus there's a sort of state machine component with the mutators. This reflects
the design of a lot of "business logic" (i.e. non-data) classes in OO languages.

A selection of thoughts to iron out further:

1) If these functions all had to live inside the state block, it would be kind of a weird exception to how things
generally work in sem1. I suspect we'll want something more like a "mutator function MyState.setFoo(...) { ... }", and
it just has to be in the same module. Then language dialects (or linters for sem1 itself) can enforce the locality of
the functions being near/in the same file/inside the state definition.
2) Can an initializer reject a set of inputs, like a struct with a requires block? Should it return the state via having
an implicit "this" or "init" function that can actually generate a MyState? (That feels cleaner than saying "you have to
return all these things".)
3) How do we make states composable? Related: If I want to modify Foo, is that always allowed? Should I just put
@Constant or @Final if a field shouldn't get changed by any mutators?
4) Implicit assumption here: Mutators should always be called in a well-defined serialized order, due to the restrictions
of unaliasable references. (These are reference types.)
5) Views are meant to pull non-reference values out of the state. Are there cases where we'd want to allow pulling
reference values out (e.g. fields that are other state types)? This goes back to the question of what composition looks
like.
6) StringBuilder in this case would just be a state containing a ListBuilder. Would ListBuilder be implemented as a
state that just contains a List, with the assumption that runtimes do something smarter than copy-on-write? Feels iffy...
7) This assumes a struct-like setup. What if we want a union of states (including, say, a state equivalent of a linked
list or Optional)?
8) Are there better names for any of the special function types?
9) Can functions mutate multiple states at once, or are they tied to a single state?
10) Should mutators use the same "return the output of an implicit init function" trick that I'm considering for
constructors? Should mutators be able to return values to the caller in addition to mutating the state? The latter seems
expected, but precludes the former. But the init function/global setter approach seems like it might be nice in that it
avoids adding a new statement type and seems to add more atomicity to the "setting" operation.
11) Following up more broadly: should there be a guarantee that nothing outside a mutator function sees the state in a
partially mutated state? What if the mutator is calling another function? Can it pass itself to that function? Can we
call another mutator function and see our state as it changed from that? The state from before we called it? Should we
even have access to "this" to pass elsewhere, vs. just the fields? (Anything that messes with the ability to extract/
inline/compose functions is probably not the right approach...)

// Not necessarily a good implementation (doubly-linked would usually be better)
// Would also want to retry a version of this where Node is a union type itself that can be a "null" value
@Export state MutableLinkedList<T> {
  head: Maybe<MLL.Node<T>>

  mutator addFront(item: T) {
    let newHead = MLL.Node(item, head)
    head = Maybe.of(newHead)
  }

  mutator addBack(item: T) {
    let newBack = MLL.Node(item)
    if (head.isSuccess()) {
      // Pretend there's type gating
      // Actually, separate construct idea for sem2: head.ifSuccess({ /* block in which "head" is success value... "*/ })
      // function Maybe.ifSuccess<T,U>(theMaybe: Maybe<T>, @UseFirstArgAsInputName doThis: (T) -> U): Maybe<U>
      // Can keep them separate by explicitly naming the input: head.ifSuccess((newName) -> { /* ... */ })
      head.value.addBack(item)
      // So the naive implementation of this is inefficient, but the obvious fix (keep a reference to the end) requires
      // an alias, which is frustrating.
    }
  }

  // What if we had some construct like...
  @KeepInstantiatedAndUpdated
  ??? getLastField(): MLL.Node {
    head.getLastField()
  }
  // Alternatively:
  alias(head) lastField: Maybe<MLL.Node> // alias(head) means the main reference is either head or referenced by head,
                                         // so for example, we shouldn't pass both lastField and head to a function that
                                         // isn't expecting aliasable references
                                         // So... when/how do we enforce this? Seems like "simultaneous" setters would
                                         // be a good thing? (Also: Is it intended to be a requirement that it be an
                                         // alias, or just a new possibility? Probably the latter, but is that clear? In
                                         // that case, is it really only relaxing restrictions, not adding them? But how
                                         // about the case where we have e.g. current and last, and we want to make sure
                                         // they're *not* the same at view time? That might be a better case to think
                                         // about...
  // Alternative notation: lastField: Maybe<MLL.Node<T>> aliasing head
  // ^ Interesting thing about having it as part of the type is that it's caring about something in the local area (a
  // variable) rather than a global... but we have a similar notion established for whether an ExpressionType could have
  // an existing alias...
}

state MutableLinkedList.Node<T> {
  item: T // final?
  next: Maybe<MLL.Node<T>> // does this need to be a separate RefMaybe type? (which would often confuse people)

  // Some notional notation for overloading?
  constructor.$1(item: T, next: MLL.Node) {
    construct(item, Maybe.success(next))
  }
  constructor.$2(item: T) {
    construct(item, Maybe.failure())
  }
}

// Another question is if there should be a layer of "wrapping"/inaccessibility between MLL's users and the MLL.Node type?

// Okay, let's write this again:

state union MutableLinkedList.NodeRef<T> {
  Null
  Node: MLL.Node<T>
}

// Arguably a little annoying that this can't be defined inline in the union; seems like a sem2 fix
state MutableLinkedList.Node<T> {
  item: T
  next: MLL.NodeRef<T>

  // Automatic constructor?

  // Not automatic
  mutator setNext(newNext: MLL.NodeRef<T>) {
    set next = newNext
  }
}

@Export state MutableLinkedList<T> {
  head: MLL.NodeRef<T>
  tail: MLL.NodeRef<T> aliasing head

  constructor() {
    let emptyRef = MLL.NodeRef.Null() // This is a referenced state, not a value
    init(emptyRef, emptyRef) // Double reference allowed due to "aliasing" condition on tail
  }

  mutator addToFront(item: T) {
    let newNode = MutableLinkedList.Node(item, head) // uh, newNode aliases head here; do we suppress head and say it's
                                                     // okay if head is set again by the end of the method?
    set head = MLL.NodeRef.Node(newNode)
  }

  mutator addToBack(item: T) {
    let newEndNode = MLL.NodeRef.Node(MLL.Node(item, MLL.NodeRef.Null())) // Man, that's verbose; the MLLs don't help
    tail.setNext(newEndNode)
    set tail = newEndNode // which suppresses newEndNode, which is fine
  }
}

// Consider some possible broken-out functions
view MutableLinkedList.get<T>(index: Natural): Maybe<T> {
  // Members are automatically in-scope in this notion, and the first argument is implied by the namespace
  head.get(index)
}
view MutableLinkedList.Node.get<T>(index: Natural): Maybe<T> {
  if (Natural.equals(index, Natural."0")) {
    success(item)
  } else {
    // This would love some enhanced notation
    next.when(
      ifNull: () -> { failure() }
      ifNode: (node) -> { node.get(Natural.minus(index, Natural."1")!) }
    )
  }
}
{'fraction_non_alphanumeric': 0.07338066218508513, 'fraction_numerical': 0.002450210466796507, 'mean_word_length': 2.984480600750939, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 38, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3053925', 'n_tokens_mistral': 4252, 'n_tokens_neox': 4094, 'n_words': 2165}